Batch Benchmark

---
description: Launch the 4-variant batch on the currently-free local GPUs (parallel when N>=4, otherwise sequential waves of N). The only sanctioned benchmark form. Pass exactly 4 submission names.
argument-hint: <name1> <name2> <name3> <name4>   (always 4 variants; parallelism = min(N_free_gpus, 4))
---

Run the **4-variant batch** against the AIME24 benchmark. **Batch
size is fixed at 4.** Parallelism depends on free-GPU count `N`:

- `N >= 4` → all 4 variants run in parallel, one per GPU.
- `0 < N < 4` → variants run in **waves of `N`** on the available
  GPUs (sequential fallback). With `N = 1` this is fully serial;
  `N = 2` runs `2 + 2`; `N = 3` runs `3 + 1`.
- `N == 0` → wait, do not launch.

This is the *only* benchmark command sanctioned by the protocol;
single-variant runs are debug-only.

The user passes **4 submission names** (not JSON paths); this
command expands each `<nameI>` into `submissions/<tag>/<nameI>.json`,
where `<tag>` is the session's agent identifier. For the standard
iterate-loop layout, the names look like
`batch_<x>_id0 batch_<x>_id1 batch_<x>_id2 batch_<x>_id3`.

Step 0 — resolve `<tag>`, detect *free* GPUs (not the physical
count — other users may be sharing this host), then count
arguments:
```bash
TAG=<your_agent_tag>           # sanitized model name, set once per session
FREE_GPUS=($(algorithm_scientist/free_gpus.sh)) || {
    echo "no free GPUs — wait, do not launch" >&2; exit 1
}
N=${#FREE_GPUS[@]}
BATCH_SIZE=4
PARALLEL=$N
[ "$PARALLEL" -gt "$BATCH_SIZE" ] && PARALLEL=$BATCH_SIZE
NAMES=($ARGUMENTS)
if [ "${#NAMES[@]}" -ne "$BATCH_SIZE" ]; then
    echo "expected $BATCH_SIZE variants, got ${#NAMES[@]}" >&2
    exit 1
fi
echo "free GPUs: ${FREE_GPUS[*]}  (N=$N, parallel=$PARALLEL, waves=$(( (BATCH_SIZE + PARALLEL - 1) / PARALLEL )))"

Refuse cases:

• N == 0 (helper exited non-zero): "No free GPUs. Wait, do not launch."
• ${#NAMES[@]} != 4: "Batches must contain exactly 4 variants. Got ${#NAMES[@]} — re-design and re-invoke." Do not silently shrink or pad the batch.

Step 1 — concurrency cap. Only one batch may run at a time on the GPUs you target:

jobs -r | wc -l

If background jobs from a previous batch are still alive, refuse to launch — tell the user to wait, or to use the wait-time activities in algorithm_scientist/memory.md. (Other users' processes are already filtered out by free_gpus.sh.)

Step 2 — pre-flight every config locally first (cheap, no GPU):

for name in "${NAMES[@]}"; do
    python -c "from vortex_torch.engine.sgl import check_engine_config; check_engine_config('submissions/${TAG}/${name}.json')" \
        || echo "[preflight] FAILED: ${TAG}/${name}"
done

Refuse to launch any submission whose preflight failed — fix the failing variant first.

Step 3 — RULER pre-filter (quick quality gate, ≥ 0.85). Run algorithm_scientist/run_ruler.py on each variant sequentially on one free GPU. Any variant scoring below 0.85 accuracy on examples/validation.jsonl has structurally broken attention — fix it (widen vortex_topk_val/vortex_topk_ratio or revise the indexer), re-pre-flight, and re-run RULER until all 4 pass before launching AIME24.

for name in "${NAMES[@]}"; do
    CUDA_VISIBLE_DEVICES=${FREE_GPUS[0]} \
        python algorithm_scientist/run_ruler.py --config "submissions/${TAG}/${name}.json"
done

Results land in summary_ruler_submissions/<tag>/<name>/latest.json. Refuse to launch any variant whose RULER accuracy < 0.85.

Step 4 — launch the 4 variants in waves of PARALLEL = min(N, 4), each pinned to a free GPU index (NOT 0…N-1), with wait between waves so a wave's GPUs are free before the next reuses them:

LOGDIR="logs/submission/${TAG}_$(date +%Y%m%d_%H%M%S)"
mkdir -p "$LOGDIR"
for start in $(seq 0 $PARALLEL $((BATCH_SIZE - 1))); do
    end=$((start + PARALLEL))
    [ "$end" -gt "$BATCH_SIZE" ] && end=$BATCH_SIZE
    for y in $(seq $start $((end - 1))); do
        name="${NAMES[$y]}"
        gpu="${FREE_GPUS[$((y - start))]}"
        CUDA_VISIBLE_DEVICES=$gpu \
            python algorithm_scientist/run_submission_aime24.py --config "submissions/${TAG}/${name}.json" \
            > "$LOGDIR/gpu${gpu}_${name}.out" \
            2> "$LOGDIR/gpu${gpu}_${name}.err" &
    done
    wait
done

When N >= 4 this is one wave of 4 (fully parallel — same behaviour as the old protocol). When N < 4 it's ceil(4/N) sequential waves on the available GPUs. Each child writes its result into summary_submissions/<tag>/<name>/<timestamp>__<hash>.json and updates summary_submissions/<tag>/<name>/latest.json itself. (The runner mirrors the config's path under submissions/ into summary_submissions/, so submissions/<tag>/batch_<x>_id<y>.json becomes summary_submissions/<tag>/batch_<x>_id<y>/... — per-agent isolation, no collisions across agents that pick the same stem.)

Step 5 — append a row to algorithm_scientist/memory.md §1 In-flight batches the moment you launch: | <tag> | <batch_id> | <UTC time> | <LOGDIR> | <name1>,…,<name4> | RUNNING |

Step 6 — while waiting (the batch takes 20–60 minutes when fully parallel, longer with N < 4 due to sequential waves; kill any child still running after 60 minutes — log the error in memory.md §4 and treat that variant as failed), do NOT idle. Spend the time reading tutorials / developer guides / source, or designing the next batch (don't launch — concurrent batches OOM the shared GPUs), or analysing children that have already produced their latest.json. See AGENTS.md §5d.

Step 7 — once the outer wait loop returns (all waves done):

• Success → for each <nameI>, read summary_submissions/<tag>/<nameI>/latest.json. Produce a comparison table with columns: name | content_hash | mean@16 | pass@16 | throughput | e2e_time. Identify the Pareto-non-dominated variants for this batch on the (throughput, mean@16) plane — a variant is dominated iff some other variant has both eq