Recover Failed Ingest

---
description: Recover a failed main-branch sweep ingest through the normal artifact-reuse path without rerunning GPU benchmarks
argument-hint: <failed-run-or-job-url | pr-number> [source-run-id]
---

Recover the official database ingest for a failed or skipped InferenceX
push-to-main `Run Sweep` workflow by creating a recovery PR that reuses validated
artifacts from an earlier PR sweep. Do not add a one-off recovery workflow.

Inputs from `$ARGUMENTS`:

- Use the first argument as `FAILED_RUN_OR_JOB_URL`.
- Use the optional second argument as `SOURCE_RUN_ID`; treat it as a candidate
  until all source, ancestry, scope, and artifact checks pass.

The most common invocation is a forgotten `/reuse-sweep-run` before merge, where
you are handed the original PR number and/or its `pull_request` sweep run (the
source) rather than a target URL. The failed target is then the push-to-main run
on that PR's merge commit — derive it in step 1. `inspect-target` needs a
run/job URL, not a bare ID.

Run from a clean InferenceX checkout with authenticated `gh`, `git`, `jq`, and
`python3`. Stop on any unexpected command failure.

## Safety rules

- Never rerun the failed target workflow or job.
- The target must be a completed `push` run of
  `.github/workflows/run-sweep.yml` on `main` whose official ingest did not
  complete.
- Reuse only a completed `pull_request` run of `run-sweep.yml`. Unpinned reuse
  requires success. A specifically pinned failed run is allowed only when
  artifact validation proves its available result set is internally consistent;
  only completed points are recovered.
- The source run must belong to the original PR being recovered.
- Stop if that PR changed the recovered configuration's execution semantics
  after the source SHA: image, model, recipe, runner, launcher, benchmark
  arguments, or config values. Unrelated edits and generator/eval-selection
  policy drift are allowed because source coverage is authoritative.
- Preserve all historical `perf-changelog.yaml` bytes. Append recovery entries
  only at the end.
- Keep exactly one full-sweep label on the recovery PR and pin the source run
  with `/reuse-sweep-run <run_id>` before pushing the changelog change.
- The final recovery branch head must have the recovery commit as its first
  parent, the source run SHA as its second parent, and the recovery commit's
  file tree unchanged.
- Never rebase, locally squash, force-push, or otherwise rewrite the recovery
  branch after attaching the source SHA. Those operations can remove the
  ancestry that makes the source run reusable. A GitHub squash merge after all
  checks pass is allowed because it does not rewrite the PR branch.
- Do not rely on `[skip-sweep]`. Reuse authorization suppresses PR benchmark
  work, and pushes to `main` ignore that marker.
- Never bypass failing or pending checks. Use admin merge only when all checks
  passed and repository policy is the sole blocker.
- Do not add co-author lines, generated-by text, bot branding, or attribution.

## 1. Inspect the target

Ensure `pydantic` and `pyyaml` are importable
(`python3 -c 'import pydantic, yaml'`); they are usually already present. If not,
install them — a plain `pip install` fails on PEP 668 managed Pythons, so use a
venv or `--break-system-packages`. Then inspect the target:

```bash
python3 -m pip install pydantic pyyaml  # only if the import check failed
python3 utils/recover_failed_ingest.py inspect-target \
  "$FAILED_RUN_OR_JOB_URL" \
  --output /tmp/infx-recovery-target.json

TARGET_RUN_ID=$(jq -r .run_id /tmp/infx-recovery-target.json)
TARGET_JOB_ID=$(jq -r .job_id /tmp/infx-recovery-target.json)
ORIGINAL_PR=$(jq -r .pr_number /tmp/infx-recovery-target.json)
ORIGINAL_MERGE_SHA=$(jq -r .merge_sha /tmp/infx-recovery-target.json)

gh run view "$TARGET_RUN_ID" \
  --repo SemiAnalysisAI/InferenceX \
  --job "$TARGET_JOB_ID" --log \
  > "/tmp/infx-target-$TARGET_RUN_ID.log"

inspect-target handles completed failed runs. If the target workflow itself was skipped, inspect it directly with gh api, identify the skipped setup or reuse job, and resolve the merge SHA to exactly one merged PR:

TARGET_RUN_ID=<run-id>
TARGET_JSON=$(gh api \
  "repos/SemiAnalysisAI/InferenceX/actions/runs/$TARGET_RUN_ID")
ORIGINAL_MERGE_SHA=$(jq -r .head_sha <<<"$TARGET_JSON")
ORIGINAL_PR=$(gh api \
  "repos/SemiAnalysisAI/InferenceX/commits/$ORIGINAL_MERGE_SHA/pulls" \
  --jq 'if length == 1 then .[0].number else error("expected one PR") end')

If you were given the original PR number or the source sweep run instead of a target URL — the usual forgotten-/reuse case — derive the target push run from the PR's merge commit:

ORIGINAL_PR=<pr-number>
ORIGINAL_MERGE_SHA=$(gh pr view "$ORIGINAL_PR" \
  --repo SemiAnalysisAI/InferenceX \
  --json mergeCommit --jq .mergeCommit.oid)
gh run list --repo SemiAnalysisAI/InferenceX \
  --workflow run-sweep.yml --event push \
  --commit "$ORIGINAL_MERGE_SHA" --limit 5 \
  --json databaseId,status,conclusion,createdAt
TARGET_RUN_ID=<matching-run-id>

Require event push, workflow path .github/workflows/run-sweep.yml, and branch main; confirm the target is no longer running before recovering. The disqualifying state is broader than failure/skipped: when /reuse-sweep-run was forgotten before merge, reuse-ingest-artifacts is skipped, the GPU jobs run (often cancelled to save cost), and because collect-results/collect-evals are not skipped, trigger-ingest still fires always() and lands a bogus ingest under the target's own run_id. So a target showing trigger-ingest=success (and concluding success or cancelled) can still hold no valid benchmark data — recovery is required. That bogus row is keyed on the target run_id and is superseded by the recovery ingest under a new run_id; leave it alone. Record the original PR and root cause.

Fetch history and inspect the exact original changelog delta:

git fetch origin main
git ca