# Replay your reviewed retry patch

Move from the two bundled examples to a file you wrote or captured yourself.
This recipe executes your implementation of the same public retry contract and
keeps the inputs beside the report. It does not launch an agent or measure a model.

Use Node.js 18 or newer and a macOS/Linux shell. Execute only code you own, have
reviewed, and trust: Node imports it with your ordinary host permissions, including
top-level code before case validation. A fresh folder keeps the starting state
clean; it provides no security isolation. This runner has no timeout.

## 1. Freeze the task and prepare a workspace

Start inside a **freshly extracted** `wtf-evals-starter` folder. The sibling folder
`my-retry-eval` must not exist. Run this block once; stop if a command fails:

```sh
mkdir ../my-retry-eval &&
cp -R . ../my-retry-eval/frozen &&
cd ../my-retry-eval &&
cp -R frozen evaluator &&
mkdir participant capture reports &&
cp frozen/task.json participant/task.json &&
cp frozen/baseline/tests.json participant/tests.json &&
printf '%s\n' \
  'export function retryDelays(attempts, base_seconds, cap_seconds) {' \
  '  throw new Error("Implement the frozen contract.");' \
  '}' > participant/retry.mjs
```

Keep `frozen/` unchanged. It records the task, six cases, baseline tests, grader,
and authored controls before you produce a candidate. Read `frozen/task.json`:
acceptance requires every submitted test, trusted test, case, and the exact-file
protection check to pass. The rule and candidate paths are implemented in
`core.mjs`; editing `task.json` alone does not change them.

All following commands run from `my-retry-eval`. Check the untouched examples:

```sh
test ! -e reports/control.json &&
node frozen/run-eval.mjs --cases frozen/cases.json --output reports/control.json
```

Expect Candidate A to pass, including 6/6 cases. Candidate B should fail, with
3/6 cases and a failed protected-file check. These are authored sanity controls,
not agent results. Stop and investigate if they differ.

## 2. Produce and review your patch

In your editor, implement `participant/retry.mjs` using the exported signature
already there and the full contract in `participant/task.json`. Use a
self-contained, synchronous module without imports or top-level work. Keep
`participant/tests.json` byte for byte as the task requires. The fixture is now
concrete: one function to implement and the actual original test file to preserve.

For a recorded agent attempt, replace both files in `participant/` with the two
resulting files and review the code before executing it. Retain the **actual
submitted** test file, even if it changed or lost a test. Do not replace it with
the clean baseline to make protection pass. If you captured only code, stop here:
test preservation is unmeasured. Do not continue with the prefilled kit test file;
it cannot supply the missing evidence.

Create `capture/notes.md` in your editor. Record whether the patch is authored,
recorded agent output, or of unknown origin; when and how it was acquired; the
task and starting files; and any known model, settings, instructions, tools, and
permissions. Mark unavailable details unknown and omit secrets. State whether the
plan preceded the attempt. An older artifact evaluated under a later plan is a
retrospective replay. These are your declarations, not authenticated provenance.

## 3. Capture both files and execute them

After review, copy the resulting files without changing them:

```sh
test ! -e capture/retry.mjs &&
test ! -e capture/tests.json &&
cp participant/retry.mjs capture/retry.mjs &&
cp participant/tests.json capture/tests.json &&
cp capture/retry.mjs evaluator/candidates/complete/retry.mjs &&
cp capture/tests.json evaluator/candidates/complete/tests.json &&
test ! -e reports/replay-01.json &&
node evaluator/run-eval.mjs --cases frozen/cases.json --output reports/replay-01.json
```

The new Node process imports your captured implementation. **Candidate A** is now
your patch; `complete` is just its fixed slot name, not a quality claim. Candidate B
remains the authored failing control. Keep `capture/`, `frozen/`, `evaluator/`,
your notes, and the report together. The snapshots preserve input bytes; a byte
hash can identify material but cannot prove when or how it was acquired.

## 4. Read the decision, then replay deliberately

Open `reports/replay-01.json`. For the candidate with `id: "complete"`, inspect
`source`, each group's `expected`, `actual`, and `passed` values, the protected
file's `before` and `after`, and finally `overall_pass`. It must be true to accept
this patch under these checks. Do not average Candidate A with the control.

Exit `0` means the runner completed, even when your candidate fails. Missing or
malformed files can prevent a report; if execution hangs, interrupt it with
Ctrl-C. An error, interruption, or absent/incomplete report is not a pass. Check
that this invocation completed and produced its new report; an old report says
nothing about a failed current invocation.

To execute the same saved inputs again, keep them unchanged and use a new name:

```sh
test ! -e reports/replay-02.json &&
node evaluator/run-eval.mjs --cases frozen/cases.json --output reports/replay-02.json
```

The existence checks prevent overwriting these reports; choose unused filenames.
For a changed patch or case set, start a new folder and freeze its task and checks
before looking at that candidate's outcomes. Retain failures and run errors too.

Two executions of one saved patch are two replays, not two independent agent
trials. Such trials require separately acquired outputs from fresh sessions with
recorded conditions. A pass here covers the selected public checks only: the six
cases omit some contract branches, and non-finite inputs are rejected by the JSON
harness before candidate execution. See [the coverage guide](README.md#what-the-default-checks-cover).
This is a first own-code replay, not a general repository evaluator or evidence of
an agent's reliability.