Replay your reviewed patch

This follows the first eval using the same public retry contract. It does not launch an agent or measure a model. Download the starter kit; the offline recipe is included.

Run only code you have reviewed and trust.

Node executes your code with ordinary host permissions, including top-level code before case validation. A fresh folder keeps the starting state clean; it provides no security isolation. This runner has no timeout. Use your own reviewed code, not unreviewed or hostile artifacts.

01 / Prepare

Freeze the task first.

Open a terminal inside a freshly extracted wtf-evals-starter folder. The sibling folder my-retry-eval must not exist. Run this block once; stop if a command fails:

mkdir ../my-retry-eval &&
cp -R . ../my-retry-eval/frozen &&
cd ../my-retry-eval &&
cp -R frozen evaluator &&
mkdir participant capture reports &&
cp frozen/task.json participant/task.json &&
cp frozen/baseline/tests.json participant/tests.json &&
printf '%s\n' \
  'export function retryDelays(attempts, base_seconds, cap_seconds) {' \
  '  throw new Error("Implement the frozen contract.");' \
  '}' > participant/retry.mjs

Keep frozen/ unchanged. It retains the task, six cases, baseline tests, grader, and authored controls before you produce a candidate. Read frozen/task.json: every submitted test, trusted test, case, and exact-file protection check must pass. The rule and paths live in core.mjs; editing task.json alone does not change them.

All following commands run from my-retry-eval. Check the untouched examples:

test ! -e reports/control.json &&
node frozen/run-eval.mjs --cases frozen/cases.json --output reports/control.json

Expect Candidate A to pass with 6/6 cases. Candidate B should fail with 3/6 cases and a failed protected-file check. These are authored sanity controls, not agent results. Investigate any difference before continuing.

02 / Produce

Make the patch yours.

In your editor, implement participant/retry.mjs using its exported signature and the full contract in participant/task.json. Use a self-contained, synchronous module without imports or top-level work. Keep participant/tests.json byte for byte. You now have one function to implement and the actual original test file to preserve.

For a recorded agent attempt, replace both files in participant/ with the two resulting files and review the code before executing it. Keep the actual submitted tests, including any removed test. Never substitute the clean baseline to make protection pass. If you captured only code, stop here: test preservation is unmeasured. Do not continue with the prefilled kit test file; it cannot supply the missing evidence.

Create capture/notes.md. Record whether the patch is authored, recorded agent output, or of unknown origin; when and how you acquired it; its task and starting files; and known model, settings, instructions, tools, and permissions. Mark missing details unknown and omit secrets. State whether the plan preceded the attempt. An older artifact under a later plan is a retrospective replay. These are your declarations, not authenticated provenance.

03 / Execute

Capture both files.

After review, copy the resulting files unchanged, then start a new Node process:

test ! -e capture/retry.mjs &&
test ! -e capture/tests.json &&
cp participant/retry.mjs capture/retry.mjs &&
cp participant/tests.json capture/tests.json &&
cp capture/retry.mjs evaluator/candidates/complete/retry.mjs &&
cp capture/tests.json evaluator/candidates/complete/tests.json &&
test ! -e reports/replay-01.json &&
node evaluator/run-eval.mjs --cases frozen/cases.json --output reports/replay-01.json

Candidate A is now your patch. complete is its fixed slot name, not a quality claim. Candidate B remains the authored failing control. Keep capture/, frozen/, evaluator/, your notes, and the report together. The snapshots preserve input bytes; a byte hash can identify material but cannot prove when or how it was acquired.

04 / Interpret

Read the evidence.

Open reports/replay-01.json. For id: "complete", inspect source, each group's expected, actual, and passed values, the protected file's before and after, and finally overall_pass. It must be true to accept this patch under these checks. Do not average it with the control.

Exit 0 means the runner completed, even when your candidate fails. Missing or malformed files can prevent a report; interrupt a hang with Ctrl-C. An error, interruption, or absent/incomplete report is not a pass. Check that this invocation completed and produced its new report; an old report says nothing about a failed current invocation.

To execute the same saved inputs again, keep them unchanged and use a new name:

test ! -e reports/replay-02.json &&
node evaluator/run-eval.mjs --cases frozen/cases.json --output reports/replay-02.json

The existence checks prevent overwriting these reports; choose unused filenames. For a changed patch or case set, start a new folder and freeze its task and checks before looking at that candidate's outcomes. Retain failures and run errors too.

Two executions of one saved patch are two replays, not two independent agent trials. Those require separately acquired outputs from fresh sessions with recorded conditions. A pass here covers the selected public checks only. The six cases omit some contract branches; non-finite inputs are rejected by the JSON harness before candidate execution. See the coverage guide. This is a first own-code replay, not a general repository evaluator or evidence of an agent's reliability.

Download the kit ↓Back to the first eval