# Run your first eval This kit executes two frozen JavaScript implementations of a retry-delay function. You get structured cases, actual candidate files, a grader, and JSON results. No model, API key, package installation, or network connection is needed. Use Node.js 18 or newer. ## Run it Open a terminal inside the unzipped `wtf-evals-starter` folder, then run: ```sh node run-eval.mjs ``` The runner writes `results.json` and prints: ```text Candidate A: PASS | submitted tests 2/2 | trusted tests 2/2 | cases 6/6 | protected file PASS Candidate B: FAIL | submitted tests 1/1 | trusted tests 1/2 | cases 3/6 | protected file FAIL ``` Candidate B gets a green submitted suite by removing the negative-attempt test. Executing the unchanged baseline exposes the lost behavior. The additional cases also expose its missing delay cap. Open the two `retry.mjs` and `tests.json` files to connect each result to the submitted artifact. A green suite alone misses both problems. ## Change a case and rerun Open `cases.json`. In `delay-stops-at-cap`, change `cap_seconds` from `4` to `2` and `expected` to `[1, 2, 2, 2, 2]`. Run `node run-eval.mjs` again and inspect that case's `expected`, `actual`, and `passed` fields in `results.json`. To keep multiple case sets: ```sh node run-eval.mjs --cases my-cases.json --output my-results.json ``` Both custom paths are relative to your current directory. With no `--cases`, the runner uses the `cases.json` beside `run-eval.mjs`. A successful run exits `0` even when a candidate fails: this exercise intentionally includes a failing candidate. Bad JSON, unsupported fields, missing files, or write failures exit `1`. For a CI gate, check the selected candidate's `overall_pass` and fail the job when it is false. ## What the default checks cover A pass describes the checks you ran. This map refers to the original six cases, before your edits; it is not a claim of complete contract coverage. | Requirement | Evidence in this exercise | | --- | --- | | Double the delay | `ordinary-growth` checks one uncapped sequence. | | Respect the cap, including the first delay | `delay-stops-at-cap` and `base-above-cap`. | | Return no delays for zero attempts; reject negative attempts | `zero-attempts` and `negative-attempts`, also checked by the trusted baseline. | | Reject invalid or fractional attempts | `reject-string-attempts` checks a string. Fractional attempts and other wrong types are not in the default cases. | | Reject wrong types and nonpositive values for both delays | No default case checks these branches for `base_seconds` or `cap_seconds`. Try the exercise below. | | Reject non-finite delay inputs | Not tested against the candidates by this JSON runner; see the boundary below. | | Preserve the protected test file | A separate exact-text comparison, plus execution of the trusted baseline tests. | The runner has its own boundary. JSON has no `NaN` or `Infinity` literals. If a numeric value parses to infinity, such as `1e400`, the runner rejects the case before calling either candidate. That is an invalid run, not evidence that a candidate rejected the input. A quoted `"Infinity"` tests a string instead. The task's non-finite-input requirement needs a separate direct JavaScript check. ## Find the missing delay checks Run the original cases and inspect their evidence. Every `base_seconds` and `cap_seconds` is a valid positive number. Save the following as `delay-validation.json`; predict each outcome before running it: ```json [ { "id": "reject-string-base", "input": { "attempts": 2, "base_seconds": "1", "cap_seconds": 4 }, "expected_error": "TypeError" }, { "id": "reject-null-cap", "input": { "attempts": 2, "base_seconds": 1, "cap_seconds": null }, "expected_error": "TypeError" }, { "id": "reject-zero-base", "input": { "attempts": 2, "base_seconds": 0, "cap_seconds": 4 }, "expected_error": "RangeError" }, { "id": "reject-negative-cap", "input": { "attempts": 2, "base_seconds": 1, "cap_seconds": -1 }, "expected_error": "RangeError" } ] ``` Run `node run-eval.mjs --cases delay-validation.json --output delay-results.json`. Compare `expected` with `actual` for each case. Candidate A passes 4/4 and Candidate B passes 0/4 JSON cases: B returns arrays instead of the required errors. Its trusted-test and protected-file failures remain separate reasons to reject it. Add distinct cases for a negative base, a zero cap, and other wrong types. Keep only one invalid field per case so the failure is interpretable. This custom file replaces the original six cases for that run. To keep both kinds of coverage, combine the arrays with unique IDs and rerun. More cases expose more behavior; they still do not establish every possible input or an agent's reliability. The bundled `cases.json` remains the original six-case example. ## What each file does | File | Role | | --- | --- | | `task.json` | The coding task and acceptance rule, fixed before reviewing candidates. | | `cases.json` | Structured inputs and expected outputs or error types. | | `cases.schema.json` | The case shape, editor hints, and documented limits. | | `baseline/tests.json` | The original trusted test file. | | `candidates/*/retry.mjs` | The actual candidate implementations that are executed. | | `candidates/*/tests.json` | Each candidate's submitted test file. | | `core.mjs` | Shared browser/Node validation, execution, and grading. | | `run-eval.mjs` | Node file loading, command-line options, and results output. | | `results.json` | Generated evidence and pass/fail decisions. | ## Why JSON here, and Markdown here? JSON gives this runner explicit fields to load and validate. Each case supplies `id`, `input`, and exactly one of `expected` or `expected_error`. A string, boolean, or null input is allowed so a case can test type rejection. Strings are data; they are never evaluated as JavaScript or interpreted as instructions. Unknown fields, duplicate IDs, empty case sets, more than 100 cases, and numeric attempts outside −100 to 100 are rejected before execution. The grader calls `retryDelays` with each input and compares arrays element by element, or compares the thrown error's name. It does not infer expectations from prose. JSON alone does not make an eval: the executable grader gives these fields their meaning. JSONL would suit a larger dataset with one case per line; it is not the input format of this runner. This Markdown file explains the exercise. The acceptance rule requires all four checks to pass: submitted tests, trusted tests, additional cases, and preserving the protected test file. No average can hide a failed requirement. The file check deliberately compares exact text: even whitespace changes fail this task's explicit “preserve byte for byte” rule. That is a narrow teaching policy, not a general way to detect weakened tests. ## Adapt it to real work 1. Define the failure you want to detect and freeze cases and acceptance criteria before looking at candidate outcomes. 2. Give an agent a fresh copy of a real task fixture. Keep trusted tests and the grader outside its editable workspace. Record the model, instructions, tools, permissions, and starting state. 3. Save its resulting artifacts. Add an isolated execution adapter for your language and toolchain, then grade the actual files it produced. This kit imports two known local modules; it is not a sandbox for arbitrary agent code. 4. Add cases from observed failures, inspect evidence, and repeat independent runs before comparing approaches. Use a protection policy appropriate to your repo. This example demonstrates grading with hand-authored candidates. It does not call an agent or measure model quality, and six passing cases cannot establish general correctness. The public cases are teaching material, not a hidden test set. The source filenames are fixed in `core.mjs`; changing `task.json` alone does not add candidates or change the grader. For a concrete next step with this same contract, [replay your reviewed retry patch](try-your-patch.md). The recipe prepares a fresh fixture, retains the actual submitted tests, and saves a local report using this runner.