A hands-on example / Coding agents

Green tests.
Good change?

The task: cap retry delays. Compare two implementations, edit the JSON cases, and run the checks to see which change the evidence supports.

No account or API key · JavaScript + JSON · Runs in your browser

01 / The task

Add a ceiling to retry delays.

Imagine this ordinary implementation request arriving in a coding agent's queue:

Add cap_seconds to the retry delay function. Double the delay after each attempt, but never exceed the cap. Keep the existing behavior and tests.
5 attempts · base 1 second · cap 4 seconds[1, 2, 4, 4, 4]

The contract also requires an empty list for zero attempts, a TypeError for invalid types or fractional attempts, and a RangeError for negative attempts or nonpositive delays. The first delay must respect the cap too.

Read the full task contract task.json
Loading the task…

The acceptance rule, fixed before running: all submitted tests, trusted baseline tests, and your JSON cases must pass. The protected baseline test file must remain unchanged.

02 / Inspect the changes

What did each candidate leave behind?

Inspect the implementation and its submitted test file. A passing suite is only useful if you know what it tested.

Candidate A

candidates/complete/

Implementation retry.mjs
Loading the implementation…
Submitted tests tests.json
Loading the tests…

Candidate B

candidates/shortcut/

Implementation retry.mjs
Loading the implementation…
Submitted tests tests.json
Loading the tests…
The protected tests from before the change baseline/tests.json
Loading the baseline…

03 / Edit & run

This is the part the runner reads.

cases.json holds structured inputs and expected outcomes. The runner calls each implementation, compares the actual output, and records the evidence. JSON makes the cases readable by code; the checks make them an eval.

Use an array of cases with a unique id, input, and either expected (an array) or expected_error (TypeError or RangeError). Try changing a cap and its expected delays.

This JSON download includes your edits. The kit ZIP contains the original example cases; unzip it, then replace its cases.json with your downloaded file.

Loading the starter files…

04 / Read the evidence

Follow the result back to the check.

No results yet. Run the checks above to compare both candidates.

What do the default checks cover?

A pass describes the checks you ran. Use this map to see which parts of the task the original six cases exercise, and which still need evidence.

Coverage of the bundled example, before your edits
RequirementEvidence in this exercise
Double the delayordinary-growth checks one uncapped sequence.
Respect the cap, including the first delaydelay-stops-at-cap and base-above-cap.
Return no delays for zero attempts; reject negative attemptszero-attempts and negative-attempts, also checked by the trusted baseline.
Reject invalid or fractional attemptsreject-string-attempts checks a string. Fractional attempts and other wrong types are not in the default cases.
Reject wrong types and nonpositive values for both delaysNo default case checks these branches for base_seconds or cap_seconds. Try the exercise below.
Reject non-finite delay inputsNot tested against the candidates by this JSON runner; see the boundary below.
Preserve the protected test fileA separate exact-text comparison, plus execution of the trusted baseline tests.

The runner has its own boundary. JSON has no NaN or Infinity literals. If a numeric value parses to infinity, such as 1e400, the runner rejects the case before calling either candidate. That is an invalid run, not evidence that a candidate rejected the input. A quoted "Infinity" tests a string instead. The task's non-finite-input requirement needs a separate direct JavaScript check.

Try it: find the missing delay checks
  1. Run the original cases and inspect their evidence. Notice that every base_seconds and cap_seconds is a valid positive number.
  2. Replace the editor contents with the four cases below. They isolate wrong types, zero, and a negative value while keeping the other inputs valid. Predict each outcome before running.
[
  {
    "id": "reject-string-base",
    "input": { "attempts": 2, "base_seconds": "1", "cap_seconds": 4 },
    "expected_error": "TypeError"
  },
  {
    "id": "reject-null-cap",
    "input": { "attempts": 2, "base_seconds": 1, "cap_seconds": null },
    "expected_error": "TypeError"
  },
  {
    "id": "reject-zero-base",
    "input": { "attempts": 2, "base_seconds": 0, "cap_seconds": 4 },
    "expected_error": "RangeError"
  },
  {
    "id": "reject-negative-cap",
    "input": { "attempts": 2, "base_seconds": 1, "cap_seconds": -1 },
    "expected_error": "RangeError"
  }
]
  1. Run the checks and compare expected with actual for each case. Then add distinct cases for a negative base, a zero cap, and other wrong types. Keep only one invalid field per case so the failure is interpretable.
  2. These four cases replace the original six in this run. To keep both kinds of coverage, combine the arrays with unique IDs and rerun. Reset example cases restores the original six; download your edits first if you want to keep them.

Check your reasoning: with exactly these four cases, Candidate A passes 4/4 and Candidate B passes 0/4 JSON cases. B returns arrays instead of the required errors. Its trusted-test and protected-file failures remain separate reasons to reject it. More cases expose more behavior; they still do not establish every possible input or an agent's reliability.

05 / Take it with you

An eval you can actually run.

The download contains these same candidates, baseline tests, JSON cases, and grader. Unzip it, open a terminal in the extracted folder, and run:

node run-eval.mjs --cases cases.json

Requires Node.js 18 or newer. No install step, dependencies, model calls, or API keys. To use your edits, replace the kit's cases.json with the file downloaded above.

JSON for cases. Markdown for context.

cases.json defines inputs and expected outcomes. core.mjs executes and grades them. results.json records the local run. README.md explains how to use and adapt the files.

How to turn this into an eval of your own agent
  1. Start with a real failure. Choose an ordinary task where it can happen. Freeze the task, clean starting fixture, checks, and acceptance rule before seeing a result.
  2. Run the agent on that fixture. Keep evaluator cases and reference answers outside its workspace. Record the model, settings, instructions, tools, permissions, and starting revision.
  3. Grade what it produced. Replace a candidate's implementation and retain its submitted test file. Run the trusted checks; inspect the full diff for weakened assertions and unrelated changes. This exercise's exact file comparison only protects one file.
  4. Repeat on fresh tasks and runs. Retain failures and execution errors as well as passes. These two hand-written examples demonstrate the grader; they cannot establish an agent's failure rate or an improvement.

Ready to use your own file? Replay a reviewed retry patch locally with a fresh fixture, captured tests, and a saved report.

Want the reasoning behind the method? Explore the sources.