Candidate A
candidates/complete/
Implementation retry.mjs
Loading the implementation…Submitted tests tests.json
Loading the tests…A hands-on example / Coding agents
The task: cap retry delays. Compare two implementations, edit the JSON cases, and run the checks to see which change the evidence supports.
01 / The task
Imagine this ordinary implementation request arriving in a coding agent's queue:
Add cap_seconds to the retry delay function. Double the delay after each attempt, but never exceed the cap. Keep the existing behavior and tests.
[1, 2, 4, 4, 4]The contract also requires an empty list for zero attempts, a TypeError for invalid types or fractional attempts, and a RangeError for negative attempts or nonpositive delays. The first delay must respect the cap too.
Loading the task…The acceptance rule, fixed before running: all submitted tests, trusted baseline tests, and your JSON cases must pass. The protected baseline test file must remain unchanged.
02 / Inspect the changes
Inspect the implementation and its submitted test file. A passing suite is only useful if you know what it tested.
candidates/complete/
Loading the implementation…Loading the tests…candidates/shortcut/
Loading the implementation…Loading the tests…Loading the baseline…03 / Edit & run
cases.json holds structured inputs and expected outcomes. The runner calls each implementation, compares the actual output, and records the evidence. JSON makes the cases readable by code; the checks make them an eval.
Use an array of cases with a unique id, input, and either expected (an array) or expected_error (TypeError or RangeError). Try changing a cap and its expected delays.
This JSON download includes your edits. The kit ZIP contains the original example cases; unzip it, then replace its cases.json with your downloaded file.
Loading the starter files…
04 / Read the evidence
No results yet. Run the checks above to compare both candidates.
A pass describes the checks you ran. Use this map to see which parts of the task the original six cases exercise, and which still need evidence.
| Requirement | Evidence in this exercise |
|---|---|
| Double the delay | ordinary-growth checks one uncapped sequence. |
| Respect the cap, including the first delay | delay-stops-at-cap and base-above-cap. |
| Return no delays for zero attempts; reject negative attempts | zero-attempts and negative-attempts, also checked by the trusted baseline. |
| Reject invalid or fractional attempts | reject-string-attempts checks a string. Fractional attempts and other wrong types are not in the default cases. |
| Reject wrong types and nonpositive values for both delays | No default case checks these branches for base_seconds or cap_seconds. Try the exercise below. |
| Reject non-finite delay inputs | Not tested against the candidates by this JSON runner; see the boundary below. |
| Preserve the protected test file | A separate exact-text comparison, plus execution of the trusted baseline tests. |
The runner has its own boundary. JSON has no NaN or Infinity literals. If a numeric value parses to infinity, such as 1e400, the runner rejects the case before calling either candidate. That is an invalid run, not evidence that a candidate rejected the input. A quoted "Infinity" tests a string instead. The task's non-finite-input requirement needs a separate direct JavaScript check.
base_seconds and cap_seconds is a valid positive number.[
{
"id": "reject-string-base",
"input": { "attempts": 2, "base_seconds": "1", "cap_seconds": 4 },
"expected_error": "TypeError"
},
{
"id": "reject-null-cap",
"input": { "attempts": 2, "base_seconds": 1, "cap_seconds": null },
"expected_error": "TypeError"
},
{
"id": "reject-zero-base",
"input": { "attempts": 2, "base_seconds": 0, "cap_seconds": 4 },
"expected_error": "RangeError"
},
{
"id": "reject-negative-cap",
"input": { "attempts": 2, "base_seconds": 1, "cap_seconds": -1 },
"expected_error": "RangeError"
}
]
Check your reasoning: with exactly these four cases, Candidate A passes 4/4 and Candidate B passes 0/4 JSON cases. B returns arrays instead of the required errors. Its trusted-test and protected-file failures remain separate reasons to reject it. More cases expose more behavior; they still do not establish every possible input or an agent's reliability.
05 / Take it with you
The download contains these same candidates, baseline tests, JSON cases, and grader. Unzip it, open a terminal in the extracted folder, and run:
node run-eval.mjs --cases cases.jsonRequires Node.js 18 or newer. No install step, dependencies, model calls, or API keys. To use your edits, replace the kit's cases.json with the file downloaded above.
cases.json defines inputs and expected outcomes. core.mjs executes and grades them. results.json records the local run. README.md explains how to use and adapt the files.
Ready to use your own file? Replay a reviewed retry patch locally with a fresh fixture, captured tests, and a saved report.