Behind the explanation
Follow the evidence.
Ten starting points for designing an eval, choosing a grader, and understanding what a result can actually support.
01 / Start with the question
What are you measuring?
Start with CheckList if you are new to evaluation. Then document your cases and decide which other outcomes matter alongside task success.
-
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList
Organizes tests around capabilities and expected behavior: basic functionality, changes that should preserve an answer, and changes that should move it in a known direction. A useful way to turn “the model seems unreliable” into concrete cases.
Limit: Focused behavioral tests reveal weaknesses; their frequency in a hand-built suite does not establish how often users encounter them.
-
Datasheets for Datasets
A structured approach to documenting why a dataset exists, who or what it contains, how it was collected, and how it should be used. For an eval set, those answers help readers understand which conclusions can travel beyond the sample.
Limit: Good documentation makes assumptions visible. It does not itself prove that the cases represent your users.
-
Holistic Evaluation of Language Models
HELM pairs defined use scenarios with multiple measures, making tradeoffs visible across dimensions such as accuracy, robustness, fairness and efficiency. It supports asking what a score leaves out before choosing a system.
Limit: A broad benchmark still reflects its chosen scenarios, prompts and measures. It does not cover every product or deployment.
02 / Treat the grader as an instrument
Who checks the check?
Human and model judgments both need a defined task, a usable rubric and validation against evidence relevant to the decision.
-
Best Practices for the Human Evaluation of Automatically Generated Text
Examines evaluation practice and explains how participant selection, instructions, scales, presentation and analysis affect human ratings. Read it before asking reviewers to score an undefined quality such as “helpfulness.”
Limit: The paper studies natural-language generation. Applying its advice to code review requires task-specific criteria and reviewer expertise.
-
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Investigates model judges alongside human preferences and documents biases involving presentation and response characteristics. It is a useful foundation for testing whether a particular judge is suitable for a particular workload.
Limit: Agreement with preferences on these benchmarks does not establish correctness or validate a newer judge on your tasks.
-
Large Language Models Are Not Fair Evaluators
Shows how changing the order of candidate answers can change a model judge's preference. The practical lesson is to probe presentation effects and examine inconsistent judgments before trusting a ranking.
Limit: The experiments cover particular models and tasks. Their bias rates should not be transferred to every judge.
03 / Inspect what the agent made
Run the patch. Preserve the context.
-
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Frames coding evaluation as applying a generated patch to a repository snapshot and executing checks for the requested fix and existing behavior. This is evidence about an artifact, beyond what the agent says it accomplished.
Limit: Passing the supplied tests cannot establish every property of a patch. The original task collection is a historical benchmark, not a current ranking recommendation.
-
SWE-bench evaluation harness
The implementation shows how repository state, execution environments, logs and evaluation runs fit together. Use it as an example of the machinery that must be controlled for results to be repeatable.
Limit: A container does not guarantee a valid setup. Dependencies, grader behavior and cached results still need checks; pin versions when running an experiment.
04 / Make the comparison interpretable
A result needs its conditions.
-
Show Your Work: Improved Reporting of Experimental Results
Explains why reporting the best score alone hides search effort and selection choices. Record the budget, configurations tried and method used to choose a result, so readers can understand the comparison.
Limit: Its analysis concerns model-development experiments. Applying the reporting lesson to agent trials is a methodological extension, not a direct agent-study result.
-
Accounting for Variance in Machine Learning Benchmarks
Studies how choices throughout a learning pipeline affect measured performance. It motivates recording sources of variation and checking whether an apparent improvement survives more than one fortunate run.
Limit: Training-pipeline variation differs from running a fixed agent. The paper supplies no universal repeat count or release threshold for an agent eval.
Method and limits
A reading list with a point of view.
WTF Evals was developed with AI-assisted research and writing. The initial research records, dated September 26, 2026, document inspection of 59 sources and distinguish original evidence, official implementation details and the site's own synthesis. The ten entries above are a selected route through that work.
This is a curated educational guide. It is not a systematic review, an independent replication of these papers, or a claim that every source received human expert review. The annotations are original summaries; the linked publications and repository are the sources to inspect.
Recommendations on this site combine those sources with practical judgment. A historical study is evidence for its stated setting, and current software documentation can change. If a source or interpretation looks wrong, report a correction.
Follow a concrete example from a coding-agent failure to a decision, including what the result cannot prove.