Checklist · Quality
LLM Evaluation Setup Checklist
Every team eventually builds an eval set. Most build one that always passes, which is the same as not having one.
The difference is entirely in case selection and in whether the grader measures the thing you care about.
Ticks are saved in this browser only — close the tab and come back to where you were. Nothing is sent anywhere and there is no account.
0 of 16 done · 5 critical outstanding
Choosing cases
0/6Invented cases test the distribution you expected. Real ones test the distribution you have, which is where the failures are.
Check this with the eval dataset builderThis is the highest-value case source available and it is free. A regression suite that does not include past regressions is not one.
Below roughly fifty, a single case flipping moves the score more than most real improvements do, and every result is noise.
Check this with the prompt ab significanceA set at ninety-eight percent has almost no room to show improvement and plenty to show regression. Keep headroom.
These are the cases nobody writes by hand and every production system receives.
Prompts get tuned to whatever you look at. A held-out set is the only defence against tuning to the test.
Check this with the dataset splitter
Grading
0/5"Looks good" is not a grader. If a person has to read every output, the set will be run once and then never again.
They are free, instant and unambiguous. Reserve the expensive judge for what cannot be checked mechanically.
Check this with the jsonl validatorAn uncalibrated judge measures its own preferences. Agreement below about eighty percent means you are optimising against the judge, not the task.
Check this with the inter rater agreement"Rate this answer one to five" produces a number with no stable meaning between runs. Named criteria produce one that does.
Check this with the eval rubric builderA grader that disagrees with itself between runs turns every comparison into a coin flip.
Process
0/5A score is meaningless without knowing which set produced it. When both are in the same repository, the diff answers that.
Anything harder than that gets run before launches and not before changes, which is exactly backwards.
On a hundred cases, a three-point difference is usually noise. Acting on noise is how prompts drift for a quarter and end up worse.
Check this with the prompt ab significanceA two hundred case set run on every change is a recurring bill. Batch pricing usually halves it and nothing about evaluation needs to be fast.
Check this with the batch savings calculatorA static set slowly stops representing your traffic. Anything that passes for a year without a new case is measuring the past.
Tools for this list
- Evaluation Dataset BuilderCases with expected outputs and difficulty, exported three ways.
- LLM Evaluation Rubric BuilderWeighted criteria with score anchors and mandatory evidence quotes.
- Prompt A/B Test Significance CalculatorTwo-proportion z-test with confidence interval and required sample size.
- Train and Validation Set SplitterSeeded shuffle and split. Same seed, same split, comparable evaluations.
Questions
How many cases do I actually need?
Fifty to see whether something is badly broken, two hundred to detect the size of change a prompt edit usually produces. Below fifty you are mostly measuring which cases you happened to pick.
Is model-as-judge reliable?
For comparative judgements against a written rubric, reasonably. For absolute quality scores, less so. Always measure agreement against human labels on a sample before trusting it, and re-measure when you change the judge model.
What if I have no production data yet?
Write thirty cases by hand covering what you expect, ship behind a flag, then replace them with real cases as they arrive. The handwritten set is scaffolding, not the finished thing.