TokenPad
Evaluation

Evaluation Dataset Builder

Turn a list of cases into a structured evaluation set with a difficulty mix.

One case per line: input | expected output | difficulty

223 characters3 lines0 tokensor drop a file

Eval Set BuilderExact
0Token change
Token change0no change
Input tokens0what you pasted
Output tokens0what you would send
Token cost of this result
Output tokens0
As input$0.00
× 100K requests$0.00

Everything on this page runs in your browser. Nothing you paste is transmitted, because there is no server here to transmit it to.

Result

Tagging difficulty is what makes the set useful

An aggregate score hides the thing you need to know. A prompt change that improves easy cases and breaks hard ones can look like an improvement, and a difficulty tag makes that visible immediately.

Aim for a deliberate mix. A set of only easy cases reports success you do not have; a set of only hard ones reports failure you cannot act on.

Stable identifiers

Case ids let you track a specific failure across runs and across model versions. Without them, comparing two evaluation runs means matching on input text, which breaks the moment anyone edits a case.

They also make regressions reportable: "case_047 broke" is actionable in a way that "the score dropped two points" is not.

Frequently asked questions

How many cases do I need?
Fifty covers the common paths for a narrow task. Two hundred starts to catch the edges. Beyond that the return flattens quickly unless your task is genuinely broad.
Where should cases come from?
Production failures first — those are the cases that already cost you something. Invented cases tend to cluster around what you imagined, which is precisely the region the system already handles.

More evaluation tools