TokenPad
Evaluation

LLM-as-Judge Prompt Builder

Build a judge prompt that does not just reward the longer answer.

What the judge is deciding

78 characters1 lines0 tokensor drop a file

LLM Judge BuilderExact
0Token change
Token change0no change
Input tokens0what you pasted
Output tokens0what you would send
Token cost of this result
Output tokens0
As input$0.00
× 100K requests$0.00

Everything on this page runs in your browser. Nothing you paste is transmitted, because there is no server here to transmit it to.

Result

The three biases that ruin judges

Verbosity bias: judges prefer longer answers regardless of quality. Position bias: in pairwise comparison they favour whichever answer came first. Self-preference: a model rates its own outputs above others.

The generated prompt states explicitly that length is not quality and that answer order is random. Neither eliminates the bias, but both measurably reduce it.

The swap test

The pairwise output ends with the instruction to run the comparison twice with the answers swapped. If the verdict flips, the judge is following position rather than quality and the result should be discarded.

It doubles your judging cost and it is the difference between an evaluation you can act on and a number that feels like one. Run it at least while you are calibrating.

Frequently asked questions

Pairwise or pointwise?
Pairwise is more reliable — models compare better than they score in the abstract — but it does not give you an absolute quality level and it scales quadratically with the number of candidates. Use pairwise to choose between two prompts, pointwise to track quality over time.
Should the judge be the same model being evaluated?
Preferably not, because of self-preference bias. Where you must, run the swap test rigorously and treat close results as ties.

More evaluation tools