TokenPad
Evaluation

Inter-Rater Agreement Calculator (Cohen's Kappa)

Find out whether your rubric is applicable before you automate it.

One case per line: rater1,rater2

79 characters8 lines0 tokensor drop a file

Rater AgreementExact
0Token change
Token change0no change
Input tokens0what you pasted
Output tokens0what you would send
Token cost of this result
Output tokens0
As input$0.00
× 100K requests$0.00

Everything on this page runs in your browser. Nothing you paste is transmitted, because there is no server here to transmit it to.

Result

Why raw agreement misleads

Two raters who both say "pass" ninety percent of the time will agree about eighty-two percent of the time by chance alone. Reporting that as agreement tells you nothing about whether they share a standard.

Kappa subtracts the agreement you would expect from the label distribution, which is why it is the number to quote and raw percentage is not.

The 0.6 rule

Below roughly 0.6, stop and fix the rubric. If two humans cannot agree on what good looks like, an automated judge built on the same criteria cannot either — it will produce consistent numbers that mean nothing.

The disagreement list is the repair tool: read the cases where raters differed and you will usually find one criterion doing all the damage.

Frequently asked questions

Can I use this for a human against an LLM judge?
Yes, and it is the standard way to validate one. Compute kappa between the judge and a human on a sample; if it matches human-to-human agreement, the judge can replace the human on that task.
What about more than two raters?
Cohen's kappa is defined for two. Fleiss' kappa generalises it — for a quick check, run pairwise comparisons and look at the spread.

More evaluation tools