Evaluation
Self-consistency
Self-consistency runs the same prompt several times at non-zero temperature and takes the most common answer.
It improves accuracy on problems with a single correct answer that the model reaches unreliably, because errors tend to scatter while the correct answer recurs.
The cost is linear in the number of samples: five samples is five times the bill and five times the latency unless run in parallel.
In practice
Sampling several answers and taking the majority raises accuracy on problems with one right answer and multiplies cost by the sample count. Five samples is five times the bill, so it earns its place on high-stakes low-volume decisions and almost never on high-volume classification, where a better prompt is cheaper than a vote.
Common questions
How many samples does self-consistency need?
Commonly three to five, with diminishing returns after. Each sample multiplies the cost, so it earns its place on high-stakes low-volume decisions and rarely on high-volume classification.
Does self-consistency work on open-ended tasks?
Poorly. It needs answers that can be compared for agreement, so it suits arithmetic and classification. For open-ended generation there is nothing to take a majority of.