Bradley-Terry ranking calculator
Fit a ranking from pairwise preference judgments: model strengths, win probabilities, and an Elo score for comparison. This is the model behind LLM arena leaderboards. Everything runs in your browser.
Computed locally in your browser. Your data is not uploaded anywhere.
How the model works
The Bradley-Terry model (Bradley & Terry, 1952) assigns each item a positive strength and models the chance that item i beats item j as strength(i) / (strength(i) + strength(j)). This calculator fits the strengths with the MM algorithm of Hunter (2004), with light regularization so an item that never loses still gets a finite score. Ties count as half a win for each side.
Elo answers a different question: it updates scores sequentially and depends on the order of the comparisons, which makes sense for ongoing tournaments but adds noise for a fixed batch of judgments. For a one-shot evaluation, report the Bradley-Terry fit. The Elo column here is a cross-check, not the headline number. Chatbot Arena moved from Elo to a Bradley-Terry fit for the same reason.
Pairwise preference beats absolute rating scales when the question is "which output is better" rather than "how good is this output." Annotators are more consistent at comparing than at scoring, which is why preference data dominates RLHF and model evaluation.
Related reading
The pairwise preference design in the showcase collects exactly this data format with Potato, whose evaluation pipeline includes the same Bradley-Terry and Elo fits. For judging LLM outputs specifically, see the guide on evaluating AI agents.