Annotation statistics tools
Free calculators for the statistics an annotation project actually needs. Paste or upload a CSV and get numbers you can put in a paper. Everything runs in your browser, so your data never leaves your machine.
Inter-annotator agreement calculator
α · κKrippendorff's alpha (nominal, ordinal, interval), Cohen's kappa, Fleiss' kappa, and raw agreement from a CSV of labels, with bootstrap confidence intervals.
Dawid-Skene consensus labels
EMTurn noisy crowd labels into consensus labels with the classic EM model. Also estimates each annotator's reliability from the data alone.
Bradley-Terry pairwise ranking
A vs BFit strengths and win probabilities from pairwise preference judgments, the model behind LLM arena leaderboards. Elo included for comparison.
Annotation power planner
n = ?Simulate how the confidence interval on Krippendorff's alpha narrows as you add items, and get a corpus-size estimate before you spend the budget.
These tools pair with our guides on measuring inter-annotator agreement, aggregating crowd labels, and statistical power in annotation studies. To collect the annotations themselves, Potato computes agreement live in its admin dashboard while your study runs.