Inter-annotator agreement calculator
Paste a CSV of labels and get Krippendorff's alpha, Cohen's kappa, Fleiss' kappa, and raw agreement, with a bootstrap 95% confidence interval on alpha. The computation runs in your browser and your data stays on your machine.
Computed locally in your browser. Your data is not uploaded anywhere.
Which coefficient should you report?
Krippendorff's alpha is the most general choice: it handles any number of annotators, missing ratings, and nominal, ordinal, or interval data. Report it when annotators did not all rate every item, which is the normal situation in crowdsourced studies. Cohen's kappa (Cohen, 1960) applies to exactly two annotators, and Fleiss' kappa (Fleiss, 1971) extends chance correction to groups where every item receives the same number of ratings. When your data violates those assumptions the calculator leaves the cell blank instead of producing a misleading number.
Interpretation conventions vary by field, but Krippendorff's own guidance is common: α ≥ 0.8 supports firm conclusions, 0.667 to 0.8 supports tentative ones, and anything lower means the labels are not reliable enough to analyze. Raw percent agreement is reported alongside these coefficients, never instead of them, because it ignores agreement that happens by chance.
The implementation follows Krippendorff (2011), "Computing Krippendorff's Alpha-Reliability," and matches the simpledorff and krippendorff Python packages to six decimal places on shared test data. The confidence interval is a percentile bootstrap over items with 1000 resamples.
Related reading
The guide on measuring inter-annotator agreement covers study design choices, and agreement for span annotations explains why token-level metrics need different treatment. Potato computes these statistics live in its admin dashboard during a study, so you can catch low agreement before the budget is spent.