Inter-Annotator Agreement Explained
How inter-annotator agreement works, when to reach for percent agreement, Cohen's or Fleiss' kappa, or Krippendorff's alpha, and what Potato reports.
Inter-annotator agreement (IAA) measures how often independent annotators give the same label. It is the standard evidence that an annotation task is well-defined and the resulting labels are trustworthy. Low agreement usually means the guidelines are unclear, not that the annotators are careless.
The general topic is inter-rater reliability, the extent to which independent raters assign the same value to the same unit. Potato computes agreement live in the admin dashboard and reports it per annotation scheme, so you can see which question your annotators split on rather than one number for the whole study. See Quality Control for the dashboard itself.
Why raw percent agreement isn't enough
Percent agreement is the simplest measure, the fraction of items annotators labeled identically. Some of that agreement happens by chance, which is what makes the raw fraction misleading. If two annotators both pick "positive" 90% of the time, they will agree on most items even when they label at random. Chance-corrected coefficients subtract the agreement you would expect from the label distribution alone.
A chance-corrected coefficient has the form:
P_observed − P_expected
κ = ─────────────────────────
1 − P_expected
where P_observed is the actual agreement and P_expected is the agreement expected by chance. A value of 1 is perfect agreement; 0 is chance-level.
The three coefficients
- Cohen's kappa: two annotators, categorical labels. The classic choice for a pair.
- Fleiss' kappa: more than two annotators, categorical labels, when different items may be judged by different raters.
- Krippendorff's alpha: the most general option. It works with any number of annotators, handles missing data, and supports nominal, ordinal, interval, and ratio data. Potato computes it for every scheme type it can classify.
Use kappa for simple categorical pairs; reach for Krippendorff's alpha when you have many annotators, incomplete overlap, or ordered/continuous labels (where "off by one" should count less than "off by four").
Fleiss' kappa or Krippendorff's alpha with three annotators?
If all three annotators labeled every item and the labels are unordered categories, either coefficient is defensible. If any item is missing a label, or the labels are ordered, use Krippendorff's alpha.
The two differ in what they assume:
- Missing labels. Fleiss' kappa expects the same number of ratings on every item. If one of the three annotators skipped an item, you have to drop that item. Krippendorff's alpha uses every rating it has.
- Level of measurement. Fleiss' kappa treats labels as unordered, so on a 1–5 scale a 4 against a 5 counts as the same disagreement as a 1 against a 5. Krippendorff's alpha takes a distance metric (nominal, ordinal, interval or ratio) and scores near-misses as smaller disagreements.
- Small samples. Krippendorff's alpha includes a correction for small samples, so on a pilot of a few dozen items the two can come apart even when the data suit both.
On complete, nominal data from a reasonable number of items, the two usually land close together. Report the one your field expects, and name the coefficient and the metric in the paper, because "agreement was 0.72" means different things under each.
Potato reports several coefficients at once rather than making you pick one up front. A categorical scheme gets percent agreement, Cohen's kappa, Fleiss' kappa and nominal alpha side by side, and an ordered scheme gets linearly and quadratically weighted kappa, Spearman's rho and ordinal alpha. Before version 2.9, a labeled Likert scale's ordinal alpha ranked the word labels alphabetically, which made it nominal in practice, so recompute ordinal agreement on labeled scales if you reported it from an earlier version. Up to version 2.10.1, ordinal alpha weighted a disagreement by the plain difference between ranks, which is not Krippendorff's ordinal metric. From 2.10.2 it matches the calculator's ordinal alpha, so recompute any ordinal alpha you reported from an earlier version. You can also check a table of labels in the browser with the agreement calculator, which computes nominal, ordinal and interval alpha alongside Cohen's and Fleiss' kappa.
Interpreting the number
There is no universal cutoff. The thresholds below are the ones conventionally used with alpha in content analysis, where researchers rely on data at 0.80 or above and set aside data under 0.67:
- ≥ 0.80: good enough to rely on.
- 0.67–0.80: usable for tentative conclusions; investigate disagreements.
- < 0.67: revisit the guidelines before trusting the labels.
Treat these as prompts to investigate rather than pass/fail gates. Always look at which items and which labels drive disagreement.
Measuring it in Potato
Agreement needs the same item labeled by more than one person, so have annotators overlap on a shared subset before you turn reporting on. min_overlap sets how many annotators an item needs before it counts toward the number:
agreement_metrics:
enabled: true
min_overlap: 2 # an item counts once this many annotators have answered it
refresh_interval: 60 # seconds between dashboard updatesNothing is reported for a scheme until min_overlap annotators have answered the same item, so a single-annotator pilot shows a message rather than a number.
For span and structured tasks, measure agreement at the level you care about, because document-level metrics hide boundary disagreements. Potato reports token-level kappa, exact and partial span F1, Krippendorff's alpha over per-character labels and a simplified form of Mathet's gamma for span schemes, which lets you separate a disagreement about the label from a disagreement about where the span starts.
When agreement is low
- Read the disagreed items and decide whether the guideline is ambiguous or the item is genuinely hard.
- Tighten definitions and add the hard cases as examples. See Writing Annotation Guidelines.
- Re-pilot. If agreement stays low on truly subjective tasks, consider capturing the disagreement itself rather than forcing a single answer.
Frequently asked questions
The questions below cover the range of Krippendorff's alpha, how it is calculated, and agreement with only two annotators.
Can Krippendorff's alpha be less than -1?
No. Alpha falls below 0 when annotators disagree more than chance predicts, and it approaches -1 without reaching it. Two annotators who disagree on every one of N items, using two labels equally often, give an alpha of exactly -1 + 1/N, which is -0.8 for 5 items and -0.95 for 20. Krippendorff (2011) gives the range that matters for reliability as 0 to 1 and attributes values below 0 to systematic disagreement or sampling error. A clearly negative alpha points to systematic disagreement, such as annotators reading a scale in opposite directions, and calls for a look at the guidelines rather than a report of the number.
How do I calculate Krippendorff's alpha?
Alpha is 1 minus the ratio of observed disagreement to the disagreement expected by chance. Observed disagreement comes from pairs of labels given to the same item, and expected disagreement comes from pairs drawn from all labels in the study, with a distance metric (nominal, ordinal, interval, or ratio) deciding how much each mismatch counts. Potato computes it per annotation scheme in the admin dashboard once min_overlap annotators have labeled the same item, and the agreement calculator computes nominal, ordinal, and interval alpha from a table of labels in the browser.
Can inter-annotator agreement be calculated with only two annotators?
Yes. Cohen's kappa is designed for exactly two annotators with categorical labels, and Krippendorff's alpha works with two as well as with more. Both need the two annotators to have labeled the same items independently, and with few items the estimate is unstable, so report the number of items alongside the coefficient.