Measuring Inter-Annotator Agreement

Use Cohen's kappa for two annotators on unordered categories, Fleiss' kappa for three or more on complete data, and Krippendorff's alpha whenever labels are missing or the scale is ordered. Potato reports alpha in the admin dashboard as annotations arrive, and a per-type report at /admin/iaa.

Inter-annotator agreement (IAA) measures how consistently independent annotators label the same items. Agreement is the standard evidence that a task is defined well enough for the labels to mean something, and it is the number reviewers ask for. Low agreement is more often a guidelines problem than an annotator problem, so the measurement is worth taking early, while the guidelines can still change.

Which agreement metric to use

Why chance correction matters

The simplest measure is percent agreement, the fraction of items two annotators labeled identically. It overstates reliability whenever the label distribution is skewed. Two annotators who each pick "positive" on 90% of items will agree on about 82% of items while labeling at random, so a reported 85% says almost nothing about whether either one read the text. Chance-corrected coefficients divide out that baseline, and all three coefficients below share the same form:

κ = (Po - Pe) / (1 - Pe)

Po is observed agreement and Pe is the agreement expected by chance. A value of 1 is perfect agreement and 0 is chance level. Negative values mean annotators agreed less often than random labeling would predict, which usually points at a label definition two people read in opposite ways.

Choosing a coefficient

The three standard coefficients differ in how many annotators they admit, whether they tolerate missing labels, and whether they treat the labels as ordered.

Cohen's kappa is the classic choice for a pair of annotators and the one most reviewers recognize. Fleiss' kappa extends the idea to more raters but expects the same number of ratings on every item, so a single skipped item has to be dropped from the calculation. Krippendorff's alpha uses every rating it has and takes a distance metric, which is what makes it the right choice on an ordered scale. On a 1 to 5 Likert item, a 4 against a 5 should count as a smaller disagreement than a 1 against a 5, and only alpha scores it that way. On complete nominal data the three usually land close together, so report the one your field expects and name both the coefficient and the distance metric, because "agreement was 0.72" means different things under each. Potato computes Cohen's kappa pairwise when a project has more than two annotators, reporting the mean across pairs alongside the per-pair breakdown, since Cohen's kappa itself is defined only for two raters.

Acceptable thresholds for alpha and kappa

Two conventions are in circulation and they disagree, so name the one you are using when you report a number. Landis and Koch (1977) proposed descriptive bands for kappa, and Potato uses them when it labels a kappa value:

Krippendorff's convention for alpha is stricter and is stated as a decision rule rather than a description. Rely on data at α ≥ 0.800, draw only tentative conclusions between 0.667 and 0.800, and discard data below 0.667.

Neither set of bands is a pass/fail gate, and both were proposed as prompts to investigate. Decide the threshold your project needs before you see the number, because a threshold chosen afterward is a description of the result rather than a standard. Then look at which items and which labels drive the disagreement, since a single confusable label pair often accounts for most of it.

Enabling agreement reporting in Potato

Agreement needs overlap, so the first setting is how many annotators see each item. Three annotators per item is the common default and gives you a per-item majority as well as a coefficient. Potato then computes agreement over the items that reached that cap:

num_annotators_per_item: 3
 
agreement_metrics:
  enabled: true
  min_overlap: 2          # items with at least 2 annotators enter the calculation
  refresh_interval: 60    # seconds between dashboard updates

min_overlap is the floor for an item to count, not a target. Setting it to 2 while collecting 3 annotators per item means the numbers start moving before the corpus is fully covered, which is the point of watching agreement during collection rather than after it.

A complete configuration with agreement reporting turned on looks like this:

annotation_task_name: "Agreement-Tracked Sentiment"
task_description: "Classify the sentiment of short texts."
 
data_files:
  - data/items.json
 
item_properties:
  id_key: id
  text_key: text
 
annotation_schemes:
  - annotation_type: radio
    name: sentiment
    description: "What is the sentiment of this text?"
    labels:
      - Positive
      - Negative
      - Neutral
 
  - annotation_type: likert
    name: intensity
    description: "How strongly is that sentiment expressed?"
    size: 5
    min_label: "Barely"
    max_label: "Strongly"
 
num_annotators_per_item: 3
 
agreement_metrics:
  enabled: true
  min_overlap: 2
  refresh_interval: 60
 
output_annotation_dir: annotation_output

The two schemes are scored differently and deliberately so. sentiment is nominal, so it gets kappa and nominal alpha; intensity is ordinal, so it gets weighted kappa and ordinal alpha, which charge a one-step disagreement less than a four-step one.

What Potato reports per annotation type

The admin dashboard reports Krippendorff's alpha per scheme as annotations come in. The fuller report lives at /admin/iaa, which classifies each scheme and computes the measures that apply to it, over the items whose per-item annotator cap is at least 2 and that have reached it.

Spans and shapes get their own families because a single coefficient hides which part of the task the annotators disagreed about. Two annotators can pick the same entity type and disagree about where the entity starts, which span_f1_partial and span_f1_exact separate. On shapes, detection_alpha answers whether annotators found the same objects and sigma answers whether they drew them in the same place, and the pair is diagnostic in a way that one blended number is not. Agreement for spans and measuring agreement on bounding boxes cover both cases.

Both reports are admin-only. /admin/iaa takes an API key and returns JSON by default:

curl -H "X-API-Key: $POTATO_ADMIN_KEY" \
  "http://localhost:8000/admin/iaa"
 
# Rendered table instead of JSON
curl -H "X-API-Key: $POTATO_ADMIN_KEY" \
  "http://localhost:8000/admin/iaa?format=html"

/admin/api/agreement returns the dashboard's own view, Krippendorff's alpha by schema with its interpretation. An undefined metric comes back as null rather than NaN, which is common early in a project when no item has two annotators yet.

If you want to check a table of labels before configuring anything, the agreement calculator computes nominal, ordinal and interval alpha alongside Cohen's and Fleiss' kappa in the browser.

When agreement comes back low

Read the disagreed items before changing anything about the annotator pool. The question to answer first is whether the guideline is ambiguous or the item is genuinely hard, because the two have different fixes and only the first is cheap. Tighten the definitions, add the hard cases to the guidelines as worked examples, and re-pilot on a fresh subset. Writing annotation guidelines covers the loop in detail.

Two cautions apply to the number itself. Agreement rises when annotators share a source rather than judging independently, so a study with model pre-labels can show high agreement and low reliability at once, which agreement cannot catch rubber-stamping covers. And on genuinely subjective tasks, agreement may stay low no matter how good the guidelines get, in which case capturing the disagreement is more honest than forcing a single answer. Adjudication and disagreement covers that path, and MACE estimates per-annotator competence when you need a single label out of disagreeing annotators.

Further reading

• Inter-Annotator Agreement Explained
• How Many Annotators Do You Need?
• Quality Control feature reference
• Quality control strategies for the checks that run alongside agreement
• Aggregating Crowd Labels
• Potato's quality control source documentation

References

Landis, J. R., and Koch, G. G. (1977). The Measurement of Observer Agreement for Categorical Data. Biometrics, 33(1), 159-174. https://doi.org/10.2307/2529310

Artstein, R., and Poesio, M. (2008). Survey Article: Inter-Coder Agreement for Computational Linguistics. Computational Linguistics, 34(4), 555-596. https://aclanthology.org/J08-4004/

Krippendorff, K. (2004). Content Analysis: An Introduction to Its Methodology, 2nd edition. Sage.