Labels With Error Bars: Item Response Theory for Annotation

Majority vote treats every annotator as equally reliable and every item as equally hard, and it reports a count instead of a probability. Item response theory (IRT) drops both assumptions: it estimates each annotator's ability and each item's difficulty from the pattern of agreement alone, then reports every label as a posterior probability with a confidence interval. It needs no gold labels and no LLM. In Potato this is the psychometrics engine, and it runs live as annotations arrive.

Three annotators label an item. Two say sarcastic, one says sincere. Majority vote calls it sarcastic and moves on, and your dataset records that with exactly as much confidence as an item where all three agreed.

That is throwing away information. The dissenter might be your most reliable annotator. The item might be one that everyone finds hard. Neither fact survives the count.

Annotator ability with confidence intervals

What is item response theory?

Item response theory comes from psychometrics, where it was built to score standardized tests. The insight is that a test tells you two things at once: how able each test-taker is, and how hard each question is. You can estimate both jointly, from nothing but the response pattern, because they constrain each other. A question that able students get wrong is a hard question. A student who gets hard questions right is an able student.

Annotation has the same shape. Annotators are test-takers, items are questions, and nobody has the answer key.

Potato fits a multiclass generalization of GLAD (Whitehill et al. 2009), which estimates three things together:

• the true label of each item, as a probability distribution
• each annotator's ability (θ), with a standard error
• each item's difficulty, and a per-item discrimination diagnostic

The fit is deterministic and takes milliseconds at annotation-study scale, so it can run continuously rather than as a post-hoc batch job.

Turning it on

psychometrics:
  enabled: true
  schema: sarcasm
  confidence_threshold: 0.95
  min_annotators_per_item: 2

That alone gives you the analytics layer. Your export changes from this:

sarcastic    (2 of 3 votes)

to this:

sarcastic    p = 0.94 [0.88 – 0.97]

The probability is the model posterior, which weighs who voted rather than only how many. The interval is a sensitivity band over the ability estimates.

Downstream, that number is worth having. You can train on soft labels, filter an evaluation set to items above a confidence floor, or treat low-probability items as genuinely ambiguous instead of quietly mislabeled. Disagreement stops being noise to be averaged away, a point worth reading alongside disagreement is signal, not noise.

Annotator ability, honestly presented

Ability is estimated from agreement patterns. A value of 1.0 is the prior for a newcomer, 0 means the annotator's labels carry no information, and negative means systematically wrong.

Every estimate comes with a standard error, and this matters more than the point estimate does. An annotator with twelve labels has a wide whisker, and the dashboard draws it wide. Do not make personnel decisions from a wide whisker. The standard errors are approximate (Fisher information at the mode) and mildly optimistic by construction. They are there to stop you over-reading a number, not to license firing someone.

If you want a simpler competence readout across any categorical scheme, MACE is the older cousin and still a good tool. The difference is that MACE runs as batch analytics, while psychometrics sits in the assignment loop and acts on the study while it runs.

The codebook-bug detector

This is the part that surprises people.

Alongside difficulty, the model estimates discrimination: on a given item, does annotator ability correlate with agreeing with the consensus? Normally it does. Able annotators agree with the eventual answer more often, which is roughly what "able" means.

When discrimination goes strongly negative, something is wrong. Your best annotators are systematically disagreeing with the crowd on that item. The likeliest explanation is not that your best annotators suddenly became bad. It is that the guideline is ambiguous or wrong for that case, and the careful annotators are the ones noticing.

Potato surfaces these under "Likely codebook bugs." They are usually the highest-value items in the whole study, because each one is a defect in the instrument rather than a defect in a person. Fix the guideline, then re-annotate. A norming room is a good place to work through them as a team.

Spending annotation budget where it buys something

Fixed redundancy, the "three annotators per item, always" rule, spends the same amount on an item everyone agrees about and an item that splits the team down the middle. That is backwards.

assignment_strategy: psychometric
num_annotators_per_item: 4
psychometrics:
  enabled: true
  confidence_threshold: 0.95
  cost_per_judgment: 0.08

With this, when an annotator asks for work, Potato ranks the remaining items by the expected information gain of that specific annotator labeling that specific item. Items whose posterior already clears confidence_threshold stop being served at all. The dashboard counts the judgments you did not have to buy, and prices them if you set cost_per_judgment.

Two caveats worth knowing before you turn it on:

• Cold start. Until enough labels exist, assignment falls back to random. The model cannot separate ability from difficulty without overlapping annotators, and it will not pretend otherwise. The dashboard shows a warming-up meter instead.
• Batching. Ranking is freshest when queues are short. Handing each annotator a batch of 200 items dilutes the adaptivity, because the ranking was computed before most of those labels existed.

Deciding redundancy before you spend anything

The question every project guesses at is how many annotators per item it needs. The study designer answers it with a Monte Carlo simulation instead:

python -m potato.psychometrics.design --items 500 --accuracy 0.75 \
    --classes 3 --target-ci 0.10 --cost 0.08

ann/item   alpha     95% interval   width  majority acc  judgments       cost
       2   0.392 [ 0.330,  0.439]   0.109         0.751       1000      80.00
       3   0.388 [ 0.338,  0.431]   0.093         0.864       1500     120.00  <- recommended

The recommendation is the cheapest redundancy whose 95% interval on Krippendorff's α is still narrower than your target. The --accuracy input is the one you should measure with a small pilot rather than guess.

The same calculation runs in your browser, if you would rather not install anything to answer the question.

What this does not do

IRT is not magic, and a few honest limits:

• It models single-choice categorical schemes, radio and likert. Multi-select is not modeled.
• It needs overlapping annotators. If every item is labeled by a different person, there is no agreement structure to learn from, and the model will tell you it is still warming up.
• With one annotator, or with unanimous labels everywhere, the fit is degenerate by design. That is correct behavior, not a bug.

What it gives you in return is a dataset where every label carries its own uncertainty, an annotator pool you understand, and a list of the places your codebook is broken.

Further reading

• Psychometrics engine — full configuration reference
• Truth Serum — the other way to score quality with no gold labels
• How many annotators do you need?
• Aggregating crowd labels — Dawid–Skene and friends
• Inter-annotator agreement