Psychometrics Engine

New in v2.7.0

Potato can treat your annotation study as what it actually is: a measurement instrument. The psychometrics layer fits an item response theory (IRT) model live, as annotations stream in, jointly estimating the true label of every item as a probability, every annotator's ability with a standard error, every item's difficulty, and a per-item discrimination diagnostic that flags likely codebook bugs.

No gold labels are needed, and no LLM is involved. The model — a multiclass generalization of GLAD (Whitehill et al. 2009) — bootstraps everything from the pattern of agreement itself, the same way a standardized test learns which questions are hard without anyone declaring it. Fits are deterministic and take milliseconds at annotation-study scale.

Configuration

# Adaptive routing is opt-in via the standard assignment strategy key.
# Omit this line to keep your existing strategy and use psychometrics
# as a pure analytics layer.
assignment_strategy: psychometric
 
# Give items a redundancy target so early-stopped items translate into
# concrete saved judgments on the dashboard.
num_annotators_per_item: 4
 
psychometrics:
  enabled: true
  schema: sarcasm            # scheme to model; default: first radio/likert scheme
  refit_interval: 5          # refit after this many new labels (fits are ~ms)
  min_observations: 20       # cold-start gate before adaptive routing engages
  min_annotators_per_item: 2 # never early-stop an item below this many annotators
  confidence_threshold: 0.95 # posterior at which an item counts as resolved
  cost_per_judgment: 0.08    # optional: expresses savings in currency
  discrimination_flag_threshold: -0.2  # codebook-bug flag sensitivity

Supported schemes are single-choice categorical: radio and likert (likert points are treated as nominal categories). Multi-select schemes are not modeled.

Labels with error bars

Instead of sarcastic (2 of 3 votes), the export says sarcastic, p = 0.94 [0.88 – 0.97]. The probability is the model posterior, which weighs who voted rather than only how many, and the interval is a ±1 standard-error sensitivity band over the ability estimates.

Downstream, you can train on soft labels, filter evaluation sets to high-confidence items, or treat low-probability items as genuinely ambiguous rather than as noise. See soft label annotation for the complementary approach of collecting distributions directly from annotators.

Annotator ability, honestly presented

Ability θ is estimated from agreement patterns. A value of 1.0 is the prior for a new annotator, 0 means their labels carry no information, and negative values mean systematically wrong. Every estimate ships with a standard error: an annotator with 12 labels has a wide whisker, and the dashboard says so.

Do not make personnel decisions from a wide whisker. The standard errors are approximate (Fisher information at the mode) and mildly optimistic by construction.

The codebook-bug detector

Item difficulty is estimated jointly with ability, but the more useful signal is discrimination: the correlation between annotator ability and answer correctness on each item.

When discrimination is strongly negative — your best annotators disagreeing with the crowd consensus — the guideline is usually wrong or ambiguous for that item, not the annotators. The dashboard surfaces these under "Likely codebook bugs." Fix the instructions, then revisit. Multiplayer Rooms are where teams typically go to resolve them.

Adaptive routing

With assignment_strategy: psychometric, when an annotator asks for work the remaining items are ranked by the exact one-step expected information gain of that annotator labeling that item. High-uncertainty items go to high-ability annotators first, and items whose posterior already exceeds confidence_threshold (with at least min_annotators_per_item annotators) stop consuming budget. Compared with a fixed N-per-item design, the saved judgments are counted on the dashboard, and priced if you set cost_per_judgment.

Two operational notes:

• Cold start. Until min_observations labels exist, assignment falls back to random — the model needs overlapping annotators before it can separate ability from difficulty. The dashboard shows a warming-up meter during this phase.
• Batching. Assignment happens when a user's queue tops up. Ranking is freshest when queues are short; very large per-user batches dilute adaptivity.

Power analysis before you spend

Every annotation project guesses at "how many annotators per item?" The study designer answers it with a seeded Monte Carlo simulation:

python -m potato.psychometrics.design --items 500 --accuracy 0.75 \
    --classes 3 --target-ci 0.10 --cost 0.08

ann/item   alpha     95% interval   width  majority acc  judgments       cost
       2   0.392 [ 0.330,  0.439]   0.109         0.751       1000      80.00
       3   0.388 [ 0.338,  0.431]   0.093         0.864       1500     120.00  <- recommended

The recommendation is the smallest redundancy whose 95% interval on Krippendorff's α is narrower than --target-ci — the cheapest design that still yields a defensibly precise agreement estimate. The --accuracy input is best measured with a small pilot. The same analysis is available on the dashboard and as an admin API (GET /psychometrics/api/design).

You can also run this interactively in the browser with the annotation power calculator.

Endpoints

All endpoints require admin access (RBAC VIEW_ADMIN_DASHBOARD; debug mode and the shared admin API key pass).

How this relates to MACE

Potato also ships MACE, which estimates annotator competence and predicted labels as post-hoc analytics. The two are cousins from the same literature with different jobs.

Use MACE for a quick competence readout on any categorical scheme (it also covers multiselect). Use psychometrics when you want difficulty-aware measurement that acts on the study while it runs.

Troubleshooting

• Dashboard says "warming up" forever — the model needs at least two distinct labels and overlapping annotators. With one annotator or unanimous labels the fit is degenerate by design. Add annotators or lower min_observations.
• assignment_strategy: psychometric but routing looks random — that is the cold-start fallback. Check the warming-up meter and min_observations.
• Abilities all near 1.0 with big whiskers — not enough overlap yet. Abilities separate as annotators accumulate shared items.
• 404 on /psychometrics/... — the psychometrics.enabled: true block is missing from the config.

Further reading

• Truth Serum — peer-prediction scoring, the other no-gold-label quality signal
• Task assignment strategies
• Inter-annotator agreement
• How many annotators do you need?
• Source documentation