# Psychometrics Engine

Source: https://www.potatoannotator.com/docs/features/psychometrics

*New in v2.7.0*

Potato can treat your annotation study as what it actually is: a measurement instrument. The psychometrics layer fits an item response theory (IRT) model **live**, as annotations stream in, jointly estimating the true label of every item as a probability, every annotator's ability with a standard error, every item's difficulty, and a per-item discrimination diagnostic that flags likely codebook bugs.

No gold labels are needed, and no LLM is involved. The model — a multiclass generalization of GLAD ([Whitehill et al. 2009](https://papers.nips.cc/paper_files/paper/2009/hash/f899139df5e1059396431415e770c6dd-Abstract.html)) — bootstraps everything from the pattern of agreement itself, the same way a standardized test learns which questions are hard without anyone declaring it. Fits are deterministic and take milliseconds at annotation-study scale.

![Annotator ability estimated live, with confidence intervals — every label gets a posterior, not a bare vote.](/images/docs/psychometrics.png)

## Configuration

```yaml
# Adaptive routing is opt-in via the standard assignment strategy key.
# Omit this line to keep your existing strategy and use psychometrics
# as a pure analytics layer.
assignment_strategy: psychometric

# Give items a redundancy target so early-stopped items translate into
# concrete saved judgments on the dashboard.
num_annotators_per_item: 4

psychometrics:
  enabled: true
  schema: sarcasm            # scheme to model; default: first radio/likert scheme
  refit_interval: 5          # refit after this many new labels (fits are ~ms)
  min_observations: 20       # cold-start gate before adaptive routing engages
  min_annotators_per_item: 2 # never early-stop an item below this many annotators
  confidence_threshold: 0.95 # posterior at which an item counts as resolved
  cost_per_judgment: 0.08    # optional: expresses savings in currency
  discrimination_flag_threshold: -0.2  # codebook-bug flag sensitivity
```

Supported schemes are single-choice categorical: `radio` and `likert` (likert points are treated as nominal categories). Multi-select schemes are not modeled.

## Labels with error bars

Instead of `sarcastic (2 of 3 votes)`, the export says `sarcastic, p = 0.94 [0.88 – 0.97]`. The probability is the model posterior, which weighs *who* voted rather than only how many, and the interval is a ±1 standard-error sensitivity band over the ability estimates.

Downstream, you can train on soft labels, filter evaluation sets to high-confidence items, or treat low-probability items as genuinely ambiguous rather than as noise. See [soft label annotation](/docs/annotation-types/soft-label) for the complementary approach of collecting distributions directly from annotators.

## Annotator ability, honestly presented

Ability θ is estimated from agreement patterns. A value of 1.0 is the prior for a new annotator, 0 means their labels carry no information, and negative values mean systematically wrong. Every estimate ships with a standard error: an annotator with 12 labels has a wide whisker, and the dashboard says so.

Do not make personnel decisions from a wide whisker. The standard errors are approximate (Fisher information at the mode) and mildly optimistic by construction.

## The codebook-bug detector

Item difficulty is estimated jointly with ability, but the more useful signal is **discrimination**: the correlation between annotator ability and answer correctness on each item.

When discrimination is strongly negative — your best annotators disagreeing with the crowd consensus — the guideline is usually wrong or ambiguous for that item, not the annotators. The dashboard surfaces these under "Likely codebook bugs." Fix the instructions, then revisit. [Multiplayer Rooms](/docs/features/multiplayer-rooms) are where teams typically go to resolve them.

## Adaptive routing

With `assignment_strategy: psychometric`, when an annotator asks for work the remaining items are ranked by the exact one-step expected information gain of *that annotator* labeling *that item*. High-uncertainty items go to high-ability annotators first, and items whose posterior already exceeds `confidence_threshold` (with at least `min_annotators_per_item` annotators) stop consuming budget. Compared with a fixed N-per-item design, the saved judgments are counted on the dashboard, and priced if you set `cost_per_judgment`.

Two operational notes:

- **Cold start.** Until `min_observations` labels exist, assignment falls back to random — the model needs overlapping annotators before it can separate ability from difficulty. The dashboard shows a warming-up meter during this phase.
- **Batching.** Assignment happens when a user's queue tops up. Ranking is freshest when queues are short; very large per-user batches dilute adaptivity.

## Power analysis before you spend

Every annotation project guesses at "how many annotators per item?" The study designer answers it with a seeded Monte Carlo simulation:

```bash
python -m potato.psychometrics.design --items 500 --accuracy 0.75 \
    --classes 3 --target-ci 0.10 --cost 0.08
```

```
ann/item   alpha     95% interval   width  majority acc  judgments       cost
       2   0.392 [ 0.330,  0.439]   0.109         0.751       1000      80.00
       3   0.388 [ 0.338,  0.431]   0.093         0.864       1500     120.00  <- recommended
```

The recommendation is the smallest redundancy whose 95% interval on Krippendorff's α is narrower than `--target-ci` — the cheapest design that still yields a defensibly precise agreement estimate. The `--accuracy` input is best measured with a small pilot. The same analysis is available on the dashboard and as an admin API (`GET /psychometrics/api/design`).

You can also run this interactively in the browser with the [annotation power calculator](/tools/annotation-power).

## Endpoints

All endpoints require admin access (RBAC `VIEW_ADMIN_DASHBOARD`; debug mode and the shared admin API key pass).

| Endpoint | Purpose |
|----------|---------|
| `GET /psychometrics/dashboard` | Live dashboard |
| `GET /psychometrics/api/stats` | Dashboard aggregates (forces a fresh fit) |
| `GET /psychometrics/api/export` | Enriched export: posteriors, bands, abilities |
| `GET /psychometrics/api/design` | Power analysis |

## How this relates to MACE

Potato also ships [MACE](/docs/features/mace), which estimates annotator competence and predicted labels as post-hoc analytics. The two are cousins from the same literature with different jobs.

| | MACE | Psychometrics |
|---|------|---------------|
| Annotator model | knowing-vs-guessing competence | continuous ability with standard errors |
| Item model | none | difficulty + discrimination (codebook-bug flags) |
| When it runs | batch analytics after N annotations | live, in the assignment loop |
| Drives assignment | no | yes — information-gain routing + early stopping |
| Uncertainty on labels | entropy | posterior probability + sensitivity interval |
| Pre-study planning | no | Monte Carlo power analysis |

Use MACE for a quick competence readout on any categorical scheme (it also covers `multiselect`). Use psychometrics when you want difficulty-aware measurement that acts on the study while it runs.

## Troubleshooting

- **Dashboard says "warming up" forever** — the model needs at least two distinct labels and overlapping annotators. With one annotator or unanimous labels the fit is degenerate by design. Add annotators or lower `min_observations`.
- **`assignment_strategy: psychometric` but routing looks random** — that is the cold-start fallback. Check the warming-up meter and `min_observations`.
- **Abilities all near 1.0 with big whiskers** — not enough overlap yet. Abilities separate as annotators accumulate shared items.
- **404 on `/psychometrics/...`** — the `psychometrics.enabled: true` block is missing from the config.

## Further reading

- [Truth Serum](/docs/features/truth-serum) — peer-prediction scoring, the other no-gold-label quality signal
- [Task assignment strategies](/docs/features/task-assignment)
- [Inter-annotator agreement](/docs/guides/inter-annotator-agreement)
- [How many annotators do you need?](/docs/guides/how-many-annotators)
- [Source documentation](https://github.com/davidjurgens/potato/blob/main/docs/advanced/psychometrics.md)