# Truth Serum

Source: https://www.potatoannotator.com/docs/features/truth-serum

*New in v2.7.0*

Majority vote fails exactly where annotation is hard: when a confident crowd is wrong and an informed minority is right. Truth Serum adds one micro-question after each label.

> *You chose **Sarcastic**. What percentage of other annotators will choose the same label as you?*

![Peer-prediction scoring — "what percentage of other annotators will choose the same label?"](/images/docs/truth-serum.png)

One slider, one click. Those predictions power the **surprisingly popular** principle ([Prelec 2004](https://www.science.org/doi/10.1126/science.1102081); [Prelec, Seung & McCoy 2017](https://www.nature.com/articles/nature21054), *Nature*): the label whose actual popularity most exceeds its predicted popularity is the best available estimate of the truth, with **no gold labels required**.

The intuition is that people who hold a correct minority view usually *know* they are in the minority, so their answer ends up more popular than anyone predicted.

To our knowledge, Potato is the first annotation tool to ship peer-prediction scoring.

## What you get

1. **Item verdicts that beat majority vote on hard items.** The dashboard's "Where the crowd is likely wrong" queue lists every instance whose surprisingly-popular label differs from its majority label. Review these first, or route them to [adjudication](/docs/guides/adjudication-and-disagreement).
2. **Annotator calibration scores.** How far each annotator's predicted agreement sits from the agreement they actually got from peers. Overconfident and miscalibrated annotators surface without any gold questions.
3. **SP-alignment.** How often each annotator's label matches the surprisingly-popular verdict — a proxy for being informed rather than merely agreeable.

## Configuration

```yaml
truth_serum:
  enabled: true
  schema: sarcasm            # scheme to collect predictions for (default: first radio)
  min_annotators: 3          # predictions per item before a verdict is computed
  # question: "What percentage of other annotators will choose the same label as you?"
```

| Option | Default | Description |
|--------|---------|-------------|
| `enabled` | `false` | Master switch. |
| `schema` | first radio scheme | Which scheme's labels get popularity predictions. |
| `question` | see above | Prompt shown above the slider. |
| `min_annotators` | `3` | Minimum predictions per item before a verdict is computed (floor: 2). |

## Method notes

Read these before citing Truth Serum in a paper.

- This is the **simplified own-answer-prediction variant**. Each annotator predicts the popularity of the label they chose, rather than a full distribution over all labels. A label's predicted popularity is the mean prediction of its supporters, its surprise is `actual % − predicted %`, and the SP verdict is the most-surprising voted label.
- Verdicts are only computed with `min_annotators` or more predictions. Small-N verdicts are noisy; 3 is a floor rather than a recommendation, and 5 or more is better.
- Calibration error compares each prediction against agreement among the *other* annotators on that item, excluding the annotator themselves.
- Annotators can revise. The latest (label, prediction) pair per instance wins.

## Data and API

Predictions persist to `{output_annotation_dir}/truth_serum/predictions.jsonl` (append-only; the latest record per annotator and instance wins).

| Endpoint | Method | Auth | Purpose |
|----------|--------|------|---------|
| `/truth_serum/api/predict` | POST | session | Record label + predicted % |
| `/truth_serum/api/mine` | GET | session | This annotator's prediction (widget restore) |
| `/truth_serum/dashboard` | GET | admin | Verdicts + calibration dashboard |
| `/truth_serum/api/stats` | GET | admin | Dashboard aggregates |
| `/truth_serum/api/export` | GET | admin | Full JSON export (verdicts + raw predictions) |

## Further reading

- [Psychometrics](/docs/features/psychometrics) — labels with posteriors and confidence intervals
- [Boundary Lab](/docs/features/boundary-lab) — quality control from counterfactual probes
- [MACE](/docs/features/mace) — competence estimation, which composes with Truth Serum's calibration view
- [Aggregating crowd labels](/docs/guides/aggregating-crowd-labels)
- [Source documentation](https://github.com/davidjurgens/potato/blob/main/docs/advanced/truth_serum.md)