Truth Serum

New in v2.7.0

Majority vote fails exactly where annotation is hard: when a confident crowd is wrong and an informed minority is right. Truth Serum adds one micro-question after each label.

You chose Sarcastic. What percentage of other annotators will choose the same label as you?

One slider, one click. Those predictions power the surprisingly popular principle (Prelec 2004; Prelec, Seung & McCoy 2017, Nature): the label whose actual popularity most exceeds its predicted popularity is the best available estimate of the truth, with no gold labels required.

The intuition is that people who hold a correct minority view usually know they are in the minority, so their answer ends up more popular than anyone predicted.

To our knowledge, Potato is the first annotation tool to ship peer-prediction scoring.

What you get

1. Item verdicts that beat majority vote on hard items. The dashboard's "Where the crowd is likely wrong" queue lists every instance whose surprisingly-popular label differs from its majority label. Review these first, or route them to adjudication.
2. Annotator calibration scores. How far each annotator's predicted agreement sits from the agreement they actually got from peers. Overconfident and miscalibrated annotators surface without any gold questions.
3. SP-alignment. How often each annotator's label matches the surprisingly-popular verdict — a proxy for being informed rather than merely agreeable.

Configuration

truth_serum:
  enabled: true
  schema: sarcasm            # scheme to collect predictions for (default: first radio)
  min_annotators: 3          # predictions per item before a verdict is computed
  # question: "What percentage of other annotators will choose the same label as you?"

Method notes

Read these before citing Truth Serum in a paper.

• This is the simplified own-answer-prediction variant. Each annotator predicts the popularity of the label they chose, rather than a full distribution over all labels. A label's predicted popularity is the mean prediction of its supporters, its surprise is actual % − predicted %, and the SP verdict is the most-surprising voted label.
• Verdicts are only computed with min_annotators or more predictions. Small-N verdicts are noisy; 3 is a floor rather than a recommendation, and 5 or more is better.
• Calibration error compares each prediction against agreement among the other annotators on that item, excluding the annotator themselves.
• Annotators can revise. The latest (label, prediction) pair per instance wins.

Data and API

Predictions persist to {output_annotation_dir}/truth_serum/predictions.jsonl (append-only; the latest record per annotator and instance wins).

Further reading

• Psychometrics — labels with posteriors and confidence intervals
• Boundary Lab — quality control from counterfactual probes
• MACE — competence estimation, which composes with Truth Serum's calibration view
• Aggregating crowd labels
• Source documentation