Quality Control Without Gold Labels

You can get a quality signal with no gold items at all by asking annotators to predict what others will say (peer-prediction scoring) or whether a label survives a small edit (counterfactual probing). Potato ships both, as Truth Serum and Boundary Lab, and neither one requires an LLM.

The standard way to check annotation quality is to plant items you already know the answer to and see who gets them wrong. Gold standards and attention checks are the default answer to "how do I know my annotators are paying attention," and they are often the right tool. They also have three real costs. You have to build them, which means expert time spent labeling items you already understand. They only cover the failure modes you anticipated. On any long-running task, experienced annotators eventually learn to spot them, and a gold item that gets recognized stops measuring anything. Both techniques below sidestep all three costs.

Peer-prediction scoring with Truth Serum

Peer-prediction scoring asks each annotator to predict how others will label an item. Truth Serum adds that prediction to one annotation scheme, so after an annotator picks a label, it asks one more question:

You chose Sarcastic. What percentage of other annotators will choose the same label as you?

The annotator answers with one slider and one click, and that extra number does a lot of work.

Truth Serum's prediction card

The surprisingly popular principle

The method is the surprisingly popular principle of Prelec, Seung and McCoy (2017), which builds on Prelec's 2004 Bayesian truth serum. The principle holds that the best estimate of the truth is the answer that is more popular than people predicted it would be, rather than the most popular answer.

Someone who holds a correct minority view usually knows they are in a minority. They can see why most people would answer differently, and they say so when you ask them to predict. Someone holding a common misconception, by contrast, tends to assume everyone agrees with them. So when an answer outperforms its own predicted popularity, the gap is evidence that the people choosing it know something the crowd does not. Majority vote fails hardest where a confident crowd is wrong and an informed minority is right, which is the case surprisingly-popular scoring is built for.

Turn on Truth Serum for a scheme:

truth_serum:
  enabled: true
  schema: sarcasm
  min_annotators: 5

Truth Serum outputs

Truth Serum produces three outputs, and none of them needed an answer key.

A "where the crowd is likely wrong" queue. The queue holds every item whose surprisingly-popular label differs from its majority label. Those items are the ones worth an expert's attention, and there are usually far fewer of them than you feared.

Calibration scores per annotator. Each score is how far a person's predicted agreement sits from the agreement they actually got. Someone who consistently predicts that 90% of others will agree with them, and consistently gets 55%, is telling you something about their self-awareness, and you learned it without planting anything.

SP-alignment. SP-alignment is how often each annotator's label matches the surprisingly-popular verdict, which is a rough proxy for being informed rather than merely agreeable.

Before you cite the method, note that Potato implements the simplified own-answer-prediction variant, in which each annotator predicts the popularity of the label they picked rather than a full distribution over all labels. Also, Potato computes no verdict for an item until it has min_annotators predictions, which defaults to 3 (values below 2 are raised to 2). The config above sets 5 because verdicts from a few predictions are noisy, and the Truth Serum documentation recommends five or more.

Counterfactual probing with Boundary Lab

Truth Serum finds items the crowd got wrong. Boundary Lab finds the places your codebook is ambiguous, and the annotators who are not really reading. The moment a label is committed, Potato shows a minimal edit of the text and asks whether the label still holds:

You said Polite. Would that survive this edit?

A Boundary Lab probe

Answering takes one click, and because the probe asks, ordinary annotation now produces three things a plain label export cannot.

Contrast sets as a by-product

Every answered probe is a labeled (original, counterfactual) pair, and those pairs form a contrast set. Contrast sets were proposed for evaluation, and across 10 NLP datasets models scored significantly lower on them than on the original test sets, by up to 25% in some cases. Pairs of the same kind also work as training data. In the study that introduced counterfactually augmented data, classifiers trained on original and counterfactually revised documents together were less sensitive to spurious features than classifiers trained on either set alone. Contrast sets are normally a separate and expensive annotation effort, and Boundary Lab produces them from the labeling you were doing anyway.

Configure Boundary Lab for a scheme:

boundary_probing:
  enabled: true
  schema: politeness
  probes_per_item: 3
  include_invariance: true
  sources: [precomputed, llm, rules]

The sources list is a fallback chain. You can ship curated counterfactuals with your data, generate them with an LLM, or let the rule-based tier produce them from lexical transforms such as negation toggles, intensifier swaps, and politeness markers. The rule-based tier is why Boundary Lab works with no LLM configured at all, and why the feature degrades gracefully rather than breaking when nobody set an API key.

Invariance probes as hidden quality control

Some probes are invariance probes, paraphrases that preserve meaning, so "Send me the slides before the meeting" becomes "Before the meeting, send me the slides." A consistent annotator never flips on those. An annotator who does is either not reading carefully or does not have a stable idea of what the label means. Either way you want to know, and you found out without planting a single fake item, because the probe is part of the ordinary work. The dashboard flags annotators below a 60% hold rate on invariance probes.

Boundary rationales and label sensitivity

When a label does flip, the annotator says what crossed the line, for example "please softens the command." Those rationales accumulate into a map of exactly where your guideline is underspecified, which is the input to a better guideline.

The dashboard also reports boundary sensitivity by label, the share of the minimal edits aimed at each label that actually flipped it. A label with a 90% flip rate is fragile and probably needs a clearer definition, and a label at 10% is stable.

Choosing between Truth Serum and Boundary Lab

The two find different things, and you can run both. The table compares what each one finds, what it costs the annotator, what it produces, and whether it needs an LLM.

If your worry is "the consensus label might be wrong on the hard cases," start with Truth Serum. If it is "I don't think my codebook is precise enough, and I'm not sure everyone is reading," start with Boundary Lab.

Neither replaces psychometrics, which tells you how confident to be in each label and which annotators to weight. Running all three measures quality continuously rather than by spot checks, and none of that measurement depends on an answer key you had to build first.

Further reading

• Truth Serum and Boundary Lab, for the full configuration
• Labels with error bars, on IRT posteriors and the codebook-bug detector
• Gold standards and attention checks, on the classical approach and when it is still the right one
• Quality control
• Writing annotation guidelines

References

Dražen Prelec, H. Sebastian Seung, and John McCoy (2017). A solution to the single-question crowd wisdom problem. Nature. https://doi.org/10.1038/nature21054

Dražen Prelec (2004). A Bayesian Truth Serum for Subjective Data. Science. https://doi.org/10.1126/science.1102081

Matt Gardner et al. (2020). Evaluating Models' Local Decision Boundaries via Contrast Sets. Findings of the Association for Computational Linguistics: EMNLP 2020. https://aclanthology.org/2020.findings-emnlp.117/

Divyansh Kaushik, Eduard Hovy, and Zachary C. Lipton (2020). Learning the Difference that Makes a Difference with Counterfactually-Augmented Data. International Conference on Learning Representations. https://arxiv.org/abs/1909.12434