The standard way to check annotation quality is to plant items you already know the answer to and see who gets them wrong. It works, but it is expensive to build, it only tests the cases you thought to plant, and experienced annotators learn to spot the plants. Two techniques get you a quality signal with no gold items at all: peer-prediction scoring, which asks annotators to predict what others will say, and counterfactual probing, which asks whether a label survives a small edit. Potato ships both, as Truth Serum and Boundary Lab.

Gold standards and attention checks are the default answer to "how do I know my annotators are paying attention," and they earn their place. But they have three real costs.

You have to build them, which means expert time spent labeling items you already understand. They only cover the failure modes you anticipated. And on any long-running task, annotators eventually recognize them. A gold item that gets reused is a gold item that stops measuring anything.

Both techniques below sidestep all three, and neither one requires an LLM.

Peer prediction: ask what others will say

After an annotator picks a label, Truth Serum asks one more thing:

You chose Sarcastic. What percentage of other annotators will choose the same label as you?

One slider. One click. That single extra number does a surprising amount of work.

Peer-prediction scoring: "what percentage of other annotators will choose the same label?" Truth Serum's prediction card

Why the prediction is informative

The method is the surprisingly popular principle (Prelec, Seung & McCoy 2017, Nature; building on Prelec's Bayesian truth serum, 2004). The claim is that the best estimate of the truth is not the most popular answer, but the answer that is more popular than people predicted it would be.

The reasoning goes like this. Someone who holds a correct minority view usually knows they are in a minority. They can see why most people would answer differently, and they say so when you ask them to predict. Someone holding a common misconception, by contrast, tends to assume everyone agrees with them. So when an answer outperforms its own predicted popularity, that gap is evidence that the people choosing it know something the crowd does not.

Majority vote fails hardest exactly where a confident crowd is wrong and an informed minority is right. That is the case surprisingly-popular scoring is built for.

yaml

truth_serum:
  enabled: true
  schema: sarcasm
  min_annotators: 5

To our knowledge Potato is the first annotation tool to ship peer-prediction scoring.

What you get out of it

Three things, none of which needed an answer key.

A "where the crowd is likely wrong" queue. Every item whose surprisingly-popular label differs from its majority label. These are the items worth an expert's attention, and there are usually far fewer of them than you feared.

Calibration scores per annotator. How far each person's predicted agreement sits from the agreement they actually got. Someone who consistently predicts that 90% of others will agree with them, and consistently gets 55%, is telling you something about their self-awareness. You learned it without planting anything.

SP-alignment. How often each annotator's label matches the surprisingly-popular verdict, which is a rough proxy for being informed rather than merely agreeable.

A caveat before you cite it: this is the simplified own-answer-prediction variant, where each annotator predicts the popularity of the label they picked rather than a full distribution over all labels. And min_annotators: 3 is a floor, not a recommendation. Five or more is where the verdicts stop being noisy.

Counterfactual probes: ask whether the label survives an edit

Truth Serum finds items the crowd got wrong. Boundary Lab finds something different: the places your codebook is ambiguous, and the annotators who are not really reading.

The moment a label is committed, Potato shows a minimal edit of the text and asks whether the label still holds.

You said Polite. Would that survive this edit?

After each label, a counterfactual probe over a colored diff: "would that still hold?" A Boundary Lab probe

Answering is one click. But because you asked, ordinary annotation now produces three things a plain label export cannot.

Contrast sets, as a by-product

Every answered probe is a labeled (original, counterfactual) pair. That is a contrast set: counterfactually-augmented data of the kind shown to improve model robustness (Gardner et al. 2020, Evaluating Models' Local Decision Boundaries via Contrast Sets; Kaushik et al. 2020).

Contrast sets are normally a separate and expensive annotation effort. Here they fall out of the labeling you were doing anyway.

yaml

boundary_probing:
  enabled: true
  schema: politeness
  probes_per_item: 3
  include_invariance: true
  sources: [precomputed, llm, rules]

The sources list is a fallback chain. You can ship curated counterfactuals with your data, generate them with an LLM, or let the rule-based tier produce them from lexical transforms such as negation toggles, intensifier swaps, and politeness markers. That last tier is why Boundary Lab works with no LLM configured at all, and it is the reason the feature degrades gracefully rather than breaking when nobody set an API key.

Invisible quality control

Some probes are invariance probes: paraphrases that preserve meaning. "Send me the slides before the meeting" becomes "Before the meeting, send me the slides."

A consistent annotator never flips on those. An annotator who does is either not reading carefully or does not have a stable idea of what the label means. Either way you want to know, and you found out without planting a single fake item. The probe is indistinguishable from the ordinary work, because it is ordinary work.

The dashboard flags annotators below a 60% hold rate on invariance probes.

Boundary rationales

When a label does flip, the annotator says what crossed the line: "please softens the command." Those rationales accumulate into a map of exactly where your guideline is underspecified, which is the input to a better guideline.

The dashboard also reports boundary sensitivity by label: of the minimal edits aimed at each label, what share actually flipped it. A label with a 90% flip rate lives on a knife's edge and probably needs a clearer definition. A label at 10% is robust.

Which one should you use?

They answer different questions, and they compose.

	Truth Serum	Boundary Lab
Finds	Items the crowd probably got wrong	Ambiguous guidelines; inattentive annotators
Costs the annotator	One slider per item	One click per probe
Produces	Better verdicts, calibration scores	Contrast sets, boundary rationales
Needs an LLM	No	No (rule-based tier)

If your worry is "the consensus label might be wrong on the hard cases," start with Truth Serum. If it is "I don't think my codebook is precise enough, and I'm not sure everyone is reading," start with Boundary Lab.

Neither replaces psychometrics, which tells you how confident to be in each label and which annotators to weight. The three of them together give you a study where quality is measured continuously rather than spot-checked, and where none of the measurement depends on an answer key you had to build first.

Quality Control Without Gold Labels