Certifying LLM Labels with a Blind Human Sample

To decide which LLM labels to keep without checking every one, have people label a blind random sample, choose a confidence threshold on half of that sample, and test on the other half that the error rate among labels above the threshold is below your target. For prevalence figures, correct the model's estimate with its errors on the same sample, which gives an interval that stays valid however good the model is.

A common way to check LLM labels is to read a few dozen outputs and, if they look right, ship the dataset with a sentence such as "an LLM labeled the data and we spot-checked it". That sentence does not say how often the labels are wrong, and a reviewer cannot tell whether the check would have caught a 15% error rate. A check that supports a number answers three questions: which machine labels to keep, how far off a rate computed from machine labels is, and whether a model can stand in for a person.

Bias in spot checks and reviewed pre-labels

Two habits make a machine label look better than it is. The first is checking the items the model was least sure about. Checking those items finds errors fast, but the checked items are not a sample of the dataset, so the error rate on them estimates nothing about the rest. The second is measuring agreement on labels people gave while the model's answer was on screen. A suggestion pulls the answer toward it, so agreement measured on reviewed pre-labels runs higher than agreement measured blind. LLM pre-annotation covers that automation bias in more detail.

A check that supports a number therefore needs three properties. The checked items are drawn at random with known inclusion probabilities. The people labeling them cannot see the model's answer. The labels that pick a rule are different from the labels that test it.

Selective prediction and selective risk

Selective prediction means a model labels only the items where its confidence clears a threshold and abstains on the rest, which go to people. The quantity to control is the selective risk, the error rate among the labels the model keeps. For example, a threshold that keeps 70% of items at a 4% error rate can be more useful than a model that labels everything at 12%.

Choosing the threshold and testing it on the same labels overstates how well it works, so the human sample is split. One half, the dev split, picks the threshold. The other half, the cert split, tests it with a one-sided binomial test of the claim that the error among accepted labels is above the target. If that claim is rejected, the threshold is certified at the stated confidence. Learn-then-Test (Angelopoulos et al., 2025) treats choosing a threshold as a multiple-testing problem. One of its procedures tests candidate thresholds in a fixed order and stops at the first that fails, which lets it try several thresholds without losing the guarantee.

These tests need larger samples than most spot checks. Suppose the target is at most 10% wrong with 90% confidence, and this is the first test of a single model with no per-class split, which Potato runs at a level of about 0.06 (the next section explains why). With 100 accepted labels in the cert split, the test passes only if at most 5 are wrong. A 5% target needs at most 1 wrong label in 100, or at most 4 in 200. A model whose true error rate is 7% passes that first test about 29% of the time with 100 accepted cert labels, and about 56% of the time with 300.

Repeated tests and the error budget

People keep labeling after the first test, and re-running the test until it passes would turn a 10% chance of a wrong certificate into something much larger. One fix is to give each test a shrinking share of the error budget. Potato gives test number k the share 6 / (pi^2 k^2) of the allowed error, so the first test gets 61%, the second 15%, and the shares sum to the whole budget over every possible test. Trying a new prompt or a new model counts as a new test, because it is one more chance to pass by luck.

Prevalence with prediction-powered inference

Many studies need a rate more than a dataset, such as the share of posts that are toxic or the share of agent runs that succeed. Counting machine labels gives a biased rate whenever the model's errors lean one way. Counting only the human sample gives an unbiased rate with a wide interval.

Prediction-powered inference (Angelopoulos et al., 2023) combines the two. It computes the rate from the machine labels on every item, then corrects it by the model's average error on the human-labeled sample. The interval is valid whatever the model's quality. PPI++ tunes how much weight the machine labels get, so a good model makes the interval narrower than people alone could and a poor one makes it no wider. When the human sample was drawn with unequal probabilities, a Horvitz-Thompson difference estimator plays the same role.

In the agent-trace example that ships with Potato, an LLM judge rated 288 synthetic agent traces written for the example, each with a reference outcome, and said 76% succeeded. The prediction-powered interval from the same judge and a human sample was 0.66 [0.57, 0.76], and the reference rate in the data was 0.60. The judge was right on 80% of individual traces and still overstated the rate by 16 points, because its errors leaned toward "success".

The alternative annotator test

A fixed accuracy target can be the wrong question when people disagree. If two careful annotators agree on 85% of items, no labeler can be shown to be 95% "right", because the reference is noisy too. The alternative annotator test (Calderon et al., 2025) asks a relative question instead. Leaving each person out in turn, it tests whether the model agrees with the remaining people at least as often as the left-out person does, within a margin. It needs items labeled by several people, and Potato's implementation waits for at least three annotators with 30 items each. LLM judge agreement covers the kappa-based version of this comparison and how to check a judge's calibration.

Contracts in Potato's labeling engine

From version 2.11, Potato's labeling engine runs these tests as contracts. It draws the blind random sample, serves it to annotators with every machine answer hidden, splits it into dev and cert, and certifies as labels arrive. This config keeps an LLM's sentiment labels where at most 10% are wrong with 90% confidence, and reports each label's prevalence with a 90% interval:

yaml

assignment_strategy: engine
 
labeling_engine:
  enabled: true
  sources:
    - id: llm
      kind: llm
      endpoint_type: vllm
      model: google/gemma-4-12B-it-qat-w4a16-ct
      base_url: http://localhost:8001/v1
      confidence: {method: logprob}
  contracts:
    - id: accuracy
      kind: selective_risk
      schema: sentiment
      error: 0.1
      confidence: 0.9
    - id: prevalence
      kind: ppi
      schema: sentiment
      confidence: 0.9
  budgets:
    human_labels: 400

A confidence of 0.9 is Potato's default for every contract. The source reads confidence from token log probabilities because a threshold needs a confidence that separates right answers from wrong ones, and a confidence the model states in words is the least reliable of the methods Potato offers. Potato's text example runs 481 reviews with Laya first and this LLM as the last tier. In that run the 10% contract was met on 165 human labels, and the engine accepted 316 machine labels, all from the LLM because Laya's threshold was never fixed. Their measured error against the reference labels was 7.6%. potato label certificate config.yaml prints what was tested, on how many labels, and with which model and prompt versions.

Every one of these guarantees is about agreement with the people who labeled the sample. If they are careless or poorly trained, a met contract certifies agreement with careless people, so check the annotators with overlap and quality control before trusting a certificate.

A confidence that never varies gives a threshold nothing to separate. The judge in the agent-trace example gave a confidence of 1.0 to every trace, so no threshold could pass, and Potato marks such a contract infeasible instead of accepting everything. The prevalence interval still worked on the same data, because it does not depend on confidence.

On subjective tasks, where people disagree for reasons that belong in the data, a single right answer per item may not exist. Compare the model's label distribution with how people split instead, which Potato's distributional contract reports against a split-half noise ceiling. Run potato label status config.yaml after the first pass to see how many more human labels each contract needs before it can pass.

References

Angelopoulos, A. N., Bates, S., Candès, E. J., et al. (2025). Learn then test: Calibrating predictive algorithms to achieve risk control. The Annals of Applied Statistics. https://doi.org/10.1214/24-AOAS1998
Angelopoulos, A. N., Bates, S., Fannjiang, C., et al. (2023). Prediction-powered inference. Science, 382(6671). https://doi.org/10.1126/science.adi6000
Angelopoulos, A. N., Duchi, J. C., and Zrnic, T. (2023). PPI++: Efficient prediction-powered inference. arXiv. https://arxiv.org/abs/2311.01453
Calderon, N., Reichart, R., and Dror, R. (2025). The alternative annotator test for LLM-as-a-judge: How to statistically justify replacing human annotators with LLMs. Proceedings of ACL 2025. https://aclanthology.org/2025.acl-long.782/