Labeling Engine Contracts

A contract is a claim about the finished labels that the labeling engine tests against human labels on a random sample. Each contract names one annotation scheme (schema) and has a confidence, 0.9 by default, which is the probability that the claim holds. A confidence of 0.9 therefore allows a 10% chance that a met contract is wrong.

The six kinds make different claims and suit different studies:

None of these contracts says the labels are free of errors. selective_risk bounds the error rate among accepted labels, and each contract that tests the machines measures them against the people who labeled the sample.

Contract status

Every contract carries one of five statuses, shown on the admin page, by potato label status, and in the certificate:

A machine label is accepted only when every selective_risk and threshold_legacy contract on its scheme is met and accepts it. The contracts state what the labels must satisfy, so an unmet one blocks acceptance rather than being skipped, and the status, the admin page and the certificate name the contract that is not met. When no rule can be in force and nothing more is being sampled (a contract is infeasible, the calibration sample is finished, or nothing could be projected), the items still without a label go to people, within budgets.human_labels.

Dev and cert splits

The engine keeps the labels that choose a rule apart from the labels that test it. Calibration requests are drawn at random with a recorded inclusion probability, are always blind, and each one goes at random into one of two splits:

• dev (humans.dev_fraction, half by default) fits calibrators and chooses thresholds;
• cert tests the chosen rule, and nothing else reads it.

Looks

Every new test of a contract runs at a stricter level, so testing again as labels arrive does not inflate the confidence. Look k gets (1 - confidence) * 6 / (pi^2 k^2) of the error budget, which sums to the whole budget over every possible look, and the first look gets 61% of it.

• Looks are counted per contract across every cascade it has been tested under, and a renamed contract keeps the looks of its old id. Trying prompts or models until one passes costs a look each time. Going back to an earlier cascade is a new look too, and its earlier result is not restored.
• Re-testing the same data is not a new look. The same cert labels, the same predictions and the same rule chosen on dev give back the stored result. A test that could not be run, for example with no cert labels in a class, spends nothing.
• potato label certify tests new cert labels and new predictions at once, as do the REST, MCP and SDK equivalents. When only dev labels changed, it returns the stored result with a note that dev now picks another rule, and --retest tests that rule at the next look.
• The planner certifies on its own as labels arrive. A met selective_risk contract stays met for its cascade, and audits watch it. A contract that is not met is tested again once its labels have grown by a quarter, and by at least 10. ppi, distributional and no_worse_than_human results are estimates rather than pass/fail tests, so the planner re-estimates them at the same milestones and once more when the calibration sample is complete.
• An edited cert label is new data. When an annotator changes their answer, the edit costs no more than a new label does. Edited back to a state already tested, that test's result returns. If none of the stored result's tests sees more errors after the edit, the result and its look stand. Otherwise the planner waits for the labels to grow by a quarter from the first edit, so a burst of corrections becomes one test, and accepted labels stay accepted meanwhile with a note on the contract. When no more labels are coming, it tests at once, and an explicit certify tests at the next look.

Labels people gave after seeing a machine answer (verify requests) and labels given outside the engine's requests never certify or estimate anything. They can train a distilled model.

selective_risk

selective_risk bounds the share of accepted machine labels that are wrong. The error key is the largest tolerated error rate among accepted labels and is required:

- id: accuracy
  kind: selective_risk
  schema: sentiment
  error: 0.05          # required: largest tolerated error among accepted labels
  confidence: 0.9
  per_class: true      # test each predicted class separately
  tolerance: 0.5       # graded families only: loss at or below this counts as correct

The rule is a confidence threshold for each cascade tier, and for each predicted class with per_class. For every tier but the last, the threshold is chosen on dev and tested once on cert. The last tier tries thresholds from the dev choice toward more lenient ones, testing each in turn and stopping at the first that fails. That procedure is fixed-sequence Learn-then-Test (Angelopoulos et al., 2025). Each test is a one-sided binomial test that the error among accepted labels is above error, with the level split across tiers and classes. A tier that fails accepts nothing, and its items go to people.

What counts as "correct" depends on the scheme family. The loss for each family is in the table, and a label counts as correct when its loss is at or below tolerance:

The losses for regions, segments, 3D objects and traces are the ones the agreement report at /admin/iaa uses, so a machine is scored against a person the way two people are scored against each other. Free text and pairwise schemes have no family yet, and the certificate lists them as unsupported.

The test needs each tested group of cert items to be a uniform sample. A group whose items were sampled with unequal probabilities is not tested, and the result names it.

On the last tier, a threshold is chosen on dev only where dev's error is at most half the target and enough cert labels are expected to pass a first test. The projection of how many more labels are needed counts only thresholds that could be chosen that way. When cert already has the labels the projection asked for and the test still fails, the next projection uses the error rate that dev and cert show together, not dev's alone. When the lowest dev error is under the target but above half of it, nothing is tested and nothing is projected. The result says so and names the error, since more labels help only if the error turns out lower.

metrics.accepted_by_tier is the rule's coverage of the whole population, calibration items included, and is not a count of delivered machine labels. The certificate's "Machine labels accepted, by source" line counts those.

The trace loss compares only the steps both sides labeled, so a person who marked some steps is compared on those steps. Steps nobody labeled are not checked, and a person who labels step 0 alone and agrees with the judge there counts as agreeing with the whole trace label. Where every step must be checked, ask people to label every step.

ppi

ppi reports an interval for each label's prevalence, or for a mean score, corrected for the machine's errors:

- id: prevalence
  kind: ppi
  schema: sentiment
  estimand: prevalence   # or mean, for numeric scores
  by: language           # optional: an interval per value of this item field
  confidence: 0.95
  max_width: 0.05        # optional: met once every interval is at most this wide

Prediction-powered inference (Angelopoulos et al., 2023) corrects the machine's estimate with its error on the human-labeled sample. The interval is valid whatever the machine's quality. A good machine makes it narrower than people alone could, and a bad one makes it no wider. With equal inclusion probabilities the engine uses PPI++ with a tuned weight, and with unequal ones (stratified or uncertainty-driven sampling) it uses the Horvitz-Thompson difference estimator. The machine label it uses is the one the cascade would deliver. The by field is copied from the items when they are registered. Without max_width, the contract is met as soon as it has enough labels to report.

no_worse_than_human

no_worse_than_human asks whether the machine agrees with people at least as often as a person does:

- id: replaces-a-person
  kind: no_worse_than_human
  schema: verdict
  epsilon: 0.1

The contract runs the alternative annotator test (Calderon et al., 2025). Leaving each person out in turn, it tests whether the machine agrees with the remaining people at least as often as the left-out person does, up to epsilon. The machine passes when it wins against at least half of the people, with the false discovery rate across people controlled.

The test needs items labeled by several people. Set humans.overlap (the share of the sample labeled more than once) and humans.overlap_k (people per item, at least 3), and the test needs at least 3 annotators with 30 items each. The contract stays insufficient until it has them. Use this contract when a fixed accuracy target would mostly measure annotator noise. If two careful people agree 85% of the time, no labeler can be shown to be 95% "right".

budget_first

budget_first projects what a human-label budget buys before you spend it:

- id: plan
  kind: budget_first
  schema: sentiment
  human_labels: 500      # or human_usd, priced at the crowd reward

The projection comes from the dev labels collected so far. For each error target on a grid, it gives the share of the corpus the machines could be certified to label with this budget, and how wide each prevalence interval would be. The contract is met once there are enough dev labels to project from. It promises nothing about the labels, which selective_risk and ppi do once the labels are in.

distributional

distributional compares the machine's label distribution with how people split, for subjective tasks where people disagree for reasons that belong in the data:

- id: disagreement
  kind: distributional
  schema: offensiveness
  max_distance: 0.15     # optional
  groups_by: age_band    # optional: a prestudy question whose answer groups the annotators

Each item labeled by several people (humans.distribution_k, 5 by default) has a human label distribution. The contract reports the mean Jensen-Shannon distance between the machine's distribution and the people's, with a bootstrap interval. It also reports a noise ceiling, the same distance between two random halves of the people on each item. A machine closer to the people than they are to each other is as close as the data allows.

The machine's distribution is its probabilities where the source gives them (decision models, k_sample, logprob), or its label as a one-hot vector. The contract is met when the interval's upper end is at most max_distance, or, without max_distance, once there are 20 items labeled by 3 or more people. When the interval's lower end is above max_distance, it is infeasible, since more items only narrow the interval around a distance that is too large.

With groups_by, the same numbers are reported per annotator group. Each group's distribution on an item is built from that group's people only, read from their answers to the named prestudy question, and if nobody answered the question the contract says so.

threshold_legacy

threshold_legacy accepts machine labels above a fixed confidence and runs no statistical test:

- id: old-rule
  kind: threshold_legacy
  schema: sentiment
  threshold: 0.8

potato label migrate-config writes this contract for icl_labeling, at the old confidence_threshold, so a translated config behaves as before, and you can also write it by hand. The certificate marks these labels uncertified and reports their measured error on the calibration labels, and the export gives them the status accepted_uncertified.

Results from the example studies

The three example studies under examples/labeling-engine/ were run with the LLM, judge and vision sources pointed at Gemma 12B on a vLLM server, instead of the Ollama models the committed configs use. Each example's README shows the source block that was changed, and with the Ollama defaults the numbers will differ.

• Text sentiment (481 reviews, Gemma 12B with logprob confidence as the last tier). selective_risk at 10% and 0.9 was met on 165 human labels, and 316 machine labels were accepted. Their measured error against the true labels was 7.6%, and the PPI intervals covered the true prevalence of all three labels.
• Images (128 COCO images, YOLOv8n and Gemma 12B as the vision model). A has-person label was certified at 10%. The box contract stayed insufficient, because YOLOv8n's boxes were wrong on 32% of the dev images at IoU 0.5.
• Agent traces (288 traces, Gemma 12B as the judge). The judge was right on 80% of traces, but its confidence was 1.0 throughout, so selective_risk was infeasible. The alternative annotator test was insufficient, with the machine at 0.82 against the people's 0.93. PPI estimated task success at 0.66 [0.57, 0.76], where the true value was 0.60 and the judge alone said 0.76, 16 points high.

Further reading

• The labeling engine
• Labeling engine configuration, including the humans keys these contracts read
• Inter-annotator agreement, and agreement over geometry, which uses the same losses for regions, segments and 3D objects

For implementation details, see the source documentation.

References

1. Angelopoulos, A. N., Bates, S., Candès, E. J., et al. (2025). Learn then test: Calibrating predictive algorithms to achieve risk control. The Annals of Applied Statistics. https://doi.org/10.1214/24-AOAS1998
2. Angelopoulos, A. N., Bates, S., Fannjiang, C., et al. (2023). Prediction-powered inference. Science, 382(6671). https://doi.org/10.1126/science.adi6000
3. Angelopoulos, A. N., Duchi, J. C., and Zrnic, T. (2023). PPI++: Efficient prediction-powered inference. arXiv. https://arxiv.org/abs/2311.01453
4. Calderon, N., Reichart, R., and Dror, R. (2025). The alternative annotator test for LLM-as-a-judge: How to statistically justify replacing human annotators with LLMs. Proceedings of ACL 2025. https://aclanthology.org/2025.acl-long.782/