The Labeling Engine

The labeling engine labels a dataset with models and uses people to measure how often the models are wrong. You declare what the finished labels must satisfy in a contract, such as "at most 5% of accepted machine labels are wrong, with 90% confidence". The engine asks people for the labels it needs to test that claim, accepts the machine labels the test supports, and sends the remaining items to people. The output is the dataset plus a certificate that records what was tested and on how many human labels.

The engine is available from Potato 2.11. The annotation server, the potato label command line, the REST API, the MCP server and a Python SDK all drive it over the same engine.sqlite file in the task directory.

When the engine fits a study

The engine pays off when people cannot label everything and a model can label most of it. If people will label every item anyway, the engine adds nothing, and an ordinary assignment strategy is the better choice. Three situations fit:

• You have more items than people can label, and a model that is right most of the time.
• You need to report how good the labels are, not only that a model made them, for a paper, a dataset release, or a training set someone else will use.
• You are evaluating a model or an agent with an LLM judge and need to know whether the judge can stand in for people.

From model prediction to final label

The engine keeps machine predictions, human requests and contract results in one store, and every item moves through the same six steps:

1. Sources are the models. A source can be an LLM, a vision-language model, a decision model (Laya, Jev or OpenJev), an object detector, an LLM judge, a model trained on the study's own labels, a jury of other sources, or labels you already have. Each source records a confidence with every prediction.
2. The cascade orders the sources. An item goes to the first source, and if the contract will not accept that answer, the next source tries it. After the last source the item goes to a person.
3. Requests are what the engine asks people to do. Each request records why it was made (e.g., calibration, certification, verification, audit), the probability that its item was sampled, and whether the person could see a machine answer. Calibration requests are drawn at random and are always blind.
4. Contracts test the machine labels against the human labels on the random sample. Half of that sample (the dev split) chooses the acceptance rule, and the other half (the cert split) tests it. A contract's status is met, insufficient (more labels needed, with an estimate of how many), infeasible, violated or stale.
5. Resolution gives every item its final label. The label is a machine label accepted under a met contract, a person's label, or pending.
6. The certificate records each contract's result, the cascade with model and prompt versions, how many human labels went to each purpose, the model spend, and a digest of the call log.

Rules behind the guarantees

A met contract is a statistical statement about the random sample that people labeled, and it holds because of how that sample is drawn and used. The engine enforces five rules:

• Blind requests show no machine answer. While a person holds a calibration, certification or audit request, the server withholds every machine answer for that item. Pre-annotations, the inline judge, the AI assistant (which answers 403) and the AI help buttons are all off. A label given after seeing the machine's answer is not an independent check of that answer.
• Selection and testing use different labels. Thresholds are chosen on dev labels and tested on cert labels. Each new test on the cert labels runs at a stricter level, so re-certifying until a test passes does not inflate the confidence.
• Only people certify. Machine annotators may take part in final-label inference, never in the labels a contract tests against.
• A change makes the certificate stale. Changing a source, a prompt, a threshold or the scheme marks every affected contract stale and moves its accepted labels to pending_recert until the contract is tested again. The human labels are kept.
• Machine predictions do not fill annotator slots. Engine predictions do not count against per-item annotator caps.

The engine does not check that the human labels are right. Every contract measures the machines against the people who labeled the sample, so a contract tested against unreliable people certifies agreement with unreliable people. Use quality control, overlap (humans.overlap) and the no_worse_than_human contract to check the people as well.

A first study

Potato ships three example studies under examples/labeling-engine/, one each for text, images and agent traces. The table lists the sources and contracts each one uses:

The text example runs in four commands. The first labels every item with the cascade and draws the first blind sample, the second serves that sample to people, and the last two test the contracts and print the certificate once people have labeled the sample:

potato label run examples/labeling-engine/text-certified-sentiment/config.yaml
potato start examples/labeling-engine/text-certified-sentiment/config.yaml -p 8000
potato label certify examples/labeling-engine/text-certified-sentiment/config.yaml
potato label certificate examples/labeling-engine/text-certified-sentiment/config.yaml --format md

To add the engine to your own study, add these keys to a config that already has data_files and an annotation scheme named sentiment. The contract asks that at most 10% of accepted machine labels be wrong, with 90% confidence, and budgets.human_labels caps how many human requests the engine may open:

assignment_strategy: engine
 
labeling_engine:
  enabled: true
  sources:
    - id: gemma
      kind: llm
      endpoint_type: vllm
      model: google/gemma-4-12B-it-qat-w4a16-ct
      base_url: http://localhost:8001/v1
      confidence: {method: logprob}
  contracts:
    - id: accuracy
      kind: selective_risk
      schema: sentiment
      error: 0.1
      confidence: 0.9
  budgets:
    human_labels: 400

The source states its confidence from token log probabilities (method: logprob), which takes one call per item and is usually the best calibrated of the confidence methods. Configuration lists every key, and Contracts says what each contract tests.

Requests on the annotation page

With assignment_strategy: engine, annotators are served the engine's requests, highest priority first. The order is adjudication, certification, calibration, extra raters for distributional contracts, items no machine could label, verification, and audits. Calibration requests are served in a fixed random order, with dev and cert requests interleaved. A person is given every open request on an item at once, one per scheme, and never two requests for the same scheme on the same item.

A short note above the item says which kind of request it is, and on a page with several questions it names the question:

• Blind. The note reads "Label this one yourself. Suggestions are off for this item." The item stays blind for that person after they save it, on reload and on Back.
• Verify. The machine's answer is filled in, and the note says to keep it if it is right and change it if it is not. The page records the prefilled answer, so you can measure how often people accept it. Nothing is saved until the person acts, so a page left open does not answer for them. Schemes whose answers the page cannot prefill (time segments, 3D objects, agent traces) are served blind instead.
• Resolve. No machine label could be accepted, so the person's answer is the label.

A request that someone holds without answering for runtime.hold_minutes (60 by default) goes back to the pool and leaves their page. A label saved on an item the engine did not serve to that person never closes someone else's request, and the engine keeps it as an organic label. When a new certification accepts machine labels, the engine cancels open verify and resolve requests on the items it now covers. Requests a person already holds stay with them, and their answer replaces the machine label.

The progress counter counts the requests a person can still take, not the corpus. When there is nothing to do yet, the annotator sees a short waiting page and is served new requests as the planner opens them. When nothing more will come, because every item has a label, the human-label budget is spent, or the person's own cap is reached, their part ends as usual with the post-study pages and, on a crowd platform, the completion code. Nobody is finished before the planner's first pass.

Pocket Mode serves engine requests too, a full batch at a time, and asks for more before it says "All caught up". Pocket Mode never shows a machine answer, so a verify request served there is recorded as blind, unless the desktop page had already shown the answer to that person. On resolve and adjudicate cards it shows the same note as the desktop page.

The admin page

When the engine is on, the admin dashboard links to Labeling engine at /admin/labeling-engine. Every chart on the page also has a table view. The page shows:

• each contract's status, coverage, look number and how many more labels it needs, with the full statement it makes (an unmet contract states its target, not a guarantee);
• the intervals of each ppi contract, beside the intervals from people alone;
• a warning for a source whose last run failed or whose every answer was an error;
• the error-against-coverage curve for the last cascade tier, with the guaranteed upper bound and the target;
• a reliability diagram per tier, plotting stated confidence against the share correct on human labels, with ECE and Brier score, and bins with few labels drawn paler;
• the audits of accepted labels, as errors against the target, and whether there are too few audits to tell;
• human requests by purpose and status;
• the model-call log, newest first, with tokens, latency and cost; and
• buttons to run a planner pass, certify now, and open the certificate.

Opened without an admin session, the page goes to the dashboard's key form. With the engine off, it says so and names the key that turns it on.

Further reading

• Labeling engine contracts: the six contract kinds and what each tests
• Labeling engine configuration: every labeling_engine key
• Command line, REST API, MCP tools and SDK
• Migrating to the labeling engine from icl_labeling, judge_calibration and pre_annotation
• Solo Mode, for one expert working with an LLM
• LLM pre-annotation and LLM judge agreement

For implementation details, see the source documentation.