Potato 2.11: The Labeling Engine and Certified Machine Labels

Potato 2.11 adds the labeling engine, which labels a dataset with models and uses a blind random sample of human labels to measure how often the models are wrong. You state what the finished labels must satisfy, for example that at most 10% of accepted machine labels are wrong with 90% confidence. The engine accepts the machine labels a statistical test supports, sends the remaining items to people, and writes a certificate that records what was tested and on how many human labels. The release also extends the agreement report to 3D objects, temporal grounding, tiered annotation and agent traces, and fixes an image bug that could count untouched items as answered.

bash

pip install --upgrade potato-annotation

Upgrade notes

Idle sessions are now signed out, after 8 hours by default. Before 2.11, session_timeout_minutes had no effect on a served app, because the check was registered on an app object that potato start discards. A session idle for longer than the setting is now cleared, and its next request goes to the login page. Kiosks and shared lab machines that stay signed in overnight will start seeing sign-outs, so raise session_timeout_minutes where a study depends on staying signed in. Annotations already saved are unaffected.

Three older keys print a note at startup. icl_labeling, judge_calibration and pre_annotation keep working exactly as before, and each prints a line beginning NOTE: to stderr that names its engine equivalent. The notes are informational, so potato validate --strict does not fail on them. The keys are planned for removal in 3.0, and potato label migrate-config writes the equivalent labeling_engine block for you.

The labeling engine

The engine is for studies with more items than people can label and a model that is right most of the time. It treats the models as sources, tried in a cascade in the order you list them, and sends an item to a person after the last source. Sources can be LLMs, vision-language models, decision models such as Laya, YOLO detectors, LLM judges, juries, models trained on the study's own labels, or labels you already have.

A contract is what the engine tests. The main one, selective_risk, bounds the share of accepted machine labels that are wrong. The engine chooses a confidence threshold on half of the blind sample and tests it on the other half with a one-sided binomial test, using fixed-sequence Learn-then-Test. Each new test runs at a stricter level, so re-certifying until a test passes does not inflate the confidence. The ppi contract reports each label's prevalence with an interval built by prediction-powered inference, which stays valid whatever the model's quality. no_worse_than_human runs the alternative annotator test for replacing a person or a judge, and distributional compares the model's label distribution with how people split on subjective items.

A config that certifies one LLM on a sentiment scheme adds two keys to an ordinary study:

yaml

assignment_strategy: engine
 
labeling_engine:
  enabled: true
  sources:
    - id: gemma
      kind: llm
      endpoint_type: vllm
      model: google/gemma-4-12B-it-qat-w4a16-ct
      base_url: http://localhost:8001/v1
      confidence: {method: logprob}
  contracts:
    - id: accuracy
      kind: selective_risk
      schema: sentiment
      error: 0.1
      confidence: 0.9
  budgets:
    human_labels: 400

confidence: {method: logprob} reads the model's confidence from its token probabilities, which takes one call per item and is usually the best calibrated of the engine's confidence methods. The contract allows at most 10% of accepted labels to be wrong, and its 90% confidence is Potato's default. budgets.human_labels caps how many human requests the engine may open, and 400 is the cap the shipped text example uses.

The text example that ships with Potato runs 481 reviews through a cascade with Laya ahead of Gemma 12B. In that run the 10% contract was met on 165 human labels, and the engine accepted 316 machine labels, all of them from Gemma because Laya's threshold was never fixed. Their measured error against the reference labels was 7.6%. The agent-trace example has 288 synthetic traces with reference outcomes. The judge was right on 80% of them but gave a confidence of 1.0 to every verdict, so selective_risk had nothing to threshold on and stayed infeasible. The PPI interval for task success from the same data, [0.57, 0.76], covered the reference rate of 0.60, while the judge alone said 0.76.

Limits of a met contract

A met contract is a statement about agreement with the people who labeled the sample. The engine does not check that those people are right, so a contract tested against careless annotators certifies agreement with careless annotators. Three rules make that statement hold. Calibration requests are blind, so the page shows no pre-annotation, judge verdict or AI help while a person holds one. Thresholds are chosen on one half of the sample and tested on the other. Only human labels certify, and any change to a source, prompt or scheme marks the affected contracts stale until they are tested again.

Interfaces to the engine

The same engine.sqlite file backs every interface, so you can start a study in one and certify in another:

potato label run|status|certify|certificate|export|calls|distill|optimize|migrate-config on the command line;
a REST API under /api/engine/v1;
five engine_* tools in the MCP server;
potato.labeling_engine.sdk.Study in Python; and
an admin page at /admin/labeling-engine with contract status, the error-against-coverage curve, a reliability diagram per source, and the model-call log.

potato label export writes the labels with their provenance, the certificate in JSON, Markdown and HTML, a dataset card and the call log. The labeling engine walks through a first study, and Labeling engine contracts covers what each contract tests and how it fails.

Solo Mode

Solo Mode can now hide its suggestion. solo_mode.suggestions withholds the LLM's label on every item or on a random sample. A suggestion on screen pulls the answer toward it, so the handoff reads agreement on the blind items. Decision models (laya and any /v1/systemone server) can be a fast first tier, and Solo Mode checks the project's ai_budget.cap_usd before labeling starts and before each batch. A gold set, a labeling jury and curriculum ordering are also new. When the engine is on, Solo Mode writes its predictions into the engine's store, so the engine can calibrate and certify them. Solo Mode has the configs.

Agreement

/admin/iaa now reports agreement on spatial_annotation, temporal_grounding, tiered_annotation and the five agent-trace schemes, which it used to skip as unsupported. 3D objects and time segments get a mean matched-set agreement, which is not chance-corrected, and agent-trace steps get Krippendorff's alpha and Cohen's kappa with each step of each trace as a unit. /admin/api/agreement no longer reports structured schemes and points to /admin/iaa instead, because reduced to label names those schemes showed perfect agreement between every pair of annotators. Two identical zero-length segments now count as agreeing. Publishing and Paper Mode leave machine raters out of gold labels and agreement figures.

Fixes that affect 2.10 studies

Three fixes can change numbers you have already collected or reported:

An image page saved a prefilled answer on its own. On an image_annotation scheme with pre_annotation, the canvas saved the prefilled shapes as the annotator's answer when it re-laid itself out after loading, so an untouched item counted as answered. In a 2.10 image study with pre-annotations, an item whose saved shapes match its pre-annotation exactly may never have been touched, so treat those items as unreviewed unless your timing data shows otherwise.
Paper Mode computed kappa with its own code. Paper Mode and the agreement module now compute Cohen's kappa through the same function as /admin/iaa, so a kappa reported from Paper Mode before 2.11 can differ from the one /admin/iaa shows for the same data.
Adjudication and Prolific auto-scaling counted machine raters. Machine annotators could send an item to adjudication, min_human_annotations was ignored, and Prolific auto-scaling counted machine annotations against per-item caps.

The remaining fixes change what a study can do. Images on hosts that send no CORS headers, such as cocodataset.org, now load. On such an image, flood fill, SAM and text-prompt segmentation now say why they cannot read it. A pre-annotated shape without a color takes its label's color instead of being dropped. Solo Mode's "Generate prompt" now asks the revision model to write the first prompt. Refusals over potato mcp connect come back as tool errors instead of successes. Edge-case discovery now works on vLLM, which did not return the edge-case fields until they were added to the response schema.

Installing 2.11

Upgrade with pip install --upgrade potato-annotation, or pull ghcr.io/davidjurgens/potato:2.11.0. The in-process Laya decision model needs the new decision extra, pip install 'potato-annotation[decision]', which stays out of all because it pulls in torch. The release notes list every change, and What's New has the summary for each 2.x release.