# Potato 2.11: The Labeling Engine and Certified Machine Labels Source: https://www.potatoannotator.com/blog/potato-2-11-labeling-engine Potato 2.11 adds the labeling engine, which labels a dataset with models and uses a blind random sample of human labels to measure how often the models are wrong. You state what the finished labels must satisfy, for example that at most 10% of accepted machine labels are wrong with 90% confidence. The engine accepts the machine labels a statistical test supports, sends the remaining items to people, and writes a certificate that records what was tested and on how many human labels. The release also extends the agreement report to 3D objects, temporal grounding, tiered annotation and agent traces, and fixes an image bug that could count untouched items as answered. ```bash pip install --upgrade potato-annotation ``` ## Upgrade notes **Idle sessions are now signed out, after 8 hours by default.** Before 2.11, `session_timeout_minutes` had no effect on a served app, because the check was registered on an app object that `potato start` discards. A session idle for longer than the setting is now cleared, and its next request goes to the login page. Kiosks and shared lab machines that stay signed in overnight will start seeing sign-outs, so raise `session_timeout_minutes` where a study depends on staying signed in. Annotations already saved are unaffected. **Three older keys print a note at startup.** `icl_labeling`, `judge_calibration` and `pre_annotation` keep working exactly as before, and each prints a line beginning `NOTE:` to stderr that names its engine equivalent. The notes are informational, so `potato validate --strict` does not fail on them. The keys are planned for removal in 3.0, and `potato label migrate-config` writes the equivalent `labeling_engine` block for you. ## The labeling engine The engine is for studies with more items than people can label and a model that is right most of the time. It treats the models as *sources*, tried in a cascade in the order you list them, and sends an item to a person after the last source. Sources can be LLMs, vision-language models, decision models such as Laya, YOLO detectors, LLM judges, juries, models trained on the study's own labels, or labels you already have. A *contract* is what the engine tests. The main one, `selective_risk`, bounds the share of accepted machine labels that are wrong. The engine chooses a confidence threshold on half of the blind sample and tests it on the other half with a one-sided binomial test, using fixed-sequence [Learn-then-Test](https://doi.org/10.1214/24-AOAS1998). Each new test runs at a stricter level, so re-certifying until a test passes does not inflate the confidence. The `ppi` contract reports each label's prevalence with an interval built by [prediction-powered inference](https://doi.org/10.1126/science.adi6000), which stays valid whatever the model's quality. `no_worse_than_human` runs the [alternative annotator test](https://aclanthology.org/2025.acl-long.782/) for replacing a person or a judge, and `distributional` compares the model's label distribution with how people split on subjective items. A config that certifies one LLM on a sentiment scheme adds two keys to an ordinary study: ```yaml assignment_strategy: engine labeling_engine: enabled: true sources: - id: gemma kind: llm endpoint_type: vllm model: google/gemma-4-12B-it-qat-w4a16-ct base_url: http://localhost:8001/v1 confidence: {method: logprob} contracts: - id: accuracy kind: selective_risk schema: sentiment error: 0.1 confidence: 0.9 budgets: human_labels: 400 ``` `confidence: {method: logprob}` reads the model's confidence from its token probabilities, which takes one call per item and is usually the best calibrated of the engine's confidence methods. The contract allows at most 10% of accepted labels to be wrong, and its 90% confidence is Potato's default. `budgets.human_labels` caps how many human requests the engine may open, and 400 is the cap the shipped text example uses. The text example that ships with Potato runs 481 reviews through a cascade with Laya ahead of Gemma 12B. In that run the 10% contract was met on 165 human labels, and the engine accepted 316 machine labels, all of them from Gemma because Laya's threshold was never fixed. Their measured error against the reference labels was 7.6%. The agent-trace example has 288 synthetic traces with reference outcomes. The judge was right on 80% of them but gave a confidence of 1.0 to every verdict, so `selective_risk` had nothing to threshold on and stayed infeasible. The PPI interval for task success from the same data, [0.57, 0.76], covered the reference rate of 0.60, while the judge alone said 0.76. ### Limits of a met contract A met contract is a statement about agreement with the people who labeled the sample. The engine does not check that those people are right, so a contract tested against careless annotators certifies agreement with careless annotators. Three rules make that statement hold. Calibration requests are blind, so the page shows no pre-annotation, judge verdict or AI help while a person holds one. Thresholds are chosen on one half of the sample and tested on the other. Only human labels certify, and any change to a source, prompt or scheme marks the affected contracts `stale` until they are tested again. ### Interfaces to the engine The same `engine.sqlite` file backs every interface, so you can start a study in one and certify in another: - `potato label run|status|certify|certificate|export|calls|distill|optimize|migrate-config` on the command line; - a REST API under `/api/engine/v1`; - five `engine_*` tools in the MCP server; - `potato.labeling_engine.sdk.Study` in Python; and - an admin page at `/admin/labeling-engine` with contract status, the error-against-coverage curve, a reliability diagram per source, and the model-call log. `potato label export` writes the labels with their provenance, the certificate in JSON, Markdown and HTML, a dataset card and the call log. [The labeling engine](/docs/labeling-engine/overview) walks through a first study, and [Labeling engine contracts](/docs/labeling-engine/contracts) covers what each contract tests and how it fails. ## Solo Mode Solo Mode can now hide its suggestion. `solo_mode.suggestions` withholds the LLM's label on every item or on a random sample. A suggestion on screen pulls the answer toward it, so the handoff reads agreement on the blind items. Decision models (`laya` and any `/v1/systemone` server) can be a fast first tier, and Solo Mode checks the project's `ai_budget.cap_usd` before labeling starts and before each batch. A gold set, a labeling jury and curriculum ordering are also new. When the engine is on, Solo Mode writes its predictions into the engine's store, so the engine can calibrate and certify them. [Solo Mode](/docs/features/solo-mode) has the configs. ## Agreement `/admin/iaa` now reports agreement on `spatial_annotation`, `temporal_grounding`, `tiered_annotation` and the five agent-trace schemes, which it used to skip as unsupported. 3D objects and time segments get a mean matched-set agreement, which is not chance-corrected, and agent-trace steps get Krippendorff's alpha and Cohen's kappa with each step of each trace as a unit. `/admin/api/agreement` no longer reports structured schemes and points to `/admin/iaa` instead, because reduced to label names those schemes showed perfect agreement between every pair of annotators. Two identical zero-length segments now count as agreeing. Publishing and Paper Mode leave machine raters out of gold labels and agreement figures. ## Fixes that affect 2.10 studies Three fixes can change numbers you have already collected or reported: - **An image page saved a prefilled answer on its own.** On an `image_annotation` scheme with `pre_annotation`, the canvas saved the prefilled shapes as the annotator's answer when it re-laid itself out after loading, so an untouched item counted as answered. In a 2.10 image study with pre-annotations, an item whose saved shapes match its pre-annotation exactly may never have been touched, so treat those items as unreviewed unless your timing data shows otherwise. - **Paper Mode computed kappa with its own code.** Paper Mode and the agreement module now compute Cohen's kappa through the same function as `/admin/iaa`, so a kappa reported from Paper Mode before 2.11 can differ from the one `/admin/iaa` shows for the same data. - **Adjudication and Prolific auto-scaling counted machine raters.** Machine annotators could send an item to adjudication, `min_human_annotations` was ignored, and Prolific auto-scaling counted machine annotations against per-item caps. The remaining fixes change what a study can do. Images on hosts that send no CORS headers, such as cocodataset.org, now load. On such an image, flood fill, SAM and text-prompt segmentation now say why they cannot read it. A pre-annotated shape without a `color` takes its label's color instead of being dropped. Solo Mode's "Generate prompt" now asks the revision model to write the first prompt. Refusals over `potato mcp connect` come back as tool errors instead of successes. Edge-case discovery now works on vLLM, which did not return the edge-case fields until they were added to the response schema. ## Installing 2.11 Upgrade with `pip install --upgrade potato-annotation`, or pull `ghcr.io/davidjurgens/potato:2.11.0`. The in-process Laya decision model needs the new `decision` extra, `pip install 'potato-annotation[decision]'`, which stays out of `all` because it pulls in torch. The [release notes](https://github.com/davidjurgens/potato/blob/master/docs/releasenotes/v2.11.0.md) list every change, and [What's New](/docs/getting-started/whats-new-v2) has the summary for each 2.x release.