Migrating to the Labeling Engine
Move icl_labeling, judge_calibration, pre_annotation and Solo Mode configs to the labeling engine with potato label migrate-config, and see what changes once you do.
Potato's older machine-labeling features keep working in 2.11, and nothing changes behavior until you add a labeling_engine block. Three keys now print an informational note at startup that names their labeling engine equivalent, and they are planned for removal in 3.0. The notes go to stderr, start with NOTE:, and are not warnings, so potato validate --strict does not fail on them.
The table lists the three keys and what each becomes:
| Older key | Engine equivalent |
|---|---|
icl_labeling | an llm source, with a threshold_legacy contract for the old rule and a selective_risk contract beside it |
judge_calibration | judge sources with k_sample confidence, whose blind sample becomes calibration requests |
pre_annotation | a static source |
Solo Mode, judge alignment and the interactive AI assistant (ai_support) are not deprecated. Solo Mode is one expert working with an LLM through its own interface, and judge alignment is a report. When the engine is on, both also write their model predictions into the engine's store, so the engine can calibrate and certify them.
potato label migrate-config
potato label migrate-config translates the older keys into a labeling_engine block:
potato label migrate-config config.yaml --out config-engine.yamlThe command reads the YAML without loading any data, adds an equivalent labeling_engine block, and writes the result with a header of notes that explain each translation. It writes the block with enabled: false, so the translated config behaves as the original did until you turn the engine on. Read the notes, check the sources and contracts, then set enabled: true and assignment_strategy: engine. The command exits with an error when the config already has a labeling_engine block, or when there is nothing to translate.
Translation table
Each older block maps to engine sources, contracts and human settings as follows:
| Older block | Sources | Contracts | Humans |
|---|---|---|---|
solo_mode with confidence_routing.tiers | one source per tier, in tier order | selective_risk per one-of-N scheme, error: 1 - end_human_annotation_agreement | pools: [solo] |
solo_mode with labeling_models | one source per model, and the first is the cascade | as above | pools: [solo] |
solo_mode.jury | one source per juror, plus a jury source as the cascade | as above | |
icl_labeling | the ai_support endpoint as an llm source | threshold_legacy at the old confidence_threshold, and selective_risk at 10% | calibration_rate = the old verification sample_rate |
judge_alignment | the judge endpoint as a judge source (with the rubric when there is one scheme) | no_worse_than_human, epsilon: 0.15, per judged scheme | overlap: 0.3 |
judge_calibration | each model as a judge source with k_sample confidence | sampling.sample_size → budgets.human_labels | |
pre_annotation with field or predictions_file | a static source, first in the cascade | ||
active_learning.llm | an llm source | ||
ai_budget.cap_usd | budgets.model_usd |
Behavior changes after migration
The engine treats human labels differently from the older features, and the notes in the translated config say where. Expect four changes:
- Fixed thresholds become tested ones. Solo Mode's routing thresholds and
icl_labeling's confidence threshold accepted machine labels above a fixed number.selective_riskchooses the threshold on dev labels and accepts only what passes a test on held-out labels, so expect fewer labels accepted at first, with a stated error bound on the ones that are. - Verification samples become uniform.
icl_labelingverified the least-confident predictions first, which finds errors quickly and gives a biased accuracy estimate. Engine calibration requests are a random sample with recorded inclusion probabilities. - Inline judging becomes blind.
judge_alignment.inlineshowed the judge's verdict while people labeled, so agreement was measured on anchored labels. Engine calibration requests never show a machine answer. - Pre-annotations are tested before they are trusted. A
staticsource's labels are accepted only where a contract certifies them, and elsewhere people see them on verify requests.
Further reading
For implementation details, see the source documentation.