Labeling Engine Configuration

Everything the labeling engine does is set in one labeling_engine block. An invalid block stops the server at startup and lists every problem at once. To have annotators served the engine's requests, also set assignment_strategy: engine. Setting that strategy without an enabled engine is a validation error.

The block has these sections, each covered below:

assignment_strategy: engine
 
labeling_engine:
  enabled: true
  schemas: [sentiment]            # default: every scheme a source can label
  sources: []
  cascade: [laya, gemma]          # default: the first source alone
  contracts: []
  humans: {}
  crowd: {}
  budgets: {}
  runtime: {}
  features: {}
  redaction: {}
  distill: {}

Top-level keys

Five keys sit directly under labeling_engine:

Sources

A source is a model, or an existing set of labels, that proposes a label and a confidence for each item. Every source has a unique id and a kind, and an unknown key on a source is an error:

Everything that can change an answer (model, prompt, temperature, options) goes into the source's version hash. A new version makes every contract that depends on the source stale.

An answer that is not a value the scheme can hold, such as a label not in its list, a number field that is not a number, or a list on a one-of-N scheme, is recorded as an error and never as a label, and the run's message counts these errors. An item whose calls fail is asked again on later runs, at most three times per source version, and after that it waits for a changed source (a fixed key or prompt is a new version). A run in which one source fails does not stop the other tiers. The next tier labels every item, and potato label run exits with status 1 and names the failed source.

Confidence methods

The engine's acceptance rules threshold on each prediction's confidence, so how a source states confidence decides how much the engine can accept. confidence.method takes one of five values:

Kind-specific options

Five source kinds read options of their own:

• decision covers Laya in-process, or any /v1/systemone server such as hosted Jev or OpenJev. One forward pass returns a probability for every option, and it works on radio, select, likert and slider schemes. endpoint_type must be laya or systemone, and systemone needs base_url, the server root (a URL ending in /v1/systemone is accepted too). The state is sent as {"text": ...} in both cases, so one checkpoint gives the same answers in-process and behind laya-serve, and options: {state_key: ""} sends the bare string instead. On rating scales (likert, slider), Laya's scores can barely follow the scale, so check a few items before relying on it. The in-process laya endpoint needs pip install 'potato-annotation[decision]'.
• detector reads boxes from an object detector on image_annotation schemes. keep_threshold (0.35) sets the score at or above which a box is part of the prediction. Boxes between candidate_threshold (0.05) and keep_threshold lower the image's confidence. label_map maps detector classes to scheme labels, and unmapped classes are ignored. image_field names the item field with the image path or URL. Paths resolve as the annotation page resolves them, with /media/x.jpg under the task's media_directory and a relative path under task_dir.
• judge takes rubric (the scheme's description by default), fields (the item fields shown to the judge, in order) and swap_order, which asks twice with the label order reversed and turns a verdict that flips into an abstention. On agent-trace schemes (process_reward, trajectory_eval, ...) the judge labels each step.
• vlm uses a vision endpoint (openai_vision, anthropic_vision, ollama_vision). For a self-hosted model behind an OpenAI-compatible server, use openai_vision with base_url.
• trained takes artifact_dir, trainer (sklearn-text by default) and run_id. Distillation usually creates this source, rather than a hand-written config.

humans

The humans block sets who labels and how the engine samples them:

crowd

The crowd block applies when crowd is in humans.pools, and Prolific is the only crowd platform the engine can drive:

Crowd recruiting is a dry run by default, and the planner reports what it would do. Spending money needs features.crowd_live_spend: true and a budgets.human_usd cap, and a config that turns on live spend without the cap is rejected. Crowdsourcing covers setting up the Prolific study itself.

budgets

The budgets block stops spending before a limit is exceeded:

runtime

The runtime block controls batching, concurrency and how long a person may hold a request:

features

Every feature is off by default except drift:

redaction

redaction.categories lists which of email, url, ip, card, ssn, phone and handle to mask, and defaults to all of them. Masking applies only with features.pii_redaction, and only to third-party APIs. Self-hosted endpoint types (vllm, ollama, laya, yolo, ...) are not masked wherever they run, and neither is any endpoint whose base_url is on the same machine.

distill

With features.distill on, the engine trains a cheap model on the study's labels and puts it first in the cascade:

Items in the cert and audit splits never go into the training set, so the distilled model is never tested on labels it was trained on.

Validation

potato validate config.yaml checks the labeling_engine block along with the rest of the config, and each error names its key, for example labeling_engine.sources[1].kind must be one of ... or labeling_engine.contracts[0].error is required for selective_risk. Validation also refuses:

• a contract key the contract kind does not read, such as a typo of error;
• a contract on a scheme no cascade source can label, with each source's reason;
• a contract on a scheme the engine cannot score (free text, pairwise);
• a decision source whose endpoint_type is not a decision model, and an llm, vlm or judge source whose endpoint_type is one; and
• no_worse_than_human with humans.overlap_k below 3.

Validation also warns when budgets.human_labels is too small to certify a selective_risk contract even if every machine answer were right. Raise the budget, or loosen error, until that warning goes away.

Further reading

• The labeling engine
• Labeling engine contracts
• Command line, REST API, MCP tools and SDK
• AI support, for the endpoint types sources share with the rest of Potato

For implementation details, see the source documentation.