# Labeling Engine Configuration

Source: https://www.potatoannotator.com/docs/labeling-engine/configuration

Everything the [labeling engine](/docs/labeling-engine/overview) does is set in one `labeling_engine` block. An invalid block stops the server at startup and lists every problem at once. To have annotators served the engine's requests, also set `assignment_strategy: engine`. Setting that strategy without an enabled engine is a validation error.

The block has these sections, each covered below:

```yaml
assignment_strategy: engine

labeling_engine:
  enabled: true
  schemas: [sentiment]            # default: every scheme a source can label
  sources: []
  cascade: [laya, gemma]          # default: the first source alone
  contracts: []
  humans: {}
  crowd: {}
  budgets: {}
  runtime: {}
  features: {}
  redaction: {}
  distill: {}
```

## Top-level keys

Five keys sit directly under `labeling_engine`:

| Key | Default | Meaning |
|---|---|---|
| `enabled` | `false` | Turn the engine on. An enabled engine needs at least one source. |
| `schemas` | all | The annotation schemes the engine labels. Under `assignment_strategy: engine`, a scheme left out is answered only on the items the engine sends to people for its own schemes, not on every item, and validation warns about it. A scheme no source can label (free text, for one) cannot be in the engine, so under this strategy it is never answered on every item. Use another assignment strategy if it must be. |
| `sources` | `[]` | The models. See [Sources](#sources). |
| `cascade` | first source | Source ids in the order they are tried. People come after the last. |
| `contracts` | `[]` | What the labels must satisfy. See [Contracts](/docs/labeling-engine/contracts). |

## Sources

A source is a model, or an existing set of labels, that proposes a label and a confidence for each item. Every source has a unique `id` and a `kind`, and an unknown key on a source is an error:

| Key | Default | Meaning |
|---|---|---|
| `id` | required | Name used in `cascade`, the call log and the certificate. |
| `kind` | required | One of `llm`, `vlm`, `decision`, `detector`, `judge`, `jury`, `trained`, `static`, `fixture`, `asr`, `lf`, `simulator`. |
| `endpoint_type` | required for `llm`, `vlm`, `decision`, `judge` | The AI endpoint, such as `vllm`, `openai`, `anthropic`, `gemini`, `ollama`, `openrouter`, `openai_vision`, `ollama_vision`, `laya` or `systemone`. `detector` defaults to `yolo`. |
| `model` | endpoint default | Model name. |
| `base_url` | none | Server URL for self-hosted or OpenAI-compatible endpoints. |
| `api_key` | none | Never hashed into a version and never exported. `${ENV_VAR}` is substituted. |
| `temperature` | `0.0` | 0 to 2. |
| `max_tokens` | `256` | Output token limit per call. |
| `timeout` | endpoint default | Seconds per call. |
| `prompt` | built from the scheme | Prompt template. Changing it makes a new source version. |
| `confidence` | `{method: auto}` | How the source states confidence. See [Confidence methods](#confidence-methods). |
| `concurrency` | `runtime.concurrency` | Parallel calls to this source. |
| `schemas` | all engine schemes | Restrict the source to these schemes. |
| `options` | `{}` | Kind-specific options, listed below. |
| `judge` | `{}` | `judge` only. The same options as `options`, under their own key, and `options` wins where both set one. |
| `members` | none | `jury` only. At least two other source ids. |
| `aggregation` | `plurality` | `jury` only. Plurality is the one rule. |
| `field` / `predictions_file` | none | `static` only. Where the existing labels are. The field holds `{scheme: label}`, or a bare label when the source labels one scheme. A field of the wrong shape on every item fails the run with a message naming it. |
| `error_rate`, `seed` | none | `fixture` only. A synthetic source with a known error rate, for tests. |
| `revision` | none | A model revision to record in the version, for example a checkpoint hash. |

Everything that can change an answer (model, prompt, temperature, options) goes into the source's version hash. A new version makes every contract that depends on the source `stale`.

An answer that is not a value the scheme can hold, such as a label not in its list, a number field that is not a number, or a list on a one-of-N scheme, is recorded as an error and never as a label, and the run's message counts these errors. An item whose calls fail is asked again on later runs, at most three times per source version, and after that it waits for a changed source (a fixed key or prompt is a new version). A run in which one source fails does not stop the other tiers. The next tier labels every item, and `potato label run` exits with status 1 and names the failed source.

### Confidence methods

The engine's acceptance rules threshold on each prediction's confidence, so how a source states confidence decides how much the engine can accept. `confidence.method` takes one of five values:

| `confidence.method` | Applies to | What it does |
|---|---|---|
| `verbalized` | `llm`, `vlm`, `judge` | The model states 0-100 in its reply. One call, and the least reliable method. |
| `k_sample` | `llm` | `samples` replies (5 by default) at `sample_temperature` (0.7 by default). The label is the most common answer and the confidence is its share. Costs k calls. |
| `logprob` | `llm` | Labels are lettered and the model answers with one letter, and the confidence is read from the first token's probabilities. One call, usually the best calibrated. Needs a backend that returns logprobs (vLLM, OpenAI-compatible) and at most 20 labels, and a scheme with more is a validation error. |
| `native` | `decision`, `detector`, `trained` | The model's own probabilities. On an `llm` source it is a validation error. |
| `auto` | all | `native` where the source has it, otherwise `verbalized`. |

### Kind-specific options

Five source kinds read options of their own:

- **`decision`** covers Laya in-process, or any `/v1/systemone` server such as hosted Jev or OpenJev. One forward pass returns a probability for every option, and it works on `radio`, `select`, `likert` and `slider` schemes. `endpoint_type` must be `laya` or `systemone`, and `systemone` needs `base_url`, the server root (a URL ending in `/v1/systemone` is accepted too). The state is sent as `{"text": ...}` in both cases, so one checkpoint gives the same answers in-process and behind `laya-serve`, and `options: {state_key: ""}` sends the bare string instead. On rating scales (`likert`, `slider`), Laya's scores can barely follow the scale, so check a few items before relying on it. The in-process `laya` endpoint needs `pip install 'potato-annotation[decision]'`.
- **`detector`** reads boxes from an object detector on `image_annotation` schemes. `keep_threshold` (0.35) sets the score at or above which a box is part of the prediction. Boxes between `candidate_threshold` (0.05) and `keep_threshold` lower the image's confidence. `label_map` maps detector classes to scheme labels, and unmapped classes are ignored. `image_field` names the item field with the image path or URL. Paths resolve as the annotation page resolves them, with `/media/x.jpg` under the task's `media_directory` and a relative path under `task_dir`.
- **`judge`** takes `rubric` (the scheme's description by default), `fields` (the item fields shown to the judge, in order) and `swap_order`, which asks twice with the label order reversed and turns a verdict that flips into an abstention. On agent-trace schemes (`process_reward`, `trajectory_eval`, ...) the judge labels each step.
- **`vlm`** uses a vision endpoint (`openai_vision`, `anthropic_vision`, `ollama_vision`). For a self-hosted model behind an OpenAI-compatible server, use `openai_vision` with `base_url`.
- **`trained`** takes `artifact_dir`, `trainer` (`sklearn-text` by default) and `run_id`. Distillation usually creates this source, rather than a hand-written config.

## humans

The `humans` block sets who labels and how the engine samples them:

| Key | Default | Meaning |
|---|---|---|
| `pools` | `[team]` | Who labels: `team`, `crowd`, `solo`. |
| `calibration_rate` | `auto` | Share of items sampled for blind calibration. `auto` picks a rate from the contracts' projections, enough labels per class to be testable, at most half of a small study and a fifth of one over 1,000 items. |
| `dev_fraction` | `0.5` | Share of calibration labels used to choose thresholds. The rest test them. |
| `min_calibration_per_class` | `10` | Floor per class before per-class contracts are tested. |
| `audit_rate` | `0.02` | Share of newly accepted machine labels sent back to people, blind, to watch for drift. |
| `overlap` | `0.0` | Share of calibration items labeled by several people. |
| `overlap_k` | `3` | People per overlap item. The `no_worse_than_human` contract needs 3. |
| `distribution_k` | `5` | People per item for `distributional` contracts. |
| `truth` | `majority` | How several people's labels on one item become the label contracts test against, `majority` or `dawid_skene`. |
| `verify_visibility` | `shown` | Whether a verify request shows the machine's answer (`shown`) or not (`blind`). |

## crowd

The `crowd` block applies when `crowd` is in `humans.pools`, and Prolific is the only crowd platform the engine can drive:

| Key | Default | Meaning |
|---|---|---|
| `provider` | `prolific` | The crowd platform. |
| `study_id` | none | The Prolific study to resize. |
| `reward_usd` | none | Reward per place, for cost projections. |
| `per_worker_quota` | `20` | Requests per worker. Places are `ceil(open requests / quota)`. |

Crowd recruiting is a dry run by default, and the planner reports what it would do. Spending money needs `features.crowd_live_spend: true` and a `budgets.human_usd` cap, and a config that turns on live spend without the cap is rejected. [Crowdsourcing](/docs/deployment/crowdsourcing) covers setting up the Prolific study itself.

## budgets

The `budgets` block stops spending before a limit is exceeded:

| Key | Meaning |
|---|---|
| `model_usd` | Stop calling priced models once this much is spent. Each call reserves its projected cost first. Models with no price, and self-hosted models on `localhost`, are not limited by it. |
| `human_usd` | Ceiling on crowd spend. Required for live crowd spend. |
| `human_labels` | Maximum human requests the engine may open, counted when each request is opened. |

## runtime

The `runtime` block controls batching, concurrency and how long a person may hold a request:

| Key | Default | Meaning |
|---|---|---|
| `batch_size` | `32` | Items per committed chunk. A killed run resumes at the next chunk without repeating calls. |
| `concurrency` | `4` | Parallel calls per source, overridden by a source's own `concurrency`. |
| `cache` | `true` | Reuse replies to identical requests from the same source version. |
| `retries` | `2` | Extra attempts after a timeout, connection error, 429 or 5xx. Every attempt is logged. |
| `planner_interval` | `30` | Seconds between planner passes in the server. |
| `hold_minutes` | `60` | Minutes a person may hold a request without answering before it goes back to the pool. The item also leaves their page, so a tab left open cannot answer it later. |

## features

Every feature is off by default except `drift`:

| Key | What it turns on |
|---|---|
| `drift` | On by default. Acts on audit results. When audits show the accepted labels' error rate is above target, the contract reads `violated`, machine labels on that scheme are no longer accepted, and fresh calibration labels are drawn. Each certification starts a new audit sequence, so certifying again after a fix restores acceptance. Set it to `false` to report a breach only, and the contract then reads "met, audits breached". |
| `distill` | Train a model on the study's labels and put it first in the cascade (see [distill](#distill)). |
| `optimize` | Have each planner pass score every subset of the cascade on dev labels and report the cheapest one that keeps the contracts' coverage, at `GET /api/engine/v1/schemes/<name>/optimize`. Nothing is applied, because adopting a cascade is a config change and a new certification. |
| `crowd_recruit` | Let the planner size the Prolific study, as a dry run unless `crowd_live_spend` is on. |
| `crowd_live_spend` | Actually change the Prolific study. Needs the `crowd` pool and `budgets.human_usd`. |
| `pii_redaction` | Mask emails, URLs, IP addresses, card numbers, SSNs, phone numbers and @handles before text goes to a third-party API. The call log records what was masked as categories and hashes, never the text. |
| `payload_capture` | Keep full prompts and replies under `<task_dir>/engine_payloads/`. Off by default, and the call log keeps hashes only. |

## redaction

`redaction.categories` lists which of `email`, `url`, `ip`, `card`, `ssn`, `phone` and `handle` to mask, and defaults to all of them. Masking applies only with `features.pii_redaction`, and only to third-party APIs. Self-hosted endpoint types (`vllm`, `ollama`, `laya`, `yolo`, ...) are not masked wherever they run, and neither is any endpoint whose `base_url` is on the same machine.

## distill

With `features.distill` on, the engine trains a cheap model on the study's labels and puts it first in the cascade:

| Key | Default | Meaning |
|---|---|---|
| `trainer` | `sklearn-text` | A trainer from `potato/training`. |
| `min_labels` | `200` | Train only once this many labels exist. |
| `include_silver` | `true` | Also train on machine labels a contract certified, marked silver. |
| `every` | `2.0` | Retrain when the label count has grown by this factor. |

Items in the cert and audit splits never go into the training set, so the distilled model is never tested on labels it was trained on.

## Validation

`potato validate config.yaml` checks the `labeling_engine` block along with the rest of the config, and each error names its key, for example `labeling_engine.sources[1].kind must be one of ...` or `labeling_engine.contracts[0].error is required for selective_risk`. Validation also refuses:

- a contract key the contract kind does not read, such as a typo of `error`;
- a contract on a scheme no cascade source can label, with each source's reason;
- a contract on a scheme the engine cannot score (free text, pairwise);
- a `decision` source whose `endpoint_type` is not a decision model, and an `llm`, `vlm` or `judge` source whose `endpoint_type` is one; and
- `no_worse_than_human` with `humans.overlap_k` below 3.

Validation also warns when `budgets.human_labels` is too small to certify a `selective_risk` contract even if every machine answer were right. Raise the budget, or loosen `error`, until that warning goes away.

## Further reading

- [The labeling engine](/docs/labeling-engine/overview)
- [Labeling engine contracts](/docs/labeling-engine/contracts)
- [Command line, REST API, MCP tools and SDK](/docs/labeling-engine/cli-and-api)
- [AI support](/docs/features/ai-support), for the endpoint types sources share with the rest of Potato

For implementation details, see the [source documentation](https://github.com/davidjurgens/potato/blob/master/docs/labeling-engine/configuration.md).