# Labeling Engine CLI, REST API, MCP Tools and SDK

Source: https://www.potatoannotator.com/docs/labeling-engine/cli-and-api

The annotation server, the command line, the REST API, the MCP tools and the Python SDK all drive the same [labeling engine](/docs/labeling-engine/overview) over the same `engine.sqlite` file in the task directory. You can start a study from the command line, serve the human requests with the server, and certify from a notebook.

Only one process drives a study at a time. A running server's planner holds the study's writer lock, and while it does, `potato label run`, `potato label certify` and `Study.run()` refuse and say so. Certify through the server instead, with `POST /api/engine/v1/certify` or the admin page. The lock exists because two drivers would draw two calibration samples and certify against each other's labels. The lock is released when its holder exits. A holder killed on the same machine is taken over at once, and one on another machine after two minutes without a heartbeat. Read-only commands (`status`, `certificate`, `calls`, `export`) work at any time.

## Command line

Every engine command takes the study's config file:

```bash
potato label <command> <config.yaml> [options]
```

The nine commands cover a study from the first model call to the exported dataset:

| Command | What it does |
|---|---|
| `run` | Label every item with the cascade, draw the blind calibration sample, certify, resolve. `--until predicted` (the default) stops after one pass, and `--until met` keeps going until every contract is met or the engine is waiting on people. Also takes `--max-items N`, `--budget-usd X` (overrides `budgets.model_usd` for this run), `--max-ticks N` (20 by default) and `--json`. A killed run resumes where it stopped without repeating model calls. Exits with status 1, naming the source, when a source's last run failed or every answer it gave was an error, or when no source can label a scheme a contract names. |
| `status` | Items, predictions, requests by purpose, contracts, spend, items with no label yet, and a line for each source with errors or a failed or paused run. `--json`. |
| `certify` | Test every contract on new cert labels and new predictions. When only dev labels changed, the stored result comes back with a note, and `--retest` tests the rule dev now picks, at the next look (see [Looks](/docs/labeling-engine/contracts#looks)). `--json`. |
| `certificate` | The quality certificate. `--format md` (the default), `html` or `json`, and `--out FILE`. |
| `export` | Labels with provenance, the certificate (JSON, Markdown, HTML), a dataset card and the call log, into `--out DIR` (`<output_annotation_dir>/labeling_engine` by default). |
| `calls` | The model-call audit log as JSON lines. `--out FILE`. |
| `distill` | Train a model on the labels so far and put it first in the cascade. |
| `optimize` | Score every subset of the cascade on dev labels and report the cheapest that keeps coverage. When no subset keeps any coverage, it recommends none. `--prompts N` also asks an LLM tier for N paraphrases of its prompt and scores each on the dev items. Those are model calls, logged and budgeted, and stored under their own source versions, while the current prompt is scored from its stored predictions. The cascade and the certificate are not changed. |
| `migrate-config` | Write a copy of the config with a `labeling_engine` block equivalent to its older AI keys. `--out FILE`. See [Migrating to the labeling engine](/docs/labeling-engine/migration). |

Checking progress spends nothing, because `status`, `certificate` and the admin page read stored results. Each new test, whether from `certify` on new labels or from `certify --retest`, spends part of a contract's error budget.

### Files written by export

`potato label export` writes four kinds of file into the output directory:

| File | Contents |
|---|---|
| `labels.jsonl`, `labels.csv` | One row per item and scheme, labeled or not, with `id`, `schema`, `label`, `label_origin` (`auto`, `human`, `adjudicated`, ...), `status` (`accepted`, `accepted_uncertified` under `threshold_legacy`, `pending`, `pending_recert`), `source`, `model`, `source_version`, `confidence`, `contract_version` and `request_ids`. A machine label whose source has changed since it was accepted reads `pending_recert` until the next pass re-certifies it. |
| `certificate.json`, `.md`, `.html` | The certificate. |
| `README.md` | A dataset card with Hugging Face front matter, built from the certificate. |
| `model_calls.jsonl` | Every model call, with the model, hashes of prompt and reply, tokens, latency, cost, cache hits and redaction categories. |

The regular exporters (`python -m potato.export` and the admin export) export people's annotations, and the [export formats](/docs/features/export-formats) page lists them. Among them, the `labeling_engine` format writes the same files as `potato label export`, and it refuses to run when the engine is not enabled. When the engine is on, CSV, Parquet and Hugging Face exports add two columns that say which engine request each annotation answered. `engine_purpose` holds the purpose, for example `calibration`, and `engine_request` holds JSON with the purpose, inclusion probability, visibility, split and request id. Use these columns to separate the blind random sample from verify answers.

## REST API

All engine routes are under `/api/engine/v1` and return JSON. Authenticate with the admin key in the `X-API-Key` header or with an admin session. Reads need the admin-dashboard permission, `certify` and `plan` need assignment management, and the certificate needs export permission.

| Method and path | Returns |
|---|---|
| `GET /status` | Items, predictions, requests, call totals, budgets, and each scheme's cascade, contracts and resolved-label counts |
| `GET /schemes/<name>/contracts` | The latest result of each contract on the scheme, marked `stale` if the cascade changed |
| `GET /schemes/<name>/risk_coverage` | For each threshold of the last tier, the share accepted, the observed error and the guaranteed upper bound, on cert labels |
| `GET /schemes/<name>/reliability` | Reliability bins, ECE and Brier score per tier, on calibration labels |
| `GET /schemes/<name>/drift` | The audit monitor per selective-risk contract |
| `GET /schemes/<name>/optimize` | The cheaper-cascade report, scored on dev labels |
| `GET /requests` | Request counts by purpose and status, and the requests themselves. Filters are `purpose`, `status` (repeat the parameter or separate values with commas), `schema` and `limit` (max 1000) |
| `GET /calls` | The call log, newest first, and totals by source. `limit` (max 500), `offset` |
| `GET /certificate` | The certificate. `format=json` (the default), `md` or `html`, and `save=1` also stores it |
| `GET /crowd` | The planner's last crowd plan, a dry run unless live spend is on |
| `POST /certify` | Test every contract now. `?retest=1` (or `{"retest": true}`) also tests a rule that only dev labels changed |
| `POST /plan` | Run one planner pass now |

The routes answer 404 when the engine is off for the study. They answer 400 with a JSON error for a parameter that is not valid, such as a `limit` that is not a number, an unknown `purpose` or `status`, or a `format` other than `json`, `md` or `html`.

For example, this request fetches the current certificate as Markdown from a server on port 8000:

```bash
curl -H "X-API-Key: $ADMIN_API_KEY" \
  "http://localhost:8000/api/engine/v1/certificate?format=md"
```

## MCP tools

The MCP server exposes five engine tools, with the same permissions as the REST routes:

- `engine_status`
- `engine_requests`
- `engine_run`, which runs one planner pass
- `engine_certify`, which takes `retest`
- `engine_certificate`, in `json`, `md` or `html`

Granting these tools while the engine is off is a validation warning. Through `potato mcp connect`, a refused call (a missing permission, a bad parameter) comes back as a tool error that carries the server's message. The [MCP server documentation](https://github.com/davidjurgens/potato/blob/master/docs/tools/mcp_server.md) covers granting tools and connecting a client.

## Python SDK

The `Study` class drives the engine from Python, for example in a notebook:

```python
from potato.labeling_engine.sdk import Study

study = Study("examples/labeling-engine/text-certified-sentiment/config.yaml")
study.run()                                   # label, sample, certify, resolve
for req in study.requests(status="open")[:5]:
    study.label(req.item_id, req.schema, "positive", user="me")
study.certify()
print(study.certificate(fmt="md"))
study.export("out/")
```

Each method mirrors a command or a route:

| Method | Does |
|---|---|
| `Study(config_path)` | Load the config and items |
| `run(until="predicted", max_ticks=20)` | As `potato label run`, and takes the writer lock |
| `label(item_id, scheme, value, user, request_id=None)` | Record a person's label. It closes `request_id`, or else a request this person holds on the item, and otherwise it is kept as an organic label, which never certifies. A second label from the same person on the same item updates their first |
| `status()` | As `GET /status` |
| `requests(status=None, purpose=None, ...)` | The request ledger |
| `certify(retest=False)` | As `potato label certify` |
| `certificate(fmt="json")` | `json`, `md` or `html` |
| `export(out_dir)` | As `potato label export` |

The example labels every open request "positive" only to show the calls. A label recorded through `Study.label` that answers no request is kept as an organic label, and organic labels never certify, so a script that labels items the engine did not request adds labels that no contract reads.

## Further reading

- [The labeling engine](/docs/labeling-engine/overview)
- [Labeling engine configuration](/docs/labeling-engine/configuration)
- [Labeling engine contracts](/docs/labeling-engine/contracts)
- [Admin dashboard](/docs/features/admin-dashboard), which links to the engine's admin page

For implementation details, see the [source documentation](https://github.com/davidjurgens/potato/blob/master/docs/labeling-engine/cli-and-api.md).