Labeling Engine CLI, REST API, MCP Tools and SDK

The annotation server, the command line, the REST API, the MCP tools and the Python SDK all drive the same labeling engine over the same engine.sqlite file in the task directory. You can start a study from the command line, serve the human requests with the server, and certify from a notebook.

Only one process drives a study at a time. A running server's planner holds the study's writer lock, and while it does, potato label run, potato label certify and Study.run() refuse and say so. Certify through the server instead, with POST /api/engine/v1/certify or the admin page. The lock exists because two drivers would draw two calibration samples and certify against each other's labels. The lock is released when its holder exits. A holder killed on the same machine is taken over at once, and one on another machine after two minutes without a heartbeat. Read-only commands (status, certificate, calls, export) work at any time.

Command line

Every engine command takes the study's config file:

potato label <command> <config.yaml> [options]

The nine commands cover a study from the first model call to the exported dataset:

Checking progress spends nothing, because status, certificate and the admin page read stored results. Each new test, whether from certify on new labels or from certify --retest, spends part of a contract's error budget.

Files written by export

potato label export writes four kinds of file into the output directory:

The regular exporters (python -m potato.export and the admin export) export people's annotations, and the export formats page lists them. Among them, the labeling_engine format writes the same files as potato label export, and it refuses to run when the engine is not enabled. When the engine is on, CSV, Parquet and Hugging Face exports add two columns that say which engine request each annotation answered. engine_purpose holds the purpose, for example calibration, and engine_request holds JSON with the purpose, inclusion probability, visibility, split and request id. Use these columns to separate the blind random sample from verify answers.

REST API

All engine routes are under /api/engine/v1 and return JSON. Authenticate with the admin key in the X-API-Key header or with an admin session. Reads need the admin-dashboard permission, certify and plan need assignment management, and the certificate needs export permission.

The routes answer 404 when the engine is off for the study. They answer 400 with a JSON error for a parameter that is not valid, such as a limit that is not a number, an unknown purpose or status, or a format other than json, md or html.

For example, this request fetches the current certificate as Markdown from a server on port 8000:

curl -H "X-API-Key: $ADMIN_API_KEY" \
  "http://localhost:8000/api/engine/v1/certificate?format=md"

MCP tools

The MCP server exposes five engine tools, with the same permissions as the REST routes:

• engine_status
• engine_requests
• engine_run, which runs one planner pass
• engine_certify, which takes retest
• engine_certificate, in json, md or html

Granting these tools while the engine is off is a validation warning. Through potato mcp connect, a refused call (a missing permission, a bad parameter) comes back as a tool error that carries the server's message. The MCP server documentation covers granting tools and connecting a client.

Python SDK

The Study class drives the engine from Python, for example in a notebook:

from potato.labeling_engine.sdk import Study
 
study = Study("examples/labeling-engine/text-certified-sentiment/config.yaml")
study.run()                                   # label, sample, certify, resolve
for req in study.requests(status="open")[:5]:
    study.label(req.item_id, req.schema, "positive", user="me")
study.certify()
print(study.certificate(fmt="md"))
study.export("out/")

Each method mirrors a command or a route:

The example labels every open request "positive" only to show the calls. A label recorded through Study.label that answers no request is kept as an organic label, and organic labels never certify, so a script that labels items the engine did not request adds labels that no contract reads.

Further reading

• The labeling engine
• Labeling engine configuration
• Labeling engine contracts
• Admin dashboard, which links to the engine's admin page

For implementation details, see the source documentation.