Why Potato?

Self-hosted, free: With LangSmith, Galileo, Encord, and Labelbox, self-hosting is itself an enterprise contract. On-premise is an add-on you negotiate with sales rather than an edition you can download.

Every feature, no paid tier: Label Studio's free edition has no agreement metrics of any kind. The hosted Starter plan, at $99 a month plus $49 per additional user, has limited agreement metrics, LLM pre-labeling and benchmarks, and the full versions are Enterprise features, priced on request. CVAT gives away honeypots and ground-truth jobs but keeps its quality reports for paying teams.

One tool for corpora and agent traces: Every agent-evaluation platform here is locked to traces. Ask one of them to label a plain text corpus for sentiment and it has nowhere to put the answer.

Multi-agent graph you can annotate: Four of these tools draw an agent interaction graph, and all four are read-only. In Potato the agents and the handoffs between them are the units a person annotates.

Human-labeled failure attribution: Galileo guesses at handoff failures with an LLM, and Labelbox lets you score trajectory steps. Neither one collects the human ground truth you would check that guess against.

Chance-corrected agreement (κ, α): The others report agreement as percent overlap, IoU, or a majority vote. Prodigy is the exception and ships Krippendorff's α and Gwet's AC2, but it has no free tier at all. Langfuse's Cohen's κ compares one human against one judge, not a pool of annotators against each other.

SFT, DPO, and process-reward export: Partial means SFT-style JSONL only, with no preference-pair or process-reward format.

Chance-corrected agreement over geometry: We have found no other annotation platform that reports chance-corrected agreement over spatial labels. CVAT's consensus engine and V7's consensus stage both compare annotators with raw IoU against a per-class threshold, and Label Studio Enterprise's metric list is exact match, numeric difference, IoU and span overlap. None of them corrects for chance, which matters because mean IoU can read about 0.95 on a corpus of large centered objects even for an annotator who never looks at the image.

Agreement on when an event happens: Temporal IoU and boundary agreement, with the matching tolerance reported as a sweep rather than at one threshold. Agreement at 0.25 s and at 2 s are different claims about the same data.

Drawing telemetry (accept latency, revisions): Time per shape, stroke dynamics, revision counts and AI-suggestion accept latency. This matters because agreement goes up when annotators rubber-stamp pre-labels, so timing records separate review from acceptance. No coordinates are recorded.

3D point clouds and cuboids: CVAT also does this, and the 3D specialists (Segments.ai, Kognic, Supervisely) go deeper, with point-cloud sequences and track propagation Potato does not have. Potato's addition is exact rotated 3D IoU between annotators.

Runs with no outbound requests: Every stylesheet, script, font and icon serves from the install across all 14 templates, verified live at 62 requests and zero external. Models are a one-time transfer. Partial means self-hostable, but the rendered page still fetches third-party assets.

Managed annotation workforce: We don't have one. You bring your own annotators or recruit through Prolific. If you need 50,000 items labeled by next month and nobody to label them, Labelbox, Encord, and Scale sell exactly that, and we don't.

Hosted, nothing to run: Potato is self-hosted only. For a lab with a spare server that's the whole appeal. If you don't want to run anything at all, it's a real cost and you should weigh it.