An annotation tool collects labels. That is the job, and Potato has done it since 2022. But a label is the output of a decision, and for most of the last decade the decision itself has been invisible: you get the answer, not the reasoning, the confidence, or the disagreement that produced it.

Potato 2.7 is built on one idea. An annotation tool should measure how judgments are made, not only record what they were. That applies to both kinds of judge: the humans who annotate, and the AI models being evaluated.

Potato 2.7

Labels with error bars

The psychometrics engine fits an item response theory model as annotations arrive, estimating each annotator's ability and each item's difficulty from the pattern of agreement alone. No gold labels, and no LLM.

Instead of sarcastic (2 of 3 votes), your export says sarcastic, p = 0.94 [0.88 – 0.97]. The probability weighs who voted, not just how many.

yaml

assignment_strategy: psychometric
psychometrics:
  enabled: true
  confidence_threshold: 0.95   # items above this stop consuming budget

Two things fall out of the model. Items whose posterior is already confident stop being served, so redundancy goes where it is actually needed. And when an item's discrimination goes negative, meaning your best annotators are the ones disagreeing with the crowd, that is usually a broken guideline rather than a bad annotator. The dashboard flags it as a likely codebook bug.

The calibration meeting, inside the tool

Every annotation team runs norming sessions, and they nearly always happen over screen-share with the outcome living in somebody's notes. Multiplayer Rooms move that session into Potato and instrument it.

Everyone votes blind. The host reveals. The group argues. Anyone may change their vote, and every change after the reveal is logged against what the majority was at that moment, which is a per-member conformity record. A live Krippendorff's α meter runs over blind votes and current votes at once, so the gap between them tells you what the discussion was actually worth, while it is happening.

Huddle rooms seed themselves from the items your team currently disagrees on. Shadow rooms let trainees watch a senior annotator work, highlights and all.

Quality control with no gold labels

Two features attack the same problem from different directions, and neither one plants a fake item.

Truth Serum asks one extra question after each label: what percentage of other annotators will choose what you chose? Those predictions feed the surprisingly-popular estimator (Prelec, Seung & McCoy 2017, Nature), which beats majority vote precisely where annotation is hard: when a confident crowd is wrong and an informed minority is right. As far as we know, Potato is the first annotation tool to ship peer-prediction scoring.

Boundary Lab shows a minimal counterfactual edit right after a label lands and asks whether the label survives it. Answering takes one click, and ordinary annotation starts producing contrast sets (Gardner et al. 2020) as a by-product. Some probes are meaning-preserving paraphrases, and an annotator who flips on those is telling you something, without a single planted gold item.

Human reasoning, captured verbatim

Think-Aloud Mode lets annotators talk while they work. Speech-to-text runs locally through faster-whisper, so nothing leaves the machine and there is no per-token cost. The transcript is stored verbatim and deliberately un-summarized, because paraphrasing a think-aloud protocol destroys the artifact you were trying to collect.

The point is not only rationales. This is a recording of how a person reasons to a label, which is the human counterpart to the process-reward tooling that segments how a model reasons. Same item, two chains of thought.

Two more, and then the agents

Paper Mode turns a project into a compilable LaTeX dataset report in one command: label distributions, an annotator table, agreement statistics with the right citations, timing, and an honest limitations paragraph with real numbers in it.

bash

python -m potato.paper config.yaml -o paper_export

Pocket Mode makes mobile annotation first-class: touch devices get a swipeable card stack with thumb-zone buttons and offline sync. Tasks that genuinely need a desktop (spans, bounding boxes, video) are never degraded onto a phone. They get a warning instead.

Evaluating AI agents

The other half of 2.7 points the same philosophy at models. The agent-evaluation suite now annotates a multi-agent run as a team rather than a flat transcript: a clickable interaction graph, cross-agent failure attribution, handoff review, per-agent and team scorecards, a tool-contention timeline, and emergent-behavior tagging across agents.

Multimodal agents get their own surfaces: GUI and computer-use trajectories with click grounding, full-duplex voice timelines with barge-in detection, video temporal grounding with live IoU, and document-table structure.

If you want the walkthrough rather than the list, we wrote one on debugging multi-agent failures.

Where the two halves meet

They are the same idea aimed at two different judges, and 2.7 wires them together.

Think-Aloud records how a person reasons to a label. Process-reward annotation captures how a model reasons, step by step. Put them beside each other on the same item and you can see where human and machine judgment diverge.

And an LLM judge is only trustworthy to the extent that the human labels you validated it against are trustworthy. The judge–human alignment dashboard tells you how well your judge agrees with people; psychometrics and Truth Serum give those people's labels confidence intervals. "The judge agrees with humans" stops being a vibe and starts being a claim with an interval attached.

Everything else

Cross-document event annotation, turn-level annotation over conversations and traces, a living-document codebook with a full-page editor, PDF cross-page linking with optional OCR, RBAC roles with per-cohort schemas, and admin and annotator dashboards in ten languages.

The install got lighter, too. AI SDKs now load lazily, so a plain pip install potato-annotation pulls in no AI SDK at all and boot time drops from about 2.0 seconds to 0.7.

Getting it

bash

pip install --upgrade potato-annotation

Potato is free, open-source under GPL-3.0-or-later, and self-hostable. The seven features above are opt-in, task-agnostic, and none of them requires an LLM. That matters if your data cannot leave your institution, or your budget cannot absorb a per-token bill.

Potato 2.0 was published at ACL 2026 (System Demonstrations). If Potato is useful in your research, that is the paper to cite.

Start with the quick start, browse what's new, or read the deep dives on labels with error bars and quality control without gold labels.