Potato 2.7: Measuring How Labels Are Made

Potato 2.7 adds seven opt-in features, and none of them needs an LLM: labels with error bars, live norming rooms, counterfactual boundary probes, peer-prediction scoring, local voice rationales, one-command dataset reports, and mobile annotation. The release also extends agent evaluation to multi-agent and multimodal runs.

An annotation tool collects labels, and Potato has done that since 2022. A label is the output of a decision, though, and for most of the last decade the decision itself has been invisible, because you get the answer and not the reasoning, the confidence, or the disagreement that produced it. Potato 2.7 is built on the idea that an annotation tool should measure how judgments are made as well as record what they were, for both kinds of judge, the humans who annotate and the AI models being evaluated.

Potato 2.7

Labels with error bars

The psychometrics engine fits an item response theory model as annotations arrive, estimating each annotator's ability and each item's difficulty from the pattern of agreement alone, with no gold labels and no LLM. Instead of sarcastic (2 of 3 votes), your export says sarcastic, p = 0.94 [0.88 – 0.97], and the probability weighs who voted as well as how many.

Enable the engine and the psychometric assignment strategy:

yaml

assignment_strategy: psychometric
psychometrics:
  enabled: true
  confidence_threshold: 0.95   # items above this stop consuming budget

Items whose posterior is already confident stop being served, so redundancy goes where it is needed. When an item's discrimination goes negative, meaning your best annotators are the ones disagreeing with the crowd, the cause is usually a broken guideline rather than a bad annotator, and the dashboard flags the item as a likely codebook bug.

Multiplayer Rooms for norming sessions

Every annotation team runs norming sessions, and they nearly always happen over screen-share with the outcome living in somebody's notes. Multiplayer Rooms move that session into Potato and instrument it.

Everyone votes blind, the host reveals, and the group argues. Anyone may change their vote, and Potato logs every change after the reveal against what the majority was at that moment, which is a per-member conformity record. A live Krippendorff's α meter runs over blind votes and current votes at once, so the gap between them tells you what the discussion was worth while it is happening. Huddle rooms seed themselves from the items your team currently disagrees on, and shadow rooms let trainees watch a senior annotator work, highlights and all.

Quality control with no gold labels

Two features approach the same problem from different directions, and neither one plants a fake item. Truth Serum asks annotators after each label what percentage of other annotators will choose what they chose. Those predictions feed the surprisingly-popular estimator of Prelec, Seung and McCoy (2017), which beats majority vote where annotation is hard, in cases where a confident crowd is wrong and an informed minority is right. Potato ships peer-prediction scoring as part of the annotation loop, so the predictions are collected alongside the labels rather than in a separate round.

Boundary Lab shows a minimal counterfactual edit right after a label lands and asks whether the label survives it. Answering takes one click, and ordinary annotation starts producing contrast sets (Gardner et al. 2020) as a by-product. Some probes are meaning-preserving paraphrases, and an annotator who flips on those has shown you a label that was not tracking meaning, without a single planted gold item.

Think-Aloud Mode for spoken reasoning

Think-Aloud Mode lets annotators talk while they work. Speech-to-text runs locally through faster-whisper, so nothing leaves the machine and there is no per-token cost. Potato stores the transcript verbatim and deliberately un-summarized, because paraphrasing a think-aloud protocol destroys the artifact you were trying to collect.

Paper Mode and Pocket Mode

Paper Mode turns a project into a compilable LaTeX dataset report in one command. The report contains label distributions, an annotator table, agreement statistics with the right citations, timing, and a limitations paragraph with real numbers in it.

Run it against a project config:

bash

python -m potato.paper config.yaml -o paper_export

Pocket Mode gives mobile annotation a real interface, a swipeable card stack for touch devices with thumb-zone buttons and offline sync. Tasks that genuinely need a desktop (spans, bounding boxes, video) are never degraded onto a phone and show a warning instead.

Multi-agent and multimodal agent evaluation

The other half of 2.7 points the same philosophy at models. The agent-evaluation suite now annotates a multi-agent run as a team rather than a flat transcript, with a clickable interaction graph, cross-agent failure attribution, handoff review, per-agent and team scorecards, a tool-contention timeline, and emergent-behavior tagging across agents.

Multimodal agents get their own annotation surfaces for GUI and computer-use trajectories with click grounding, full-duplex voice timelines with barge-in detection, video temporal grounding with live IoU, and document-table structure. For a walkthrough, see debugging multi-agent failures.

Connecting human labels to model evaluation

The two halves of 2.7 are the same idea aimed at two different judges, and 2.7 wires them together in two places. First, Think-Aloud Mode records how a person reasons to a label, and process-reward annotation captures how a model reasons, step by step. Put them beside each other on the same item and you can see where human and machine judgment diverge.

Second, an LLM judge is only as trustworthy as the human labels you validated it against. The judge-human alignment dashboard tells you how well your judge agrees with people. Psychometrics puts a confidence interval on each of those people's labels, and Truth Serum adds a surprisingly-popular verdict for each item and a calibration score for each annotator, so "the judge agrees with humans" becomes a claim with an interval attached.

Other additions and a lighter install

Potato 2.7 also adds cross-document event annotation, turn-level annotation over conversations and traces, a living-document codebook with a full-page editor, PDF cross-page linking with optional OCR, role-based access control with per-cohort schemas, and admin and annotator dashboards in ten languages.

AI SDKs now load lazily, so a plain pip install potato-annotation pulls in no AI SDK at all. On an install that does have the SDKs, server boot import time drops from about 2.0 seconds to 0.7, because none of them load at boot anymore.

Installing Potato 2.7

Upgrade with pip:

bash

pip install --upgrade potato-annotation

The full release notes list every change in 2.7.

Potato is free, open source under GPL-3.0-or-later, and self-hostable. The seven features above are opt-in and task-agnostic, and none of them requires an LLM, which matters if your data cannot leave your institution or your budget cannot absorb a per-token bill.

Potato 2.0 was published at ACL 2026 (System Demonstrations), and that paper is the one to cite if Potato is useful in your research.

Start with the quick start, browse what's new, or read the deep dives on labels with error bars and quality control without gold labels.

Thanks to Aldo Costa (@Eyecatch3r), who contributed automatic batch assignment for annotators who arrive without a pre-assigned batch in PR #163.

References

Dražen Prelec, H. Sebastian Seung, and John McCoy (2017). A solution to the single-question crowd wisdom problem. Nature. https://doi.org/10.1038/nature21054

Matt Gardner et al. (2020). Evaluating Models' Local Decision Boundaries via Contrast Sets. Findings of the Association for Computational Linguistics: EMNLP 2020. https://aclanthology.org/2020.findings-emnlp.117/

David Jurgens, Michael Chen, and Lina Iyer (2026). Potato 2.0: A Comprehensive Annotation Platform with AI-in-the-Loop Support. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations). https://aclanthology.org/2026.acl-demo.37/