News, tutorials, and updates from the Potato team.
Potato 2.9 imports brat, doccano, Prodigy, CoNLL, and REFI-QDA projects, transcribes and diarizes audio locally, and records where each image shape came from.
Mean IoU between annotators can stay near 0.95 even for one who never looks. Potato reports σ, which estimates the chance baseline from other images.
Potato 2.8 adds computer vision, deep zoom, 3D point clouds, depth maps, robot episodes, and video-model evaluation, with chance-corrected agreement for each.
Get notified about new tutorials, releases, and community highlights.
Multiplayer Rooms measure a norming session with blind votes and a live agreement meter. Think-Aloud Mode records spoken reasoning locally, with no LLM.
Peer-prediction scoring and counterfactual boundary probes catch inattentive annotators and broken guidelines without planting a single gold item.
The Potato 2.0 paper is in the ACL 2026 System Demonstrations. It covers 39 task types, labeling of AI agent outputs, and AI-in-the-loop annotation.
How to run human evaluation of voice, video, and document agents in Potato, from turn-taking timelines and live IoU to speech-error tags and table cells.
Annotation pipelines are built to erase disagreement, but on subjective tasks the disagreement is the data. A guide to telling genuine label variation from error, and keeping it in Potato.
Potato 2.6 lets you assign one annotator to most items and three to a stratified sample, raise coverage on items where annotators disagree, and route the contested ones to an adjudicator.
Using an LLM to grade model outputs is easy. Knowing whether to believe it is the hard part. A walk through Potato 2.6's blind human calibration: k-sample voting, Cohen's and Fleiss' kappa, and expected calibration error.
A look at QDA Mode, the upcoming Potato 2.6 workspace for qualitative data analysis: a living codebook, in-vivo coding, analyst memos, cases, and full-text search over a whole corpus.
Deciding annotator count and overlap for an annotation project: rules of thumb for objective and subjective tasks, the coverage-versus-overlap tradeoff, and how to set it in Potato.
A practical guide to evaluating retrieval-augmented generation: score retrieval relevance and answer faithfulness separately, and mark unsupported claims with span annotation in Potato.
Use Potato's trajectory_eval schema for per-step error localization with a hierarchical error taxonomy, severity weights, and a running quality score across agent traces.
Set up GitHub PR-style code review annotation in Potato with inline diff comments, file-level quality ratings, and approve or reject verdicts for coding agent output.
How to set up human evaluation of AI agent output in Potato, from trace import through trace-level and per-step schema design to analyzing the exported results.
Step-by-step tutorial on Potato's Solo Mode: label a seed set yourself, let an LLM label the rest, and use the accuracy checks to decide when it is good enough to stop.
Migrate from Label Studio to Potato, convert project configs, annotation schemas, and exported data with this step-by-step guide covering common annotation types.
Run a Potato annotation study on Prolific with URL-direct login, a completion code, attention checks, and API-side submission monitoring.
Build an object detection task in Potato, from a three-label starter config to a production config for driving footage, then export to COCO or YOLO.
Push Potato annotations to the Hugging Face Hub with one export command, then reshape the exported splits for classification, NER, audio, and image training.
Build a complete sentiment classification task with radio buttons, tooltips, and keyboard shortcuts for efficient labeling.
Configure Potato for toxicity detection, hate speech classification, and sensitive content labeling with annotator wellbeing in mind.
Set up annotation for detecting specific sounds like speech, music, applause, or environmental noises with timestamp spans.
Set up your first Potato annotation project in 5 minutes, from pip install to a running server with a YAML config, sample data, and your first labeled instances.
Build a music tagging task in Potato with audio playback, track metadata on screen, multi-label genre tagging, and Likert scales for energy and danceability.
Run Potato in production behind nginx with TLS, in Docker or on a VM, and why a Potato server runs as a single process rather than several workers.
Configure an audio transcription review task in Potato with waveform visualization, variable-speed playback, and inline text correction interfaces for ASR quality evaluation.
Our paper on Potato was accepted at EMNLP 2022. Learn about the research behind the tool and how to cite it in your work.