Blog

Measuring How LLM Suggestions Anchor Annotators Using Blind Items

Annotators shown an LLM's label give different labels than those who are not. Hide the suggestion on a random sample and compare blind with shown agreement.

Oct 22, 2026

Inter-Annotator Agreement for 3D Objects, Time Segments and Agent Traces

Reducing structured annotations to label names makes every annotator agree. Match 3D boxes and segments by IoU, and score agent traces one step at a time.

Oct 20, 2026

Certifying LLM Labels with a Blind Human Sample

Keep an LLM's labels only where a blind random human sample shows the error rate is under target, and report prevalence with model-corrected intervals.

Oct 17, 2026

Announcements8 min read

Potato 2.11: The Labeling Engine and Certified Machine Labels

Potato 2.11 labels a dataset with models, tests the machine labels on a blind human sample, and writes a certificate. Idle sessions now sign out after 8 hours.

Oct 15, 2026

AI Auto-Labeling Compared: Label Studio, CVAT, Labelbox, Prodigy, Argilla and Potato

Six annotation tools compared on how model and LLM labels enter a project, how people review them, and what each reports about machine-label error rates.

Oct 13, 20269 min read

Self-Hosted Annotation Tools: What Each Needs to Run and Where to Host It

Label Studio, CVAT, doccano, Argilla, INCEpTION and Potato compared on services, database, sign-in and backups for a self-hosted install, plus hosts by cost.

Oct 9, 20269 min read

Tutorials5 min read

Deploy Buttons: Let Another Lab Run Your Annotation Study

potato deploy button adds a Deploy to Heroku, Render or AWS Launch Stack button to a study's repository, so collaborators run it in their own accounts.

Oct 8, 2026

Announcements7 min read

Potato 2.9: Import from brat, doccano, Prodigy, CoNLL and QDA Tools

Potato 2.9 imports brat, doccano, Prodigy, CoNLL, and REFI-QDA projects, transcribes and diarizes audio locally, and records where each image shape came from.

Sep 13, 2026

Announcements12 min read

potato-skill: Build an Annotation Study with Claude Code, Codex, or Cursor

potato-skill gives a coding agent 28 reference files for Potato, so it designs the annotation study, builds the interface, and checks that it renders.

Sep 12, 2026

Agreement Cannot Catch Rubber-Stamping

When annotators accept model pre-labels without reading them, agreement rises and adjudication falls. Timing records separate review from rubber-stamping.

Sep 9, 2026

Stay updated

Get notified about new tutorials, releases, and community highlights.

Partial Is a Real Outcome: Annotating Robot Episodes

Many robot episodes end in partial success, and binary labels discard it. Annotation needs a timeline, min/max downsampling, and relabeling.

Sep 2, 20265 min read

Shipping Segmentation That Runs in a Browser Tab

Click-to-segment and text prompting run in the browser with no GPU, using a verified encoder contract, a measured quantization, and a hand-written tokenizer.

Aug 27, 20265 min read

Raw IoU Is Not Agreement

Mean IoU between annotators can stay near 0.95 even for one who never looks. Potato reports σ, which estimates the chance baseline from other images.

Aug 21, 2026

Announcements9 min read

Potato 2.8: Computer Vision, 3D, Robot Episodes, and Spatial Agreement

Potato 2.8 adds computer vision, deep zoom, 3D point clouds, depth maps, robot episodes, and video-model evaluation, with chance-corrected agreement for each.

Aug 18, 2026

Announcements10 min read

Reading the Writing Process: Keystroke Logging for Free-Text Annotation

Potato records how annotators produce free-text answers without recording what they type, and flags responses that were pasted rather than written.

Aug 6, 2026

Tutorials7 min read

Annotating ASR Transcripts: A Worked Example

A walkthrough from a folder of Whisper output to labeled speaker turns, covering the annotation unit, diarization, the config, and export with timings intact.

Jul 31, 2026

Judging the Judge: Human and Model Reasoning, Side by Side

An LLM judge is only as trustworthy as the human labels it was checked against. Give those labels intervals, and compare human and model reasoning per item.

Jul 27, 20267 min read

Announcements

Potato 2.7.1: The Transcript Already Exists

Potato 2.7.1 reads 21 transcript and subtitle formats, loads them from sidecar files next to your media, and converts a folder of ASR output to a data file.

Jul 26, 20266 min read

Instrumenting the Calibration Meeting

Multiplayer Rooms measure a norming session with blind votes and a live agreement meter. Think-Aloud Mode records spoken reasoning locally, with no LLM.

Jul 23, 2026

Quality Control Without Gold Labels

Peer-prediction scoring and counterfactual boundary probes catch inattentive annotators and broken guidelines without planting a single gold item.

Jul 19, 2026

Labels With Error Bars: Item Response Theory for Annotation

Majority vote throws away most of what annotators told you. Item response theory gives every label a posterior probability and a sensitivity band.

Jul 16, 2026

Announcements7 min read

Potato 2.7: Measuring How Labels Are Made

Potato 2.7 adds seven opt-in features that need no LLM, including labels with error bars and norming rooms, plus multi-agent and multimodal agent evaluation.

Jul 13, 2026

Validated Survey Instruments for Annotation Studies: Personality, Affect, Wellbeing, and Demographics

When who annotates matters, use a validated questionnaire rather than one you invented. Potato ships 55 survey instruments for prestudy and poststudy phases.

Jul 4, 20267 min read

Documenting Your Annotation Dataset: Data Statements, Datasheets, and a Release Checklist

Record the curation rationale, annotator pool, guidelines, and intended use when you release an annotation dataset, and see how much of it Potato captures.

Jun 30, 20266 min read

Announcements3 min read

Potato 2.0 at ACL 2026

The Potato 2.0 paper is in the ACL 2026 System Demonstrations. It covers 39 task types, labeling of AI agent outputs, and AI-in-the-loop annotation.

Jun 28, 2026

Evaluating Voice and Video Agents

How to run human evaluation of voice, video, and document agents in Potato, from turn-taking timelines and live IoU to speech-error tags and table cells.

Jun 24, 2026

Evaluating Computer-Use Agents, Step by Step

Evaluate computer-use and GUI agents step by step in Potato by judging each action, checking click grounding on the screenshot, and reviewing each tool call.

Jun 23, 2026

Debugging Multi-Agent Failures: A Walkthrough

How to find why a multi-agent LLM system failed using Potato: the interaction graph, failure attribution, handoff review, per-agent scorecards, tool-contention timeline, and emergent-behavior tagging.

Jun 22, 2026

Announcements

Potato 2.6: Qualitative Data Analysis Meets Agent Evaluation

Potato 2.6 is out: QDA Mode for qualitative coding, an LLM-as-judge calibration and alignment workflow, trajectory editing that produces SFT and DPO training data, a 3x faster boot, and a relicense to GPL-3.0-or-later.

Jun 21, 20266 min read

Announcements

Potato 2.6.2: A Complete Open-Source Agent-Evaluation Suite

The 2.6.x line turns Potato into a full, free agent-evaluation platform: trace ingestion from OpenTelemetry, LangGraph, CrewAI, and AutoGen, multi-agent team annotation with a clickable interaction graph, multimodal-agent schemas for GUI, voice, and video, plus a model arena, CI gating, and curation.

Jun 20, 20265 min read

Disagreement Is Signal, Not Noise: When to Keep Annotator Disagreement Instead of Resolving It

Annotation pipelines are built to erase disagreement, but on subjective tasks the disagreement is the data. A guide to telling genuine label variation from error, and keeping it in Potato.

Jun 18, 2026

Adaptive Annotator Coverage for Large Datasets

Potato 2.6 lets you assign one annotator to most items and three to a stratified sample, raise coverage on items where annotators disagree, and route the contested ones to an adjudicator.

Jun 16, 2026

Closing the Loop: Routing Agent Errors and Judge Disagreements Back to Humans

Human review time is the scarcest resource in agent evaluation. Potato 2.6 pairs a signal-based triage queue with judge-human alignment so the worst traces reach people first and your LLM judge keeps getting better.

Jun 11, 2026

Collecting Annotator Demographics Responsibly: What to Ask and How to Ask It

Annotator identity shapes labels on subjective tasks, so demographics are worth collecting, but only with consent and a reason. What to ask, what to leave alone, and how to run the flow in Potato.

Jun 11, 2026

From Evaluation to Training Data: Trajectory Editing for SFT and DPO

Most agent evaluation stops at a score. Potato 2.6's trajectory_edit schema lets annotators rewrite a wrong step instead of rating it, and exports each correction as supervised fine-tuning targets and DPO preference pairs.

Jun 6, 20266 min read

How to Get Reliable Labels on Agent Trajectories

Annotating an agent's multi-step trace is harder than labeling a tweet. A guide to designing the taxonomy, measuring step-level agreement, and adjudicating disagreement, with a Potato config.

Jun 3, 20266 min read

Can You Trust Your LLM Judge? Calibrating LLM-as-Judge Against Humans

Using an LLM to grade model outputs is easy. Knowing whether to believe it is the hard part. A walk through Potato 2.6's blind human calibration: k-sample voting, Cohen's and Fleiss' kappa, and expected calibration error.

Jun 1, 2026

Bringing Qualitative Coding to Potato: Codebooks, Memos, and In-Vivo Codes

A look at QDA Mode, the upcoming Potato 2.6 workspace for qualitative data analysis: a living codebook, in-vivo coding, analyst memos, cases, and full-text search over a whole corpus.

May 26, 2026

LLM Annotators vs Humans: When to Automate a Labeling Job and When Not To

A practical guide to deciding when an LLM can annotate your data, where model annotators fail, and how to combine automation with human verification in Potato.

May 25, 2026

Guides3 min read

Choosing an Open-Source Annotation Tool in 2026

How to pick an open-source data annotation tool, the questions that narrow the choice quickest, and where Potato fits among Label Studio, Prodigy, Doccano, brat, and Argilla.

May 20, 2026

Codebooks for AI Annotators: Turning a Coding Scheme into a Reliable LLM Labeler

How to write an annotation codebook an LLM can actually follow, validate its labels against human coders, and keep a person in the loop, with a worked Potato config.

May 15, 20268 min read

Finding Hallucinations with Span Annotation

Catch model hallucinations and factual errors by highlighting the exact words and labeling what's wrong, MQM-style, with span annotation in Potato.

May 6, 20262 min read

Guides2 min read

How Many Annotators Do You Actually Need?

Deciding annotator count and overlap for an annotation project: rules of thumb for objective and subjective tasks, the coverage-versus-overlap tradeoff, and how to set it in Potato.

Apr 22, 2026

Guides3 min read

How to Evaluate RAG Systems with Human Annotation

A practical guide to evaluating retrieval-augmented generation: score retrieval relevance and answer faithfulness separately, and mark unsupported claims with span annotation in Potato.

Apr 8, 2026

Announcements10 min read

Coding Agent Annotation: Evaluating Claude Code, Aider, and SWE-Agent Traces

Potato renders coding agent traces as diffs and terminal blocks, with process reward and code review schemas and converters for Claude Code, Aider, SWE-Agent.

Mar 27, 2026

How to Collect Process Reward Data for Training Better Coding Agents

Collect per-step reward labels for PRM training in Potato, with first-error and per-step modes and a script to shape the export for training.

Mar 26, 2026

Tutorials

Watch, Pause, and Rewind: Live Coding Agent Observation in Potato

Tutorial for setting up live coding agent observation with Ollama, Anthropic API, or Claude Agent SDK. Includes pause, rollback, branching, and trajectory export.

Mar 25, 202611 min read

Comparing AI Agents Side by Side: Binary, Scale, and Multi-Dimension Modes

Set up pairwise agent comparison in Potato across its three modes — binary preference, a preference scale, and per-dimension judgment with required justification.

Mar 24, 202610 min read

Guides13 min read

Per-Step Error Localization: Using Trajectory Evaluation to Find Where Agents Fail

Use Potato's trajectory_eval schema for per-step error localization with a hierarchical error taxonomy, severity weights, and a running quality score across agent traces.

Mar 23, 2026

Tutorials11 min read

GitHub PR-Style Code Review for AI Coding Agents

Set up GitHub PR-style code review annotation in Potato with inline diff comments, file-level quality ratings, and approve or reject verdicts for coding agent output.

Potato 2.1.0 brings the instance display system, visual AI support for image and video annotation, span linking, multi-field spans, and layout customization.

Feb 14, 20266 min read

Using Visual AI to Speed Up Image and Video Annotation

Configure YOLO, Ollama, OpenAI, and Claude vision endpoints in Potato so a model proposes boxes and labels that annotators accept or reject.

Feb 14, 202612 min read

Migrating from Label Studio to Potato

Migrate from Label Studio to Potato, convert project configs, annotation schemas, and exported data with this step-by-step guide covering common annotation types.

Jan 20, 2026

Running Annotation Studies on Prolific

Run a Potato annotation study on Prolific with URL-direct login, a completion code, attention checks, and API-side submission monitoring.

Jan 20, 2026

Guides9 min read

Legal Document Annotation Best Practices

Annotate contracts, court filings and regulatory text in Potato: section-level items, clause and entity schemes, keyword highlighting, and quality control for expert annotators.

Jan 18, 2026

Medical Image Annotation with Potato

Best practices for annotating medical images in Potato, DICOM display, radiology report labeling, adverse event extraction, and IRB-compliant self-hosted deployment.

Jan 18, 2026

Tutorials

Polygon Annotation for Segmentation Tasks

Build a polygon segmentation task in Potato: the config, the drawing keys annotators use, the normalized output format, and when a brush mask is the better tool.

Jan 18, 20267 min read

Active Learning in Potato: Prioritizing the Items Worth Labeling

How Potato reorders the annotation queue by classifier uncertainty, with config for the sampling settings, the cold start, and keeping enough random items in the mix.

Jan 15, 20264 min read

Tutorials7 min read

Drawing Bounding Boxes for Object Detection

Build an object detection task in Potato, from a three-label starter config to a production config for driving footage, then export to COCO or YOLO.

Jan 15, 2026

Exporting Annotations to Hugging Face Datasets

Push Potato annotations to the Hugging Face Hub with one export command, then reshape the exported splits for classification, NER, audio, and image training.

Configure Potato for toxicity detection, hate speech classification, and sensitive content labeling with annotator wellbeing in mind.

Jan 12, 2026