Blog

How to Collect Process Reward Data for Training Better Coding Agents

Step-by-step guide to collecting per-step reward signals for PRM training using Potato. Covers first-error mode, per-step annotation, and export to training pipelines.

Mar 26, 202611 min read

Watch, Pause, and Rewind: Live Coding Agent Observation in Potato

Tutorial for setting up live coding agent observation with Ollama, Anthropic API, or Claude Agent SDK. Includes pause, rollback, branching, and trajectory export.

Mar 25, 202612 min read

Guides14 min read

Comparing AI Agents Side by Side: Binary, Scale, and Multi-Dimension Modes

Set up pairwise agent comparison in Potato with three modes: binary preference, continuous scale, and per-dimension multi-criteria judgment with required justification.

Mar 24, 2026

Guides15 min read

Per-Step Error Localization: Using Trajectory Evaluation to Find Where Agents Fail

Use Potato's trajectory_eval schema for per-step error localization with hierarchical error taxonomies, severity scoring, and running score tracking across agent traces.

Mar 23, 2026

Tutorials12 min read

GitHub PR-Style Code Review for AI Coding Agents

Set up GitHub PR-style code review annotation in Potato with inline diff comments, file-level quality ratings, and approve or reject verdicts for coding agent output.

Mar 22, 2026

Guides13 min read

Potato vs LangSmith and Langfuse for Agent Evaluation: A Practical Comparison

Compare Potato with LangSmith and Langfuse for AI agent evaluation across trace rendering, annotation, coding agent support, live observation, pricing, and self-hosting.

Mar 21, 2026

Stay updated

Get notified about new tutorials, releases, and community highlights.

MT-Bench-Style Rubric Evaluation for AI Agents in Potato

Set up multi-criteria rubric evaluation with custom criteria, configurable rating scales, and dimension weights for systematic AI agent evaluation using Potato's rubric_eval.

Mar 20, 202616 min read

Release

Potato 2.4.0: Web Agent Annotation, Live Evaluation, and HuggingFace Integration

Potato 2.4.0 ships web agent trace review, real-time live agent evaluation, an LLM chat sidebar, HuggingFace Hub export, webhooks, SSO/OAuth, and five active learning strategies.

Mar 17, 20265 min read

Announcements11 min read

Potato 2.3: Annotazione Agentica, Solo Mode e il Futuro della Valutazione Umana

Potato 2.3.0 introduce l'annotazione agentica con 12 convertitori di formato traccia, Solo Mode per l'etichettatura collaborativa umano-LLM, Best-Worst Scaling, SSO/OAuth, esportazione Parquet e 15 progetti demo.

Mar 9, 2026

Guides10 min read

Valutare gli Agenti AI: Una Guida Completa all'Annotazione Umana delle Tracce degli Agenti

Scopri come configurare la valutazione umana degli output di agenti AI utilizzando le funzionalità di annotazione agentica di Potato, dall'importazione delle tracce alla progettazione degli schemi di annotazione.

Mar 8, 2026

Tutorials12 min read

Solo Mode: Come un Solo Annotatore Può Etichettare 10.000 Esempi

Tutorial passo per passo sull'utilizzo di Solo Mode di Potato per etichettare in modo efficiente grandi dataset con la collaborazione umano-LLM, riducendo il costo dell'annotazione fino al 90%.

Mar 7, 2026

Guides11 min read

Annotare gli Agenti di Navigazione Web: Dalle Tracce WebArena alla Valutazione Umana

Come utilizzare il display delle tracce degli agenti web di Potato per valutare agenti di navigazione web autonomi, con screenshot passo per passo, overlay SVG e schemi di annotazione per passo.

Mar 6, 2026

Potato 2.2: Events, Entity Linking, Export, and 55 Survey Instruments

Potato 2.2.0 adds 9 new annotation schemas, a pluggable export system, MACE competence estimation, 55 validated survey instruments, and remote data sources.

Feb 20, 20265 min read

Potato 2.1: Instance Display, Visual AI, and Span Linking

Potato 2.1.0 brings the instance display system, visual AI support for image and video annotation, span linking, multi-field spans, and layout customization.

Feb 14, 20266 min read

Guides10 min read

Usare l'AI Visiva per Accelerare l'Annotazione di Immagini e Video

Configura il rilevamento di oggetti assistito dall'AI, la pre-annotazione e la classificazione per task su immagini e video con YOLO, Ollama, OpenAI e Claude.

Feb 14, 2026

Guides8 min read

Migrating from Label Studio to Potato

Migrate from Label Studio to Potato, convert project configs, annotation schemas, and exported data with this step-by-step guide covering common annotation types.

Jan 20, 2026

Guides4 min read

Running Annotation Studies on Prolific

Integrate Potato with Prolific for crowdsourced annotation, completion URL setup, participant ID tracking, attention checks, payment configuration, and quality filtering.

Jan 20, 2026

Guides6 min read

Legal Document Annotation Best Practices

Annotate legal documents in Potato, contracts, court filings, and regulatory text, with span labeling, entity extraction, and privacy-first self-hosted deployment.

Jan 18, 2026

Medical Image Annotation with Potato

Best practices for annotating medical images in Potato, DICOM display, radiology report labeling, adverse event extraction, and IRB-compliant self-hosted deployment.

Jan 18, 20265 min read

Polygon Annotation for Segmentation Tasks

Configure polygon drawing tools in Potato for image segmentation tasks, with tips for complex shapes, overlapping regions, multi-class polygons, and export to COCO format.

Jan 18, 20264 min read

Guides3 min read

Active Learning: Annota in Modo Più Intelligente, Non Più Faticoso

Come usare l'uncertainty sampling per prioritizzare le annotazioni e ridurre il lavoro totale di etichettatura fino al 50%.

Tutorials4 min read

Drawing Bounding Boxes for Object Detection

Set up bounding box annotation for computer vision in Potato, configure label colors, minimum box size, multi-class support, validation rules, and COCO/YOLO export.

Guides5 min read

Exporting Annotations to Hugging Face Datasets

Convert Potato annotations to HuggingFace Datasets format, covering JSON export structure, dataset card generation, Hub upload, and integration with transformers training.

Tutorials7 min read

Image Comparison and Preference Tasks

Build side-by-side image comparison tasks in Potato for preference ranking, A/B testing, and visual quality assessment, with randomized order and pairwise scoring.

Introducing Potato 2.0: AI-Powered Annotation

Potato 2.0 ships AI-powered pre-annotation with OpenAI and Claude, multimedia support for audio and video, active learning, bounding box annotation, and a redesigned UI.

Jan 15, 20263 min read

Automatic Keyword Highlighting

Configure AI-powered keyword highlighting in Potato to draw annotator attention to important terms. Covers OpenAI, Claude, and custom keyword list configuration.

Jan 15, 20264 min read

Guides7 min read

Deploying to Amazon Mechanical Turk

Run Potato annotation tasks on Amazon Mechanical Turk, HIT configuration, qualification tests, approval and rejection workflows, bonus payments, and quality monitoring.

Creating a Sentiment Analysis Task

Build a complete sentiment classification task with radio buttons, tooltips, and keyboard shortcuts for efficient labeling.

Guides7 min read

Content Moderation Annotation Setup

Configure Potato for toxicity detection, hate speech classification, and sensitive content labeling with annotator wellbeing in mind.

Jan 12, 2026

Guides5 min read

Custom HTML Templates in Potato

Build custom annotation interfaces in Potato using HTML templates, CSS styling, and JavaScript, side-by-side layouts, custom widgets, and embedded media displays.

Jan 12, 2026

Pronunciation Assessment Annotation

Build a pronunciation quality annotation task in Potato with audio playback, waveform visualization, Likert rating scales, and per-recording free-text feedback fields.

Jan 12, 20266 min read

Speaker Diarization Annotation

Build a speaker identification task in Potato with interactive audio waveforms, timestamp markers, speaker label assignment, and inter-annotator agreement measurement.

Jan 12, 20264 min read

Tutorials3 min read

Video Annotation with Frame-by-Frame Controls

Set up video annotation in Potato with frame-by-frame navigation, timestamp markers, temporal event labeling, and per-segment classification using radio or span schemas.

Jan 12, 2026

Audio Event Detection and Tagging

Set up annotation for detecting specific sounds like speech, music, applause, or environmental noises with timestamp spans.

Jan 10, 2026

Tutorials2 min read

Iniziare con Potato in 5 Minuti

Una guida rapida per configurare il tuo primo progetto di annotazione con Potato, dall'installazione ai tuoi primi dati etichettati.

Jan 10, 2026

Guides6 min read

Measuring Inter-Annotator Agreement

Calculate and interpret Cohen's Kappa, Fleiss' Kappa, and Krippendorff's Alpha for Potato annotation projects, with Python code examples and interpretation guidelines.

Jan 10, 2026

Potato Featured at EMNLP 2022

Our paper on Potato was accepted at EMNLP 2022. Learn about the research behind the tool and how to cite it in your work.

Jan 10, 20263 min read

Quality Control for Crowdsourced Annotation

Best practices for ensuring annotation quality in annotation projects, including practical strategies you can implement with and beyond Potato.

Jan 10, 20267 min read

Tutorials4 min read

Image Classification with Potato

Set up image classification in Potato with thumbnail previews, zoom controls, multi-label checkbox schemas, and keyboard shortcuts for high-throughput visual annotation.

Guides4 min read

Integrare gli LLM per Suggerimenti di Annotazione Intelligenti

Scopri come usare OpenAI, Claude o Gemini per fornire suggerimenti e hint intelligenti ai tuoi annotatori.

Music Genre Classification Annotation

Create a music annotation task in Potato with waveform visualization, 30-second preview playback, hierarchical genre label trees, and pairwise preference comparisons.

Guides6 min read

Deploying Potato on a Server

Deploy Potato to production with Docker, nginx, SSL, and cloud platforms including AWS, GCP, and Azure. Includes systemd service configuration and scaling tips.

Multi-Object Tracking Annotation

An overview of multi-object tracking annotation concepts and how Potato's video annotation capabilities can support basic tracking workflows.

Jan 6, 20264 min read

Classifying Emotions in Speech

Create an audio emotion classification task in Potato with interactive waveform display, playback speed controls, Likert scales, and configurable emotion label sets.

Jan 5, 20266 min read

Tutorials4 min read

Building Your First NER Annotation Task

Step-by-step guide to building a named entity recognition task in Potato, span annotation config, label colors, overlapping spans, keyboard shortcuts, and CoNLL export.

Jan 5, 2026

Guides4 min read

Understanding Potato Data Formats

A deep dive into JSON and JSONL data formats in Potato, structuring instances for text, image, audio, video, and multimodal annotation with real config examples.

Jan 5, 2026