Blog

How to Collect Process Reward Data for Training Better Coding Agents

Step-by-step guide to collecting per-step reward signals for PRM training using Potato. Covers first-error mode, per-step annotation, and export to training pipelines.

Mar 26, 202611 min read

Watch, Pause, and Rewind: Live Coding Agent Observation in Potato

Tutorial for setting up live coding agent observation with Ollama, Anthropic API, or Claude Agent SDK. Includes pause, rollback, branching, and trajectory export.

Mar 25, 202612 min read

Guides14 min read

Comparing AI Agents Side by Side: Binary, Scale, and Multi-Dimension Modes

Set up pairwise agent comparison in Potato with three modes: binary preference, continuous scale, and per-dimension multi-criteria judgment with required justification.

Mar 24, 2026

Guides15 min read

Per-Step Error Localization: Using Trajectory Evaluation to Find Where Agents Fail

Use Potato's trajectory_eval schema for per-step error localization with hierarchical error taxonomies, severity scoring, and running score tracking across agent traces.

Mar 23, 2026

Tutorials12 min read

GitHub PR-Style Code Review for AI Coding Agents

Set up GitHub PR-style code review annotation in Potato with inline diff comments, file-level quality ratings, and approve or reject verdicts for coding agent output.

Mar 22, 2026

Guides13 min read

Potato vs LangSmith and Langfuse for Agent Evaluation: A Practical Comparison

Compare Potato with LangSmith and Langfuse for AI agent evaluation across trace rendering, annotation, coding agent support, live observation, pricing, and self-hosting.

Mar 21, 2026

Stay updated

Get notified about new tutorials, releases, and community highlights.

MT-Bench-Style Rubric Evaluation for AI Agents in Potato

Set up multi-criteria rubric evaluation with custom criteria, configurable rating scales, and dimension weights for systematic AI agent evaluation using Potato's rubric_eval.

Mar 20, 202616 min read

Release

Potato 2.4.0: Web Agent Annotation, Live Evaluation, and HuggingFace Integration

Potato 2.4.0 ships web agent trace review, real-time live agent evaluation, an LLM chat sidebar, HuggingFace Hub export, webhooks, SSO/OAuth, and five active learning strategies.

Mar 17, 20265 min read

Announcements9 min read

Potato 2.3: Agentische Annotation, Solo Mode und die Zukunft menschlicher Evaluierung

Potato 2.3.0 führt agentische Annotation mit 12 Trace-Format-Konvertern, Solo Mode für menschlich-LLM-kollaboratives Labeling, Best-Worst Scaling, SSO/OAuth, Parquet-Export und 15 Demo-Projekte ein.

Mar 9, 2026

Guides9 min read

KI-Agenten evaluieren: Ein vollständiger Leitfaden zur menschlichen Annotation von Agenten-Traces

Erfahren Sie, wie Sie mit Potatos agentischen Annotationsfunktionen eine vollständige Evaluierungspipeline für KI-Agenten einrichten – von der Trace-Importierung bis zum Annotationsschema-Design.

Mar 8, 2026

Tutorials11 min read

Solo Mode: Wie ein Annotator 10.000 Beispiele beschriften kann

Schritt-für-Schritt-Tutorial zur effizienten Beschriftung großer Datensätze mit Potatos Solo Mode und menschlich-LLM-Kollaboration – Annotationskosten um bis zu 90 % reduzieren.

Mar 7, 2026

Guides10 min read

Web-Browsing-Agenten annotieren: Von WebArena-Traces zur menschlichen Evaluierung

Wie man Potatos Web-Agent-Trace-Anzeige nutzt, um autonome Web-Browsing-Agenten mit schrittweisen Screenshots, SVG-Overlays und schrittweisen Annotationsschemata zu evaluieren.

Mar 6, 2026

Potato 2.2: Events, Entity Linking, Export, and 55 Survey Instruments

Potato 2.2.0 adds 9 new annotation schemas, a pluggable export system, MACE competence estimation, 55 validated survey instruments, and remote data sources.

Feb 20, 20265 min read

Potato 2.1: Instance Display, Visual AI, and Span Linking

Potato 2.1.0 brings the instance display system, visual AI support for image and video annotation, span linking, multi-field spans, and layout customization.

Feb 14, 20266 min read

Guides9 min read

Visuelle KI zur Beschleunigung von Bild- und Video-Annotation einsetzen

KI-gestützte Objekterkennung, Pre-Annotation und Klassifizierung für Bild- und Videoaufgaben mit YOLO, Ollama, OpenAI und Claude einrichten.

Feb 14, 2026

Guides8 min read

Migrating from Label Studio to Potato

Migrate from Label Studio to Potato, convert project configs, annotation schemas, and exported data with this step-by-step guide covering common annotation types.

Jan 20, 2026

Guides4 min read

Running Annotation Studies on Prolific

Integrate Potato with Prolific for crowdsourced annotation, completion URL setup, participant ID tracking, attention checks, payment configuration, and quality filtering.

Jan 20, 2026

Guides6 min read

Legal Document Annotation Best Practices

Annotate legal documents in Potato, contracts, court filings, and regulatory text, with span labeling, entity extraction, and privacy-first self-hosted deployment.

Jan 18, 2026

Medical Image Annotation with Potato

Best practices for annotating medical images in Potato, DICOM display, radiology report labeling, adverse event extraction, and IRB-compliant self-hosted deployment.

Jan 18, 20265 min read

Polygon Annotation for Segmentation Tasks

Configure polygon drawing tools in Potato for image segmentation tasks, with tips for complex shapes, overlapping regions, multi-class polygons, and export to COCO format.

Jan 18, 20264 min read

Guides3 min read

Active Learning: Intelligenter annotieren, nicht mehr

Wie man Unsicherheits-Sampling nutzt, um Annotationen zu priorisieren und den gesamten Beschriftungsaufwand um bis zu 50 % zu reduzieren.

Tutorials4 min read

Drawing Bounding Boxes for Object Detection

Set up bounding box annotation for computer vision in Potato, configure label colors, minimum box size, multi-class support, validation rules, and COCO/YOLO export.

Guides5 min read

Exporting Annotations to Hugging Face Datasets

Convert Potato annotations to HuggingFace Datasets format, covering JSON export structure, dataset card generation, Hub upload, and integration with transformers training.

Tutorials7 min read

Image Comparison and Preference Tasks

Build side-by-side image comparison tasks in Potato for preference ranking, A/B testing, and visual quality assessment, with randomized order and pairwise scoring.

Introducing Potato 2.0: AI-Powered Annotation

Potato 2.0 ships AI-powered pre-annotation with OpenAI and Claude, multimedia support for audio and video, active learning, bounding box annotation, and a redesigned UI.

Jan 15, 20263 min read

Automatic Keyword Highlighting

Configure AI-powered keyword highlighting in Potato to draw annotator attention to important terms. Covers OpenAI, Claude, and custom keyword list configuration.

Jan 15, 20264 min read

Guides7 min read

Deploying to Amazon Mechanical Turk

Run Potato annotation tasks on Amazon Mechanical Turk, HIT configuration, qualification tests, approval and rejection workflows, bonus payments, and quality monitoring.

Creating a Sentiment Analysis Task

Build a complete sentiment classification task with radio buttons, tooltips, and keyboard shortcuts for efficient labeling.

Guides7 min read

Content Moderation Annotation Setup

Configure Potato for toxicity detection, hate speech classification, and sensitive content labeling with annotator wellbeing in mind.

Jan 12, 2026

Guides5 min read

Custom HTML Templates in Potato

Build custom annotation interfaces in Potato using HTML templates, CSS styling, and JavaScript, side-by-side layouts, custom widgets, and embedded media displays.

Jan 12, 2026

Pronunciation Assessment Annotation

Build a pronunciation quality annotation task in Potato with audio playback, waveform visualization, Likert rating scales, and per-recording free-text feedback fields.

Jan 12, 20266 min read

Speaker Diarization Annotation

Build a speaker identification task in Potato with interactive audio waveforms, timestamp markers, speaker label assignment, and inter-annotator agreement measurement.

Jan 12, 20264 min read

Tutorials3 min read

Video Annotation with Frame-by-Frame Controls

Set up video annotation in Potato with frame-by-frame navigation, timestamp markers, temporal event labeling, and per-segment classification using radio or span schemas.

Jan 12, 2026

Audio Event Detection and Tagging

Set up annotation for detecting specific sounds like speech, music, applause, or environmental noises with timestamp spans.

Jan 10, 2026

Tutorials2 min read

In 5 Minuten mit Potato loslegen

Eine schnelle Anleitung zur Einrichtung Ihres ersten Annotationsprojekts mit Potato – von der Installation bis zu Ihren ersten beschrifteten Daten.

Jan 10, 2026

Guides6 min read

Measuring Inter-Annotator Agreement

Calculate and interpret Cohen's Kappa, Fleiss' Kappa, and Krippendorff's Alpha for Potato annotation projects, with Python code examples and interpretation guidelines.

Jan 10, 2026

Potato Featured at EMNLP 2022

Our paper on Potato was accepted at EMNLP 2022. Learn about the research behind the tool and how to cite it in your work.

Jan 10, 20263 min read

Quality Control for Crowdsourced Annotation

Best practices for ensuring annotation quality in annotation projects, including practical strategies you can implement with and beyond Potato.

Jan 10, 20267 min read

Tutorials4 min read

Image Classification with Potato

Set up image classification in Potato with thumbnail previews, zoom controls, multi-label checkbox schemas, and keyboard shortcuts for high-throughput visual annotation.

Guides4 min read

LLMs für intelligente Annotations-Hinweise integrieren

Erfahren Sie, wie Sie OpenAI, Claude oder Gemini nutzen, um Ihren Annotatoren intelligente Hinweise und Vorschläge bereitzustellen.

Music Genre Classification Annotation

Create a music annotation task in Potato with waveform visualization, 30-second preview playback, hierarchical genre label trees, and pairwise preference comparisons.

Guides6 min read

Deploying Potato on a Server

Deploy Potato to production with Docker, nginx, SSL, and cloud platforms including AWS, GCP, and Azure. Includes systemd service configuration and scaling tips.

Multi-Object Tracking Annotation

An overview of multi-object tracking annotation concepts and how Potato's video annotation capabilities can support basic tracking workflows.

Jan 6, 20264 min read

Classifying Emotions in Speech

Create an audio emotion classification task in Potato with interactive waveform display, playback speed controls, Likert scales, and configurable emotion label sets.

Jan 5, 20266 min read

Tutorials4 min read

Building Your First NER Annotation Task

Step-by-step guide to building a named entity recognition task in Potato, span annotation config, label colors, overlapping spans, keyboard shortcuts, and CoNLL export.

Jan 5, 2026

Guides4 min read

Understanding Potato Data Formats

A deep dive into JSON and JSONL data formats in Potato, structuring instances for text, image, audio, video, and multimodal annotation with real config examples.

Jan 5, 2026