What's New

This page covers new features and improvements across Potato v2.x releases.

Potato 2.8.0

Released August 2026

Potato was a text annotation platform with an image mode. It now covers images, video, gigapixel scans, 3D point clouds, depth maps, robot episodes, generative-video rollouts and vision-language grounding. For each of them it reports chance-corrected agreement.

Upgrade if you installed from PyPI. Wheels up to 2.7.1 declared package data with a single-level glob, so 27 templates in subdirectories were missing from every wheel. Solo mode's setup routes, the admin pages, judge calibration and the corpus map were all broken for anyone who ran pip install; anyone running from a git checkout never saw it. Fixed in PR #164 by @0x6362.

pip install --upgrade potato-annotation

Agreement over geometry and time

• Chance-corrected agreement for spatial labels, decomposed into detection (did annotators find the same objects), classification (did they call them the same thing), and localization (did they put them in the same place, reported as σ against an empirical chance baseline plus a KS test).
• STAPLE for mask consensus. It estimates a latent boundary and a sensitivity and specificity for each rater. On a corpus where two careful annotators are outnumbered three-to-two by noisy ones, majority vote scores Dice 0.846 against truth and STAPLE scores 1.000.
• Exact rotated 3D IoU, so cuboid agreement is correct for drone, handheld and indoor data rather than only level boxes.
• Temporal agreement reported as a sweep across matching tolerances, because agreement at 0.25 s is a different claim from agreement at 2 s.
• Undefined values explain themselves. α is genuinely undefined on a unanimous corpus, and a bare NaN reads as a broken computation.

This also fixes a bug: adjudication compared the keys of an annotation, and image schemas store everything under one key named _data. Every image pair therefore scored 1.0 agreement, so two annotators who agreed on nothing looked unanimous and no image was ever routed for review.

Vision

• Geometry primitives: polylines, ellipses, 2D cuboids, keypoint sets with skeleton topology and COCO visibility flags, and instance-keyed brush masks.
• Click-to-segment in the browser — MobileSAM under ONNX Runtime Web, about 132 ms per click, no GPU and no per-click network call.
• Open-vocabulary text prompting: type a phrase, get every match boxed. Grounding DINO, Apache-2.0, also in the browser.
• Video mask propagation with SAM 2's memory modules, measured at 0.974–0.979 IoU per frame with no decay across the sequence. This one runs server-side by design; the browser has a separate, lighter carry-forward path.
• Deep zoom for gigapixel images, served as both DZI and IIIF Image API 3.0, with brush masks at the source's full resolution.
• Media ingest: multi-page 16-bit TIFF, HEIC, camera RAW, and ProRes/MKV/MOV through a cached server-side proxy.
• 15 import and 29 export formats, 11 of them round-tripping. Darwin works in both directions.

Image annotation keyboard shortcuts now follow V7 conventions by default: b brush, r rectangle, f fill, k keypoint, v select. Set keybinding_profile: legacy on the schema to restore the old bindings for a study already in the field.

3D, depth and robotics

• spatial_annotation for PCD, PLY, LAS, KITTI .bin and .xyz clouds, with 3D cuboids, points, polylines and per-point segments, octree level-of-detail, and orthographic slab panels. Rotation is stored as a quaternion, so the mounting tilt a yaw-only field would silently discard survives a KITTI round-trip.
• Depth maps in 16-bit PNG/TIFF, NPY, PFM and EXR, with windowing, colormaps, a metres readout under the cursor, and unprojection into the same 3D viewer.
• Calibration-aware projection, so a cuboid drawn in 3D appears in every camera image and can be checked in 2D.
• episode_annotation puts N synchronized video streams and M robot time-series lanes on one timeline, with phase segmentation, per-phase outcomes and dense reward curves. Imports LeRobot v2, RLDS/TFDS and HDF5. Lane downsampling preserves minima and maxima, so a one-frame force spike survives the trip to a 300-pixel lane.

Evaluation

• rollout_evaluation shows 2–N generated videos on one clock and asks the annotator to mark the frame at which the world stops making sense, then tag which physical or causal property broke. Panels can be blinded and stably shuffled per annotator.
• grounding_eval and region_caption score grounding by IoU at four thresholds and pointing by point-in-region hit rate, because a point has no area and IoU against one is always zero. Ungroundedness is counted separately.

Everything else

• Threaded conversations: the dialogue display renders reply structure from each turn's reply_to. potato convokit imports any ConvoKit corpus and --format convokit exports back, with no convokit dependency.
• Drawing telemetry — time per shape, stroke dynamics, revision counts and AI-suggestion accept latency. Pre-labels accepted wholesale make every quality measure look better, including agreement, because the geometry is identical to careful work. Timing is the only place the difference shows.
• Live database ingestion: rows created after startup become annotatable within poll_interval_seconds with no restart (#166).
• Machine-readable specs — a JSON Schema covering 159 config keys, 61 annotation types and 24 display types, plus an OpenAPI document with 419 paths. Both are generated from the code and CI fails when either drifts, so an editor or a coding agent can check a config without inventing options.
• No outbound requests: every stylesheet, script, font and icon now serves from the install across all 14 templates. styles.css had been opening with an @import of Google Fonts, which sent every annotator's IP address to a third party on page load, and the login page carried a third-party favicon. Both are gone.
• Admin Instances tab went from quadratic to linear: 2,000 items dropped from 15.1 s to 23 ms.

Potato 2.7.2

Released August 2026

Keystroke logging for free-text fields. Potato records the timing behind an answer and never the characters, which turns out to be enough to tell typing apart from pasting.

• Keystroke logging — content-blind capture of pauses, bursts, revisions and pastes on every free-text field, with about forty summary features computed per response. Off by default.
• Writing-process detection — six named rules with explicit thresholds that report the feature values behind every flag. Thresholds can be refitted to your own project, and there is a supervised path if you have labels.
• Ethics guidance — sample consent language, retention and deletion, the base-rate problem, and why a flag is a prompt to look rather than a finding.
• Opt-in exports — export_include_typing_dynamics writes a summary sidecar, and --format keystrokes writes the raw streams to Parquet. Neither is on by default, so behavioral data never lands in a dataset release by accident.

Disclosure to annotators defaults to on, and turning it off logs a warning at startup.

Potato 2.7.1

Released July 2026

Transcript ingestion, on the premise that the transcript already exists. Someone ran Whisper, or downloaded captions, or was handed a corpus, and the next step should not be writing a conversion script.

• 21 transcript and subtitle formats, up from 6. Whisper, WhisperX, whisper.cpp, AWS Transcribe, Deepgram, AssemblyAI, Rev.ai, SPoRC, SubRip, WebVTT, SubStation Alpha, TTML/DFXP, YouTube json3 and srv1/srv2/srv3, NIST CTM, Praat TextGrid, and ELAN EAF. Detection is by file contents, not extension.
• Sidecar transcript files — a data file can point at media/int_001.srt on disk instead of inlining the transcript. Paths resolve under task_dir through the standard path-security check, so your transcripts stay files you can diff and re-export.
• potato transcripts — a converter that turns a folder of ASR output into an annotation-ready data file, pairing transcripts to media by basename. --dry-run reports what it detected before writing anything, and --emit-config prints a matching config.
• One format vocabulary across four schemas — speech_transcript, voice_interaction, and tiered_annotation now accept everything the audio_dialogue display accepts. Previously only the display did.
• Annotating Whisper Transcripts and Annotating YouTube Subtitles walk through both starting points end to end.

ASR and diarization still run upstream. Potato reads their output; it does not transcribe.

Potato 2.7.0

Released July 2026

Potato 2.7 is built on one idea: an annotation tool should measure how judgments are made, not only record what they were. That applies to both kinds of judge — the humans who annotate, and the AI models being evaluated.

Measuring how humans label

Seven opt-in features, none of which requires an LLM.

• Psychometrics engine — a live item-response-theory layer that gives every label a posterior and a confidence interval (p = 0.94 [0.88–0.97]) instead of a bare majority vote. It estimates annotator ability and item difficulty from agreement patterns alone, routes items to the annotator whose judgment carries the most information, and flags likely codebook bugs.
• Multiplayer Rooms — the calibration meeting your team already runs over screen-share, built into the tool and instrumented: blind vote, host reveal, discussion, revote, with a live Krippendorff's α meter showing what the session was worth.
• Boundary Lab — counterfactual probes after each label ("would that still hold?"), which turn ordinary annotation into contrast sets and catch inconsistent annotators without planting gold items.
• Truth Serum — surprisingly-popular peer-prediction scoring, which beats majority vote where annotation is hard, with no gold labels.
• Think-Aloud Mode — annotators talk while they work; speech-to-text runs fully locally and the verbatim transcript becomes the rationale.
• Paper Mode — one command turns a project into a compilable LaTeX dataset report, agreement statistics and citations included.
• Pocket Mode — first-class mobile annotation with a swipeable card stack, offline sync, and PWA install.

Evaluating AI agents

The agent-evaluation suite grew to cover team structure and multimodal agents: a clickable agent interaction graph, cross-agent failure attribution, handoff review, per-agent and team scorecards, tool-contention timelines, and emergent-behavior tagging, plus GUI/computer-use trajectories, full-duplex voice timelines, video temporal grounding, and document-table structure.

Everywhere else

Cross-document event annotation, turn-level annotation, a living-document codebook with a full-page editor, PDF cross-page linking with optional OCR, RBAC roles with per-cohort schemas, admin and annotator dashboards in 10 languages, and a much lighter install — AI SDKs now load lazily, dropping boot time from roughly 2.0s to 0.7s.

Potato 2.6.0

Released June 2026

Potato 2.6 moves into qualitative data analysis and deepens the agent-evaluation toolkit. It adds QDA Mode, an LLM-as-judge calibration and alignment workflow with a signal-based triage queue, and trajectory-editing schemas that produce SFT and DPO training data. Potato is also relicensed to GPL-3.0-or-later (from PolyForm Shield).

QDA Mode

An opt-in qda_mode turns Potato into a collaborative qualitative-coding workspace. Enabling it composes a living codebook, in-vivo coding, analyst memos, cases, and full-text search, with defaults tuned for one analyst coding a whole corpus.

qda_mode:
  enabled: true            # codebook + memos + cases + search
codebook_invivo_key: i     # mint a code from a text selection
search:
  enabled: true
  annotator_claim: true

Learn more about QDA Mode →

LLM-as-Judge Calibration and Alignment

Auto-label data with one or more LLM judges, then run a blind human calibration pass to measure accuracy, agreement, and calibration error. A separate single-judge alignment workflow tracks Cohen's kappa against your human gold labels as you refine the rubric, with an optional inline verdict during annotation.

Learn more about Judge Calibration → · Judge Alignment →

Signal-Based Triage Queue

Prioritize the annotation queue by a per-item quality signal — an agent error, a production thumbs-down, a low score, or any custom field — so reviewers see the most-suspect items first instead of in arrival order.

triage:
  enabled: true
  signal_field: quality_score
  invert_signal: true
assignment_strategy: priority

Learn more about the Triage Queue →

Trajectory Editing for SFT/DPO

The new trajectory_edit and trajectory_correction schemas let annotators rewrite agent trace steps. The exporter turns each original/corrected pair into supervised fine-tuning targets (trajectory_sft.jsonl) and DPO preference pairs (trajectory_dpo.jsonl).

Learn more about Trajectory Editing →

eval_trace Display

A three-pane agent-trace display — Reasoning, Function Calls, and Final Answer — built for continuous evaluation, where traces arrive over a webhook, a Langfuse poller, or a watched directory and are judged as they land.

Learn more about eval_trace →

Workflow and Assignment

• Heterogeneous coverage — per-item annotator caps, inter-annotator agreement reporting, and adjudication routing for tasks where items need different numbers of annotators.
• Reclaim abandoned assignments — recover assignments left by Prolific or QC-blocked workers, with configurable retention and idempotent reclaim.
• Custom Batch assignment strategy — assign predefined batches of items to specific annotators.
• Reverse-proxy URL prefixes — serve Potato under a sub-path behind a reverse proxy.

Licensing

Potato is now released under GPL-3.0-or-later, relicensed from PolyForm Shield. You can use, modify, and redistribute it, including commercially, as long as derivative works stay under the GPL. See the About page for details.

Performance and Stability

• Roughly 3× faster boot. The ML stack is no longer eager-loaded at startup: import time dropped from about 6.5s to 2s, a 50k-item boot from about 10s to 5.7s, and resident memory from about 750MB to 365MB.
• Schema rename. annotation_type: highlight is now span, with a migration in place. Update old configs by renaming the type; existing span configs are unaffected.
• A broad QA-hardening wave touched route registration, training phases, Prolific, persistence, export and survey handling, webhooks, solo mode, and active learning.

Potato 2.5.0

Released 2026

A qualitative-coding wave that makes Potato credible for qualitative-research workflows alongside its NLP and ML annotation surface.

• Inter-annotator agreement — Cohen's kappa (pairwise) and Fleiss' kappa (N raters) in addition to Krippendorff's alpha, surfaced via the admin agreement API. See the inter-annotator agreement guide.
• New exporters — codebook (per-schema CSV with code hierarchy, color, description, and use count) and quotation_report (per-span CSV with text, offsets, source document, and coder).
• Code analytics — admin endpoints for pairwise code co-occurrence and a codes-by-attribute crosstab over existing instance metadata.

Potato 2.4.5

Released 2026

A stability and tooling release.

• Validated refinement — a pluggable framework for iteratively improving solo-mode annotation guidelines, with guardrails against guideline contradictions and metadata leakage.
• Config validator — a new python -m potato.validate_cli CLI that checks config keys against the known-config schema.
• Security — a fix for a path-traversal sibling-prefix bypass in path validation (GHSA-q9m2-fhv9-3jcf).
• Fixes across save/navigation state-sync, Prolific integration, multi-phase navigation, and quality-control feedback.

Potato 2.3.0

Released March 9, 2026

Potato 2.3 is the largest release in Potato's history, introducing agentic annotation, Solo Mode, Best-Worst Scaling, SSO/OAuth authentication, Parquet export, 15 new demo projects, and security hardening.

Agentic Annotation

A complete system for evaluating AI agents through human annotation. Includes 12 trace format converters, 3 specialized display types, and 9 pre-built annotation schemas.

12 Trace Format Converters — Import agent traces from OpenAI, Anthropic, SWE-bench, OpenTelemetry, MCP, CrewAI/AutoGen/LangGraph, LangChain, LangFuse, ReAct, WebArena/VisualWebArena, ATIF, and raw browser recordings. Auto-detection available.

agentic:
  enabled: true
  trace_converter: react       # or openai, anthropic, webarena, auto, etc.
  trace_file: "data/traces.jsonl"

3 Display Types:

• Agent Trace Display — Color-coded step cards with collapsible observations, JSON pretty-printing, and timeline sidebar for tool-using agents
• Web Agent Trace Display — Full screenshots with SVG overlays showing click targets, text inputs, and scroll actions; filmstrip navigation for browsing agents
• Interactive Chat Display — Live chat mode (annotator interacts with agent via proxy) and trace review mode for conversational agents

Per-Turn Ratings — Rate individual steps alongside the overall trace for fine-grained evaluation.

9 Pre-Built Schemas — agent_task_success, agent_step_correctness, agent_error_taxonomy, agent_safety, agent_efficiency, agent_instruction_following, agent_explanation_quality, agent_web_action_correctness, agent_conversation_quality.

Agent Proxy System — OpenAI, HTTP, and echo proxies for live agent evaluation.

Learn more about Agentic Annotation →

Solo Mode

A 12-phase intelligent workflow where a single human annotator collaborates with an LLM to label entire datasets, achieving 95%+ agreement with multi-annotator pipelines while requiring only 10-15% of total human labels.

The 12 Phases:

1. Seed Annotation — human labels 50 diverse instances
2. Initial LLM Calibration — LLM labels using seed examples
3. Confusion Analysis — identify systematic disagreement patterns
4. Guideline Refinement — LLM proposes, human approves updated guidelines
5. Labeling Function Generation — ALCHEmist-inspired programmatic rules
6. Active Labeling — human labels most informative instances
7. Automated Refinement Loop — iterative re-labeling with improved guidelines
8. Disagreement Exploration — human resolves LLM/LF conflicts
9. Edge Case Synthesis — LLM generates ambiguous examples for human labeling
10. Cascaded Confidence Escalation — human reviews lowest-confidence labels
11. Prompt Optimization — DSPy-inspired automated prompt search
12. Final Validation — random sample review

solo_mode:
  enabled: true
  llm:
    endpoint_type: openai
    model: "gpt-4o"
    api_key: ${OPENAI_API_KEY}
  seed_count: 50
  accuracy_threshold: 0.92

Multi-Signal Instance Prioritization — 6 weighted pools (uncertain, disagreement, boundary, novel, error_pattern, random) for selecting the most valuable instances.

Learn more about Solo Mode →

Best-Worst Scaling

Efficient comparative annotation where annotators select the best and worst items from tuples. Automatic tuple generation with balanced incomplete block designs and three scoring methods (Counting, Bradley-Terry, Plackett-Luce).

annotation_schemes:
  - annotation_type: bws
    name: fluency
    tuple_size: 4
    best_description: "Most Fluent"
    worst_description: "Least Fluent"

Learn more about Best-Worst Scaling →

SSO & OAuth Authentication

Production-ready authentication with Google OAuth (domain restriction), GitHub OAuth (organization restriction), and generic OIDC (Okta, Azure AD, Auth0, Keycloak). Supports auto-registration, mixed mode, and session management.

authentication:
  method: google_oauth
  google_oauth:
    client_id: ${GOOGLE_CLIENT_ID}
    client_secret: ${GOOGLE_CLIENT_SECRET}
    allowed_domains:
      - "umich.edu"
    auto_register: true

Learn more about SSO & OAuth →

Parquet Export

Export annotations to Apache Parquet format, producing three structured files: annotations.parquet, spans.parquet, and items.parquet. Supports snappy, gzip, zstd, lz4, and brotli compression, incremental export, and date/annotator partitioning. Compatible with pandas, DuckDB, PyArrow, Polars, and Hugging Face Datasets.

parquet_export:
  enabled: true
  output_dir: "output/parquet/"
  compression: zstd
  auto_export: true

Learn more about Parquet Export →

15 New Demo Projects

New demos in project-hub/ covering agentic annotation (5 demos), Solo Mode (3 demos), Best-Worst Scaling (3 demos), authentication (2 demos), and export workflows (2 demos). Start any demo with potato start config.yaml.

Security Hardening

• Cryptographically secure session tokens with configurable expiration
• CSRF protection enabled by default
• Rate limiting on authentication endpoints
• Input sanitization for user-provided content
• Dependency audit with all packages updated
• Content Security Policy headers

Other Improvements

• Custom trace converters for unsupported agent frameworks
• Hybrid Solo Mode with multi-annotator verification sampling
• BWS admin dashboard tab with score convergence charts
• Incremental Parquet export with date partitioning

v2.2 vs v2.3 Comparison

Potato 2.2.0

Released February 20, 2026

Potato 2.2 is a major feature release with 9 new annotation schemas, a pluggable export system, MACE competence estimation, 55 validated survey instruments, and remote data sources.

New Annotation Schemas (9)

Event Annotation — N-ary event structures with trigger spans and typed argument roles. Annotate events like ATTACK, HIRE, and TRAVEL with constrained entity arguments and hub-spoke arc visualization.

annotation_schemes:
  - annotation_type: event_annotation
    name: events
    span_schema: entities
    event_types:
      - type: "ATTACK"
        trigger_labels: ["EVENT_TRIGGER"]
        arguments:
          - role: "attacker"
            entity_types: ["PERSON", "ORGANIZATION"]
            required: true

Learn more about Event Annotation →

Entity Linking — Link span annotations to external knowledge bases (Wikidata, UMLS, custom REST APIs). Add an entity_linking: block to any span schema to enable KB search and linking.

Learn more about Entity Linking →

Triage — Prodigy-style accept/reject/skip interface for rapid data screening. Customizable labels, keyboard shortcuts, and auto-advance for high-throughput annotation.

Learn more about Triage →

Pairwise Comparison — Compare two items with binary (click preferred tile) or scale (slider) modes. Supports items_key, allow_tie, scale: block with configurable range.

Learn more about Pairwise Comparison →

Conversation Trees — Annotate hierarchical conversation structures with per-node ratings, path selection, and branch comparison.

Learn more about Conversation Trees →

Coreference Chains — Group coreferring text mentions into chains with visual indicators. Supports entity types, singleton control, and multiple highlight modes.

Learn more about Coreference Chains →

Segmentation Masks — New fill, eraser, and brush tools for pixel-level image segmentation.

Bounding Box for PDF/Documents — Draw boxes on PDF pages for document annotation tasks.

Discontinuous Spans — allow_discontinuous: true enables selecting non-contiguous text segments as a single span.

Intelligent Annotation

MACE Competence Estimation — Variational Bayes EM algorithm that jointly estimates true labels and annotator competence scores (0.0-1.0). Works with radio, likert, select, and multiselect schemas.

mace:
  enabled: true
  trigger_every_n: 10
  min_annotations_per_item: 3

Learn more about MACE →

Option Highlighting — LLM-based highlighting of likely correct options for discrete annotation tasks. Highlights top-k options with a star indicator while dimming less-likely options.

ai_support:
  option_highlighting:
    enabled: true
    top_k: 3
    dim_opacity: 0.4

Learn more about Option Highlighting →

Diversity Ordering — Embedding-based clustering and round-robin sampling to ensure annotators see diverse content rather than similar items in sequence.

assignment_strategy: diversity_clustering
diversity_ordering:
  enabled: true
  prefill_count: 100

Learn more about Diversity Ordering →

Export System

A new pluggable export CLI (python -m potato.export) converts annotations to 6 industry-standard formats: COCO, YOLO, Pascal VOC, CoNLL-2003, CoNLL-U, and Segmentation Masks.

python -m potato.export --config config.yaml --format coco --output ./export/

Learn more about Export Formats →

Remote Data Sources

Load annotation data from URLs, S3, Google Drive, Dropbox, Hugging Face, Google Sheets, and SQL databases via the new data_sources: config block. Includes partial loading, caching, and credential management.

Learn more about Remote Data Sources →

Survey Instruments

55 validated questionnaires across 8 categories (Personality, Mental Health, Affect, Self-Concept, Social Attitudes, Response Style, Short-Form, Demographics). Use in prestudy/poststudy phases with instrument: "tipi".

Learn more about Survey Instruments →

Other Improvements

• Video object tracking with keyframe interpolation
• External AI config file support
• Form layout grid improvements
• Format handlers for PDF, Word, code, and spreadsheets

Potato 2.1.0

Released February 5, 2026

Potato 2.1 introduces the instance display system, visual AI support, span linking, multi-field span annotation, and layout customization.

Instance Display System

A new instance_display config block that separates content display from annotation. Display any combination of images, videos, audio, text, and dialogues alongside any annotation schemes.

instance_display:
  fields:
    - key: image_url
      type: image
      display_options:
        max_width: 600
        zoomable: true
    - key: description
      type: text
 
annotation_schemes:
  - annotation_type: radio
    name: category
    labels: [nature, urban, people]

Supports 24 display types including text, html, image, video, audio, depth_map, dialogue, conversation_tree, pairwise, code, spreadsheet, document, pdf, gallery, and five agent-trace displays.

Learn more about Instance Display →

Multi-Field Span Annotation

Span annotation schemes now support a target_field option to annotate across multiple text fields in the same instance.

annotation_schemes:
  - annotation_type: span
    name: source_entities
    labels: [PERSON, ORGANIZATION]
 
  - annotation_type: span
    name: summary_entities
    labels: [PERSON, ORGANIZATION]

Learn more about Span Annotation →

Span Linking

A new span_link annotation type for creating typed relationships between annotated spans. Supports directed and undirected links, n-ary relationships, visual arc display, and label constraints.

annotation_schemes:
  - annotation_type: span
    name: entities
    labels:
      - name: "PERSON"
        color: "#3b82f6"
      - name: "ORGANIZATION"
        color: "#22c55e"
 
  - annotation_type: span_link
    name: relations
    span_schema: entities
    link_types:
      - name: "WORKS_FOR"
        directed: true
        allowed_source_labels: ["PERSON"]
        allowed_target_labels: ["ORGANIZATION"]
        color: "#dc2626"

Learn more about Span Linking →

Visual AI Support

Four new vision endpoints for AI-powered image and video annotation assistance:

• YOLO — Fast local object detection
• Ollama Vision — Local vision-language models (LLaVA, Qwen-VL)
• OpenAI Vision — GPT-4o cloud vision
• Anthropic Vision — Claude with vision

Features include object detection, pre-annotation, classification, hints, scene detection, keyframe detection, and object tracking.

Learn more about Visual AI Support →

Layout Customization

Create sophisticated custom visual layouts using HTML templates and CSS. Potato generates an editable layout file, or you can provide a fully custom template with grid layouts, color-coded options, and section styling.

task_layout: layouts/custom_task_layout.html

Three example layouts included: content moderation, dialogue QA, and medical review.

Learn more about Layout Customization →

Label Rationales

A fourth AI capability that generates balanced explanations for why each label might apply, helping annotators understand different classification perspectives.

ai_support:
  features:
    rationales:
      enabled: true

Learn more about AI Support →

Other Improvements

• 50+ new tests for improved reliability
• Responsive design improvements
• Enhanced project-hub organization with layout examples
• Bug fixes across annotation types

v2.0 vs v2.1 Comparison

Potato 2.0

Potato 2.0 is a major release that adds AI assistance, active learning, training phases, multi-phase workflows, and a MySQL backend.

AI Support

Integrate Large Language Models to assist annotators with intelligent hints, keyword highlighting, and label suggestions.

Supported providers:

• OpenAI (GPT-4, GPT-3.5)
• Anthropic (Claude 3, Claude 3.5)
• Google (Gemini)
• Ollama (local models)
• vLLM (self-hosted)

ai_support:
  enabled: true
  endpoint_type: openai
  ai_config:
    model: gpt-4
    api_key: ${OPENAI_API_KEY}
  features:
    hints:
      enabled: true
    label_suggestions:
      enabled: true

Learn more about AI Support →

Audio Annotation

Full-featured audio annotation with waveform visualization powered by Peaks.js. Create segments, label time regions, and annotate speech with keyboard shortcuts.

Key features:

• Waveform visualization
• Segment creation and labeling
• Per-segment annotation questions
• 15+ keyboard shortcuts
• Server-side waveform caching

annotation_schemes:
  - annotation_type: audio_annotation
    name: speakers
    mode: label
    labels:
      - Speaker A
      - Speaker B

Learn more about Audio Annotation →

Active Learning

Automatically prioritize annotation instances based on model uncertainty. Train classifiers on existing annotations and focus annotators on the most informative examples.

Capabilities:

• Multiple classifier options (LogisticRegression, RandomForest, SVC, MultinomialNB)
• Various vectorizers (TF-IDF, Count, Hashing)
• Model persistence across restarts
• LLM-enhanced selection
• Multi-schema support

active_learning:
  enabled: true
  schema_names:
    - sentiment
  min_instances_for_training: 30
  update_frequency: 50
  classifier:
    type: LogisticRegression

Learn more about Active Learning →

Training Phase

Qualify annotators with practice questions before the main task. Provide immediate feedback and ensure quality through configurable passing criteria.

Features:

• Practice questions with known answers
• Immediate feedback and explanations
• Configurable passing criteria
• Retry options
• Progress tracking in admin dashboard

phases:
  training:
    enabled: true
    data_file: "data/training.json"
    passing_criteria:
      min_correct: 8
      total_questions: 10

Learn more about Training Phase →

Enhanced Admin Dashboard

Comprehensive monitoring and management interface for annotation tasks.

Dashboard tabs:

• Overview: High-level metrics and completion rates
• Annotators: Performance tracking, timing analysis
• Instances: Browse data with disagreement scores
• Configuration: Real-time settings adjustment

admin_api_key: ${ADMIN_API_KEY}

Learn more about Admin Dashboard →

Database Backend

MySQL support for large-scale deployments with connection pooling and transaction support.

database:
  type: mysql
  host: localhost
  database: potato_db
  user: ${DB_USER}
  password: ${DB_PASSWORD}

Potato automatically creates required tables on first startup.

Annotation History

Complete tracking of all annotation changes with timestamps, user IDs, and action types. Enables auditing and behavioral analysis.

{
  "history": [
    {
      "timestamp": "2024-01-15T10:30:00Z",
      "user": "annotator_1",
      "action": "create",
      "schema": "sentiment",
      "value": "Positive"
    }
  ]
}

Multi-Phase Workflows

Build complex annotation workflows with multiple sequential phases:

1. Consent - Informed consent collection
2. Pre-study - Demographics and screening
3. Instructions - Task guidelines
4. Training - Practice questions
5. Annotation - Main task
6. Post-study - Feedback surveys

phases:
  consent:
    enabled: true
    data_file: "data/consent.json"
  prestudy:
    enabled: true
    data_file: "data/demographics.json"
  training:
    enabled: true
    data_file: "data/training.json"
  poststudy:
    enabled: true
    data_file: "data/feedback.json"

Learn more about Multi-Phase Workflows →

v2.0 Configuration Changes

New Configuration Structure

Potato 2.0 uses a cleaner configuration format:

v1 (old):

data_files:
  - data.json
id_key: id
text_key: text
output_file: annotations.json

v2 (new):

data_files:
  - "data/data.json"
 
item_properties:
  id_key: id
  text_key: text
 
output_annotation_dir: "output/"
output_annotation_format: "json"

Security Requirement

Configuration files must now be located within the task_dir:

# Valid - config.yaml is in the project directory
task_dir: "."
 
# Valid - config in configs/ subdirectory
task_dir: "my_project/"

Quick Comparison

Migration Guide

Updating Your Configuration (v1 to v2)

1. Data configuration

   yamlCopy

   # Old
   id_key: id
   text_key: text
    
   # New
   item_properties:
     id_key: id
     text_key: text

2. Output configuration

   yamlCopy

   # Old
   output_file: annotations.json
    
   # New
   output_annotation_dir: "output/"
   output_annotation_format: "json"

3. Config file location Ensure your config file is inside the project directory.

Starting the Server

# v2 command
python -m potato start config.yaml -p 8000
 
# Or shorthand
potato start config.yaml

Getting Started

Ready to try Potato? Start with the Quick Start Guide or explore specific features:

v2.3 Features:

• Agentic Annotation - Evaluate AI agents with trace converters and step-level displays
• Solo Mode - Human-LLM collaborative labeling
• Best-Worst Scaling - Comparative annotation with scoring
• SSO & OAuth - Google, GitHub, and OIDC authentication
• Parquet Export - Columnar data export

v2.2 Features:

• Event Annotation - N-ary event structures
• Entity Linking - Knowledge base linking
• Triage - Rapid data screening
• Coreference Chains - Entity coreference
• Conversation Trees - Hierarchical dialogue annotation
• MACE - Annotator competence estimation
• Option Highlighting - AI-assisted option guidance
• Diversity Ordering - Embedding-based item ordering
• Export Formats - Export CLI with 6 formats
• Remote Data Sources - Cloud data loading
• Survey Instruments - 55 validated questionnaires

v2.1 Features:

• Instance Display - Multi-modal content display
• Visual AI Support - AI for image and video annotation
• Span Linking - Entity relationship annotation

Core Features:

• AI Support - Intelligent annotation assistance
• Active Learning - Smart instance prioritization
• Audio Annotation - Waveform-based annotation
• Training Phase - Annotator qualification
• Admin Dashboard - Monitoring and management