# Potato — Full Documentation Potato is a free, open-source data annotation tool from the University of Michigan. This file concatenates the full documentation as Markdown for AI ingestion. Source: https://www.potatoannotator.com --- # Quick Start Section: Getting Started Source: https://www.potatoannotator.com/docs/getting-started/quick-start > Get Potato running in under 5 minutes. Install with pip, write a YAML config, launch the server, and annotate your first dataset without writing code. This guide takes you from a fresh install to annotating data in under 5 minutes. It shows how to write a task yourself. The other way to start is to describe the study to a coding agent and let potato-skill design and build it, which [Quick Start with a Coding Agent](/docs/getting-started/quick-start-with-a-coding-agent) covers. ![Potato in five steps: install, configure, run the server, annotate in the browser, and export](/images/blog/quickstart-flow.svg "Potato in five steps") ## Prerequisites - Python 3.7 or higher - `pip` package manager - A terminal ## Installation We recommend installing Potato into a virtual environment so it doesn't collide with your system Python: ```bash python -m venv potato-env source potato-env/bin/activate # Windows: potato-env\Scripts\activate pip install potato-annotation ``` Verify the install: ```bash python -c "import potato; print(potato.__version__)" ``` You should see `2.9.1` printed. From version 2.9, `potato --version` reports the same number, and adds the commit when you are running from a checkout. ## Create Your First Project ### 1. Create a project directory ```bash mkdir my-annotation-task cd my-annotation-task ``` ### 2. Create your data file Create a `data` folder and a file called `data.json` inside it: ```bash mkdir data ``` ```json [ {"id": "1", "text": "I love this product! It's amazing."}, {"id": "2", "text": "This is the worst experience ever."}, {"id": "3", "text": "It's okay, nothing special."} ] ``` Potato also accepts `.jsonl` (one JSON object per line), `.csv`, and `.tsv` for `data_files`. Point at a file with the matching extension and Potato picks the reader from it. ### 3. Create your configuration Create a file called `config.yaml` in your project directory: > **Path sandbox**: Any relative paths inside your config (like `data_files` or `output_annotation_dir`) are resolved against `task_dir` and must stay inside it. Paths that resolve outside `task_dir` are rejected to prevent path traversal. ```yaml annotation_task_name: "Sentiment Analysis" # All relative paths below are resolved against task_dir. task_dir: "." # Data configuration data_files: - "data/data.json" # Tell Potato which fields in each record to use. # id_key = unique identifier; text_key = the text shown to annotators. item_properties: id_key: id text_key: text # Where per-user annotation state is stored (relative to task_dir). output_annotation_dir: "annotation_output/" # Also write a periodic flat export, because the per-user state files are # awkward to analyze. No exporter is named "json", so use "jsonl". export_annotation_format: "jsonl" # Annotation scheme annotation_schemes: - annotation_type: radio name: sentiment description: "What is the sentiment of this text?" # required for radio schemes labels: - Positive - Negative - Neutral # Skip password authentication so anyone can log in with just a username. require_password: false ``` > **Tip**: You can validate your config without launching the full server using the [Preview CLI](/docs/tools/preview-cli). It's the fastest way to catch typos before you start annotating. ### 4. Start the server ```bash potato start config.yaml -p 8000 ``` Equivalent long form: ```bash python -m potato start config.yaml -p 8000 ``` If port 8000 is already in use, pick another with `-p 8001`. To stop the server, press `Ctrl+C` in the terminal. > **Faster first run**: Add `--debug` to bypass the login screen entirely while you're experimenting: `potato start config.yaml --debug`. ### 5. Open your browser Navigate to [http://localhost:8000](http://localhost:8000). You'll see a login screen with a single **username** field. Type anything (e.g. `alice`) and click **Continue**, and you're annotating. Each time you select a label and click **Next**, your annotation is saved automatically. There's no separate "save" button. ## Where Your Annotations Go Potato writes per-user state as you annotate: ``` my-annotation-task/ ├── config.yaml ├── data/ │ └── data.json └── annotation_output/ # created automatically └── alice/ └── user_state.json # one folder per username ``` `user_state.json` holds the full annotation state for that user. To produce a consolidated, flat file across all annotators, run Potato's export CLI, which the [export documentation](https://github.com/davidjurgens/potato/blob/master/docs/data-export/export_formats.md) describes. Run `python -m potato.export --list-formats` to see every format it registers. ## Common First-Run Issues - **`Address already in use`** — port 8000 is taken. Use `-p 8001` (or any free port). - **`KeyError: 'description'`** — a radio (or similar) scheme is missing its required `description` field. Every scheme needs one. - **`Path ... is outside the task directory`** — a `data_files` or `output_annotation_dir` path escapes `task_dir`. Keep everything inside the project folder, or set `task_dir` to an absolute path that contains them. - **Login page shows a password field** — you forgot `require_password: false`, or you're on a cached browser tab. Restart the server and hard-refresh. ## Frequently Asked Questions The questions below cover installing Potato, running it in Docker, and checking a config before annotators see it. ### How do I install Potato? Install it from PyPI with `pip install potato-annotation`, on Python 3.7 or later. Installing into a virtual environment keeps Potato's dependencies apart from your system Python, and `potato --version` confirms the install by printing the version number. ### Can I run Potato in Docker? Yes. Potato publishes a prebuilt image, `ghcr.io/davidjurgens/potato`, to GitHub Container Registry. Your project folder mounts into the container at `/app` and the server listens on port 7860 inside it, so one image serves every task. Set `POTATO_SECRET_KEY` so that annotator sessions survive a restart. [Deploying Potato on a Server](/blog/server-deployment-guide) gives the full `docker run` command. ### How do I check a Potato config before annotators see it? Run `potato validate config.yaml --strict`, which checks the config the way the server does at boot and turns warnings, such as an unrecognized key, into failures. `potato preview config.yaml` then shows what the config declares and renders the interface in a browser, so you see the task as an annotator will before anyone logs in. ## What's Next? - Learn about [Configuration Basics](/docs/getting-started/configuration-basics) for more options - Have a coding agent design and build your next task with [potato-skill](/docs/getting-started/quick-start-with-a-coding-agent) - Explore different [Annotation Types](/docs/annotation-types/radio-multiselect) - Set up [User Management](/docs/core-concepts/user-management) for your team - Discover [What's New in v2](/docs/getting-started/whats-new-v2) including AI support and active learning ## Tools & Utilities Once you're comfortable with the basics, check out these helpful tools: - [Preview CLI](/docs/tools/preview-cli) - Validate configurations without running the server - [Debugging Guide](/docs/tools/debugging-guide) - Debug flags and troubleshooting tips - [User Simulator](/docs/tools/simulator) - Test with simulated annotators --- # potato-skill: Quick Start with a Coding Agent Section: Getting Started Source: https://www.potatoannotator.com/docs/getting-started/quick-start-with-a-coding-agent > Describe what you need labeled to Claude Code, Codex, or Cursor, and potato-skill designs the annotation study, builds it in Potato, and checks that it renders. **You can start a Potato project in two ways. Write the task yourself, as the [Quick Start](/docs/getting-started/quick-start) shows, or describe the study to a coding agent and let [potato-skill](https://github.com/davidjurgens/potato-skill) design and build it.** This page covers the second route. Both end in the same place, a Potato annotation task you can run, change, and share. ## What the skill decides potato-skill is an [agent skill](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview), a package of instructions and helper scripts that a coding agent loads when a request needs them. It ships 28 reference files and nine helper scripts, and it covers the design of the study as well as the build: - what to ask annotators, and on what scale - how many annotators each item needs - how to train annotators and check their work - the interface annotators see - exporting the results in the format the next step needs Some of those it settles on its own, and some it hands back. The rule it works to is that a decision is made locally when being wrong is cheap to fix, and handed back when being wrong is expensive, irreversible, or not the agent's to make. It takes a 5-point scale with every point labeled, keyboard shortcuts on any question with nine labels or fewer, random item order with a fixed `random_seed`, and all questions required. It asks you about the unit of annotation, the number of annotators per item when agreement is going to be reported, the consent wording, and any label set with no "unclear" option. The task it builds runs on Potato, so it gets what Potato already does: a login for each annotator, several annotators per item, [agreement statistics](/docs/guides/inter-annotator-agreement), [Prolific and MTurk](/docs/guides/crowdsourcing-prolific-mturk) integration, and export to the formats downstream tools read. ## Requirements Potato has to be installed where the agent runs commands, because the skill's helper scripts import Potato's registries and call the `potato` command: ```bash pip install potato-annotation ``` You also need a coding agent. The skill was written for Claude Code, and it installs into Codex and Cursor as well. ## Installing the skill In [Claude Code](https://code.claude.com/docs/en/discover-plugins), install it from the plugin marketplace: ```text /plugin marketplace add davidjurgens/potato-skill /plugin install potato-skill@potato ``` For Codex and Cursor, the [`skills` CLI](https://github.com/vercel-labs/skills) installs it into the project you are working in: ```bash npx skills add davidjurgens/potato-skill --agent codex cursor ``` ## What a Codex or Cursor install includes The `skills` CLI writes `.agents/skills/potato-skill/` with `SKILL.md`, all 28 references, and all nine scripts, so a Codex or Cursor project gets the same material a Claude Code install gets. It leaves `AGENTS.md` and `.cursor/rules` untouched, and how each tool discovers a skill in that directory is up to the tool. The repository also carries an `AGENTS.md` at its root, which Codex and Cursor read by filename with no install step. That file is a short summary against roughly 79,000 words across the references, and it points at reference files and scripts that only a full install puts on disk. Use it when you want an agent to have the check-before-handoff loop and the rules that bite, and use the `skills` CLI when you want the reference material behind them. ## Describing the study Say what needs labeling, who will label it, and what the labels are for. You do not need to know Potato's annotation types or its config format. A request can be this short: > I want 200 tweets labeled for sentiment on a 5-point scale by three annotators each, with a consent page and two attention checks. Build it and start it. The skill works out the question wording, the scale, how items are assigned, and how annotators are checked. It does not block on the questions it wants confirmed. It states its assumptions, builds the task, verifies it, and asks alongside something you can already click through. ## Checks the agent runs before handing it back A config that validates tells you the server will start, and tells you nothing about whether a person can do the work. The skill instructs the agent to render the task and look at it: ```bash potato validate config.yaml --strict potato preview config.yaml --screenshot shot-01.png ``` `--strict` turns an unrecognized config key from a warning into a failure, which catches a typo that would otherwise leave a feature silently off. `--screenshot` boots a real Potato server and drives it in headless Playwright, recording console errors, uncaught page exceptions, and HTTP responses of 400 or above. It exits 0 only when all three lists are empty. Three of the nine helpers automate what can be measured rather than judged. `check_ui.py` measures the live layout and reports schemes or a Next button below the fold, empty media widgets, and duplicate shortcuts. `boot_and_check.py` reports every feature that is configured but loaded nothing. `walk_task.py` walks a running task the way an annotator would and reports where it stops. Open the task in your browser and label a few items yourself before you invite anyone. ## The task it produces A Potato task is a project folder holding the task's config, its data, and its instructions. Everything in the [Quick Start](/docs/getting-started/quick-start) applies to it. You start it with `potato start config.yaml`, edit it by hand, add annotators, and export the labels. Because the result is a Potato task folder, other people can use it too. A colleague can run the same study on new data, you can rerun it next year, and you can publish it with the paper that used it. The [showcase](/showcase) is a collection of tasks shared this way, taken from published papers. ## Existing designs and annotations - **A published design.** The skill can start from one of 440 annotation designs taken from published papers, with their label sets, question wording, and instructions. You can browse them in the [showcase](/showcase), and `find_design.py` searches them for a design close to the one you described. - **Another tool's annotations.** `potato import` reads 20 formats, including Labelbox, CVAT, and COCO, so a half-finished project can carry on in Potato. - **A study that is already running.** `study_status.py` reports progress for each annotator and agreement for each question, and the skill says which fixes are safe mid-study and which would corrupt the data already collected. ## Potato's MCP servers An agent can reach the same answers over MCP instead of the shell. `potato mcp serve --root .` exposes 12 authoring tools that read the same registries as the CLI, including `render_task_screenshot`, which returns the rendered page as an image. `potato mcp connect` bridges a running study and adds 12 live tools covering status, progress, annotators, agreement, assignment, and export. The bridge needs an `mcp` block in the config listing the tools to expose, and a token from `potato mcp issue-token`. ## Tests against Potato's registries A plausible wrong identifier is worse than no documentation, because an agent will use it. Three of the 28 references are generated from Potato's own registries and cannot drift from what the server enforces: all 61 annotation types, each with a worked example lifted from a real project, the documented top-level config keys, and the documented sub-keys. CI covers the skill's core files, checking that every annotation type, display type, config key, operator, strategy name, and command named in them exists in Potato. Every YAML sample is spliced into a working config and run through Potato's real validator, and the worked example boots a real server. Its reference material is published at [davidjurgens.github.io/potato-skill](https://davidjurgens.github.io/potato-skill/), readable without installing anything. Anthropic [recommends reviewing any skill](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview#security-considerations) before you install it. ## Comparing the two routes The two routes differ in what you start from, not in what you end up with: | | Write the task yourself | Describe it to a coding agent | |---|---|---| | You start with | a YAML config | a description in plain language | | Suits | learning Potato, small tasks | new studies, crowdsourced studies with training and checks, annotation types you have not used | | You end with | a Potato task | a Potato task | You can switch routes at any point. An agent can change a task you wrote, and you can edit one the agent built. ## Questions The questions below cover setting up potato-skill with a coding agent and finding its source. ### How do I use Potato with a coding agent? Install Potato and potato-skill where the agent runs commands, then describe the study to the agent in plain language. `pip install potato-annotation` installs Potato. In Claude Code, `/plugin marketplace add davidjurgens/potato-skill` followed by `/plugin install potato-skill@potato` installs the skill, and in Codex or Cursor, `npx skills add davidjurgens/potato-skill --agent codex cursor` does. Tell the agent what needs labeling, who will label it, and what the labels are for. potato-skill builds the task, renders it to check that it works, and hands back a list of the decisions it made for you. ### Where is potato-skill on GitHub? potato-skill is at [github.com/davidjurgens/potato-skill](https://github.com/davidjurgens/potato-skill), released under the GPL-3.0-or-later. Its reference material is published at [davidjurgens.github.io/potato-skill](https://davidjurgens.github.io/potato-skill/), so you can read what the skill tells the agent before you install it. ## Next steps - [Quick Start](/docs/getting-started/quick-start), for writing a task by hand - [Configuration Basics](/docs/getting-started/configuration-basics) - [Crowdsourcing on Prolific and MTurk](/docs/guides/crowdsourcing-prolific-mturk) - [Measuring inter-annotator agreement](/docs/guides/inter-annotator-agreement) - [Configs your coding agent can check](/docs/guides/configs-your-coding-agent-can-check) - [potato-skill on GitHub](https://github.com/davidjurgens/potato-skill) --- # Installation Section: Getting Started Source: https://www.potatoannotator.com/docs/getting-started/installation > Install Potato with pip on macOS, Windows, or Linux. Covers Python versions, virtual environments, optional extras, and upgrading from 2.7.1 or earlier. Potato installs with pip. This guide covers the Python versions it supports, the optional extras, and the upgrade path from 2.7.1 or earlier. ## Upgrading from 2.7.1 or earlier Wheels up to 2.7.1 declared their package data with a single-level glob, so 27 templates stored in subdirectories never made it into the published package. If you installed with `pip`, solo mode's setup routes, the admin dashboard, judge calibration and the corpus map were all broken. Running from a git checkout was unaffected, which is why the test suite never caught it. Version 2.8.0 fixes the packaging. It was reported and fixed in [PR #164](https://github.com/davidjurgens/potato/pull/164) by [@0x6362](https://github.com/0x6362). ```bash pip install --upgrade potato-annotation ``` ## Requirements - **Python**: 3.7 or higher - **Operating System**: macOS, Linux, or Windows - **Memory**: 512MB RAM minimum ## Install via pip The recommended way to install Potato: ```bash pip install potato-annotation ``` ## Optional extras The default install pulls in no machine-learning stack. Add the extras you need: ```bash pip install 'potato-annotation[ai]' # LLM endpoints pip install 'potato-annotation[vision]' # local vision models pip install 'potato-annotation[db]' # database data sources pip install 'potato-annotation[transcribe]' # local transcription and diarization (2.9+) ``` `vision` is deliberately excluded from `[all]`, because torch is a multi-gigabyte install and nothing in the default experience needs it. Model weights are a separate, explicit step: ```bash potato download-models --list potato download-models mobile_sam ``` Nothing is bundled and nothing downloads unless you ask for it. For an air-gapped install, run the downloads on a connected machine and copy the directory across. ## Install from source For the latest development version: ```bash git clone https://github.com/davidjurgens/potato.git cd potato pip install -e . ``` ## Install specific version To install a specific version: ```bash pip install potato-annotation==2.9.1 ``` ## Verify installation Check that Potato is installed correctly: ```bash potato --version ``` You should see the version, plus the commit when you are running from a checkout: ``` potato 2.9.1 ``` ## Upgrade To upgrade to the latest version: ```bash pip install --upgrade potato-annotation ``` ## Optional dependencies Potato 2.9.1 declares these extras: `ai`, `formats`, `viz`, `preview`, `mcp`, `agent`, `export`, `huggingface`, `publish`, `auth`, `langchain`, `db`, `deploy`, `vision` and `transcribe`. Pick the ones a task needs, or take `all`: ```bash # Database-backed data sources pip install 'potato-annotation[db]' # What a coding agent needs: the MCP server plus browser rendering pip install 'potato-annotation[agent]' # Every extra except vision, which stays out because torch is multi-gigabyte pip install 'potato-annotation[all]' ``` ## Troubleshooting ### Permission errors If you encounter permission errors, try: ```bash pip install --user potato-annotation ``` ### Virtual environment (recommended) We recommend using a virtual environment: ```bash python -m venv potato-env source potato-env/bin/activate # On Windows: potato-env\Scripts\activate pip install potato-annotation ``` ## Next Steps - Follow the [Quick Start](/docs/getting-started/quick-start) guide - Learn about [Configuration Basics](/docs/getting-started/configuration-basics) --- # What's New Section: Getting Started Source: https://www.potatoannotator.com/docs/getting-started/whats-new-v2 > What's new in Potato v2.x. Version 2.8 adds computer vision, 3D point clouds, depth maps, robot episodes and world-model evaluation, plus chance-corrected agreement over geometry and time. 61 annotation types, 24 display types. This page covers new features and improvements across Potato v2.x releases. --- ## Potato 2.9.0 *Released September 2026* Potato imports projects from brat, doccano, Prodigy, CoNLL and REFI-QDA, and transcribes and diarizes audio locally. Exports record whether an image shape was drawn by a person, proposed by a model or imported, and which room a vote was cast in. Most of the release is bug fixes. **Run `potato validate --strict` on each config before upgrading.** Several changes can make a working 2.8.2 study refuse to start or behave differently: duplicate scheme names in one phase are refused, `min_annotators_per_instance` became an alias for the per-item cap, `require_fully_annotated: true` is enforced, a phase that fails to load aborts the boot, and span offsets are code points rather than UTF-16 units. Agreement values for set-valued, order-valued, matrix and labelled ordinal schemes change and should be recomputed. The [release notes](https://github.com/davidjurgens/potato/blob/master/docs/releasenotes/v2.9.0.md#upgrading) have the full list. ```bash pip install --upgrade potato-annotation ``` ### Import and export - **Importers for brat, doccano, Prodigy, CoNLL-2003/CoNLL-U and REFI-QDA `.qdpx`**, and a `.qdpx` exporter. See [Importing a Project](/docs/tools/importing-projects). - **Provenance in exports.** Each image shape records whether a person drew it or a model proposed it (`source`, `ai_model`, `confidence`, `edited`, `carried_over`, `import_format`). Votes cast in a multiplayer room carry the room, role, initial vote, whether it changed after the reveal, and the voter count. - Span exports in csv, jsonl and parquet include the covered text. - `potato export --format adjudication` is new, and every schema type now has an "adopt an answer" adjudication control (only radio and likert had one). ### Audio, video and transcripts - **Local transcription and speaker diarization** with faster-whisper and sherpa-onnx, no PyTorch and no Hugging Face token: install `potato-annotation[transcribe]`, then run `potato transcripts --transcribe --diarize --num-speakers N -o out.json`. See [Transcript Formats](/docs/annotation-types/transcript-formats#transcribing-locally). - `segment_schemes` renders any annotation type inside an audio or video segment. It used to be accepted and ignored. ### Agreement, quality and AI - Agreement over time, joined against codebook revisions, with a re-calibration trigger on `/admin/iaa`. - A model-output review mode that includes the empty-prediction slice, so recall can be computed. - `ai_budget.cap_usd` refuses to start a run it cannot afford, and `ai_budget.prices` sets per-model rates in the config. - `ai_support` records `final_annotation` and `pre_annotation_seeds`, so acceptance rates can be measured from your own data. ### Deployment and interface - Reverse-proxy and path-prefix support on every page ([#168](https://github.com/davidjurgens/potato/pull/168), Aldo Costa). See [Reverse Proxy](/docs/deployment/reverse-proxy). - `card_sort` works without a mouse, and moves are announced to screen readers. - `resizable`, `max_height` and `min_height` work on every display type. ### Fixes that affect results - `/admin/api/agreement` never produced a number, and compared stored values rather than label names, so every categorical schema showed perfect agreement. - The second annotator on a two-per-item study was sent to the done page, and every answer they then gave was lost. - AI suggestions were about the wrong item under every assignment strategy except `fixed_order`. - Sign-ups were never saved under the default in-memory config, so after a restart anyone could claim an existing username. Thanks to Aldo Costa ([@Eyecatch3r](https://github.com/Eyecatch3r)) for [#168](https://github.com/davidjurgens/potato/pull/168) and [#171](https://github.com/davidjurgens/potato/pull/171), and to [@ruthenian8](https://github.com/ruthenian8) for reporting and diagnosing [#170](https://github.com/davidjurgens/potato/issues/170). --- ## Potato 2.8.2 *Released August 2026* 2.8.1 did not contain two of the five security fixes its release notes described: the SSRF fix on the media endpoints (`GHSA-9394-85jj-2mhj`) and the debug-mode hardening (`GHSA-r4ww-vc5h-35qf`). 2.8.2 ships both. If you upgraded to 2.8.1 for either issue, upgrade again. See the [2.8.2 release notes](https://github.com/davidjurgens/potato/blob/master/docs/releasenotes/v2.8.2.md). --- ## Potato 2.8.1 *Released August 2026* A security release. It fixed remote code execution through the live coding agent (`GHSA-cm4p-5rfv-x252`), an admin config readable without the admin key, and a trace-ingestion webhook that accepted every request. **Upgrade if you run Potato anywhere other than localhost**, and go straight to 2.8.2, which carries the two fixes 2.8.1 was missing. --- ## Potato 2.8.0 *Released August 2026* Potato was a text annotation platform with an image mode. It now covers images, video, gigapixel scans, 3D point clouds, depth maps, robot episodes, generative-video rollouts and vision-language grounding. For each of them it reports chance-corrected agreement. **Upgrade if you installed from PyPI.** Wheels up to 2.7.1 declared package data with a single-level glob, so 27 templates in subdirectories were missing from every wheel. Solo mode's setup routes, the admin pages, judge calibration and the corpus map were all broken for anyone who ran `pip install`; anyone running from a git checkout never saw it. Fixed in [PR #164](https://github.com/davidjurgens/potato/pull/164) by [@0x6362](https://github.com/0x6362). ```bash pip install --upgrade potato-annotation ``` ### Agreement over geometry and time - **Chance-corrected agreement for spatial labels**, decomposed into detection (did annotators find the same objects), classification (did they call them the same thing), and localization (did they put them in the same place, reported as σ against an empirical chance baseline plus a KS test). - **STAPLE** for mask consensus. It estimates a latent boundary and a sensitivity and specificity for each rater. On a corpus where two careful annotators are outnumbered three-to-two by noisy ones, majority vote scores Dice 0.846 against truth and STAPLE scores 1.000. - **Exact rotated 3D IoU**, so cuboid agreement is correct for drone, handheld and indoor data rather than only level boxes. - **Temporal agreement reported as a sweep** across matching tolerances, because agreement at 0.25 s is a different claim from agreement at 2 s. - Undefined values explain themselves. α is genuinely undefined on a unanimous corpus, and a bare `NaN` reads as a broken computation. This also fixes a bug: adjudication compared the *keys* of an annotation, and image schemas store everything under one key named `_data`. Every image pair therefore scored 1.0 agreement, so two annotators who agreed on nothing looked unanimous and no image was ever routed for review. ### Vision - **Geometry primitives**: polylines, ellipses, 2D cuboids, keypoint sets with skeleton topology and COCO visibility flags, and instance-keyed brush masks. - **Click-to-segment in the browser**: MobileSAM under ONNX Runtime Web, about 132 ms per click, no GPU and no per-click network call. - **Open-vocabulary text prompting**: type a phrase, get every match boxed. Grounding DINO, Apache-2.0, also in the browser. - **Video mask propagation** with SAM 2's memory modules, measured at 0.974–0.979 IoU per frame with no decay across the sequence. This one runs server-side by design; the browser has a separate, lighter carry-forward path. - **Deep zoom** for gigapixel images, served as both DZI and IIIF Image API 3.0, with brush masks at the source's full resolution. - **Media ingest**: multi-page 16-bit TIFF, HEIC, camera RAW, and ProRes/MKV/MOV through a cached server-side proxy. - **15 import and 29 export formats**, 11 of them round-tripping. Darwin works in both directions. Image annotation keyboard shortcuts now follow V7 conventions by default: `b` brush, `r` rectangle, `f` fill, `k` keypoint, `v` select. Set `keybinding_profile: legacy` on the schema to restore the old bindings for a study already in the field. ### 3D, depth and robotics - **`spatial_annotation`** for PCD, PLY, LAS, KITTI `.bin` and `.xyz` clouds, with 3D cuboids, points, polylines and per-point segments, octree level-of-detail, and orthographic slab panels. Rotation is stored as a quaternion, so the mounting tilt a yaw-only field would silently discard survives a KITTI round-trip. - **Depth maps** in 16-bit PNG/TIFF, NPY, PFM and EXR, with windowing, colormaps, a metres readout under the cursor, and unprojection into the same 3D viewer. - **Calibration-aware projection**, so a cuboid drawn in 3D appears in every camera image and can be checked in 2D. - **`episode_annotation`** puts N synchronized video streams and M robot time-series lanes on one timeline, with phase segmentation, per-phase outcomes and dense reward curves. Imports LeRobot v2, RLDS/TFDS and HDF5. Lane downsampling preserves minima and maxima, so a one-frame force spike survives the trip to a 300-pixel lane. ### Evaluation - **`rollout_evaluation`** shows 2–N generated videos on one clock and asks the annotator to mark the frame at which the world stops making sense, then tag which physical or causal property broke. Panels can be blinded and stably shuffled per annotator. - **`grounding_eval`** and **`region_caption`** score grounding by IoU at four thresholds and pointing by point-in-region hit rate, because a point has no area and IoU against one is always zero. Ungroundedness is counted separately. ### Everything else - **Threaded conversations**: the `dialogue` display renders reply structure from each turn's `reply_to`. `potato convokit` imports any ConvoKit corpus and `--format convokit` exports back, with no `convokit` dependency. - **Drawing telemetry**: time per shape, stroke dynamics, revision counts and AI-suggestion accept latency. Pre-labels accepted wholesale make agreement rise while accuracy does not, because annotators who accept the same suggestion are unanimous by construction. Timing is a process signal that separates review from rubber-stamping where the labels cannot. - **Live database ingestion**: rows created after startup become annotatable within `poll_interval_seconds` with no restart ([#166](https://github.com/davidjurgens/potato/issues/166)). - **Machine-readable specs**: a JSON Schema covering 159 config keys, 61 annotation types and 24 display types, plus an OpenAPI document with 419 paths. Both are generated from the code and CI fails when either drifts, so an editor or a coding agent can check a config without inventing options. - **No outbound requests**: every stylesheet, script, font and icon now serves from the install across all 14 templates. `styles.css` had been opening with an `@import` of Google Fonts, which sent every annotator's IP address to a third party on page load, and the login page carried a third-party favicon. Both are gone. - **Admin Instances tab** went from quadratic to linear: 2,000 items dropped from 15.1 s to 23 ms. --- ## Potato 2.7.2 *Released August 2026* Keystroke logging for free-text fields. Potato records the timing behind an answer and never the characters, which turns out to be enough to tell typing apart from pasting. - **[Keystroke logging](/docs/features/keystroke-logging)**: content-blind capture of pauses, bursts, revisions and pastes on every free-text field, with about forty summary features computed per response. Off by default. - **[Writing-process detection](/docs/features/writing-process-detection)**: six named rules with explicit thresholds that report the feature values behind every flag. Thresholds can be refitted to your own project, and there is a supervised path if you have labels. - **[Ethics guidance](/docs/features/keystroke-logging-ethics)**: sample consent language, retention and deletion, the base-rate problem, and why a flag is a prompt to look rather than a finding. - **Opt-in exports**: `export_include_typing_dynamics` writes a summary sidecar, and `--format keystrokes` writes the raw streams to Parquet. Neither is on by default, so behavioral data never lands in a dataset release by accident. Disclosure to annotators defaults to on, and turning it off logs a warning at startup. --- ## Potato 2.7.1 *Released July 2026* Transcript ingestion, on the premise that the transcript already exists. Someone ran Whisper, or downloaded captions, or was handed a corpus, and the next step should not be writing a conversion script. - **[21 transcript and subtitle formats](/docs/annotation-types/transcript-formats)**, up from 6. Whisper, WhisperX, whisper.cpp, AWS Transcribe, Deepgram, AssemblyAI, Rev.ai, SPoRC, SubRip, WebVTT, SubStation Alpha, TTML/DFXP, YouTube `json3` and `srv1`/`srv2`/`srv3`, NIST CTM, Praat TextGrid, and ELAN EAF. Detection is by file contents, not extension. - **Sidecar transcript files**: a data file can point at `media/int_001.srt` on disk instead of inlining the transcript. Paths resolve under `task_dir` through the standard path-security check, so your transcripts stay files you can diff and re-export. - **`potato transcripts`**: a converter that turns a folder of ASR output into an annotation-ready data file and pairs each transcript with its media by basename. `--dry-run` reports what it detected before writing anything, and `--emit-config` prints a matching config. - **One format vocabulary across four schemas**: `speech_transcript`, `voice_interaction`, and `tiered_annotation` now accept everything the `audio_dialogue` display accepts. Previously only the display did. - **[Annotating Whisper Transcripts](/docs/guides/annotating-whisper-transcripts)** and **[Annotating YouTube Subtitles](/docs/guides/annotating-youtube-subtitles)** walk through both starting points end to end. In 2.7.1 the ASR and diarization steps run upstream and Potato reads their output. Potato 2.9 added local transcription and diarization through `potato transcripts --transcribe --diarize`, and [Think-Aloud Mode](/docs/features/think-aloud) has transcribed annotators' own speech locally since 2.7.0. --- ## Potato 2.7.0 *Released July 2026* Potato 2.7 measures *how* judgments are made as well as recording what they were, both for the humans who annotate and for the AI models being evaluated. ### Measuring how humans label Seven opt-in features, none of which requires an LLM. - **[Psychometrics engine](/docs/features/psychometrics)**: a live item-response-theory layer that gives every label a posterior probability and a sensitivity interval (`p = 0.94 [0.88–0.97]`) instead of a bare majority vote. It estimates annotator ability and item difficulty from agreement patterns alone, routes items to the annotator whose judgment carries the most information, and flags likely codebook bugs. - **[Multiplayer Rooms](/docs/features/multiplayer-rooms)**: the calibration meeting your team already runs over screen-share, built into the tool and instrumented: blind vote, host reveal, discussion, revote, with a live Krippendorff's α meter showing what the session was worth. - **[Boundary Lab](/docs/features/boundary-lab)**: counterfactual probes after each label ("would that still hold?"), which turn ordinary annotation into contrast sets and catch inconsistent annotators without planting gold items. - **[Truth Serum](/docs/features/truth-serum)**: surprisingly-popular peer-prediction scoring, which beats majority vote where annotation is hard, with no gold labels. - **[Think-Aloud Mode](/docs/features/think-aloud)**: annotators talk while they work; speech-to-text runs fully locally and the verbatim transcript becomes the rationale. - **[Paper Mode](/docs/features/paper-mode)**: one command turns a project into a compilable LaTeX dataset report, agreement statistics and citations included. - **[Pocket Mode](/docs/features/pocket-mode)**: first-class mobile annotation with a swipeable card stack, offline sync, and PWA install. ### Evaluating AI agents The agent-evaluation suite grew to cover team structure and multimodal agents: a clickable [agent interaction graph](/docs/agent-evaluation/multi-agent-evaluation), cross-agent failure attribution, handoff review, per-agent and team scorecards, tool-contention timelines, and emergent-behavior tagging, plus [GUI/computer-use trajectories, full-duplex voice timelines, video temporal grounding, and document-table structure](/docs/agent-evaluation/multimodal-agent-evaluation). ### Everywhere else Cross-document event annotation, turn-level annotation, a living-document codebook with a full-page editor, PDF cross-page linking with optional OCR, RBAC roles with per-cohort schemas, admin and annotator dashboards in 10 languages, and a much lighter install, since AI SDKs now load lazily and boot time drops from roughly 2.0s to 0.7s. --- ## Potato 2.6.0 *Released June 2026* Potato 2.6 moves into qualitative data analysis and deepens the agent-evaluation toolkit. It adds QDA Mode, an LLM-as-judge calibration and alignment workflow with a signal-based triage queue, and trajectory-editing schemas that produce SFT and DPO training data. Potato is also **relicensed to GPL-3.0-or-later** (from PolyForm Shield). ### QDA Mode An opt-in `qda_mode` turns Potato into a collaborative qualitative-coding workspace. Enabling it composes a living codebook, in-vivo coding, analyst memos, cases, and full-text search, with defaults tuned for one analyst coding a whole corpus. ```yaml qda_mode: enabled: true # codebook + memos + cases + search codebook_invivo_key: i # mint a code from a text selection search: enabled: true annotator_claim: true ``` [Learn more about QDA Mode →](/docs/qda/qda-mode) ### LLM-as-Judge Calibration and Alignment Auto-label data with one or more LLM judges, then run a blind human calibration pass to measure accuracy, agreement, and calibration error. A separate single-judge alignment workflow tracks Cohen's kappa against your human gold labels as you refine the rubric, with an optional inline verdict during annotation. [Learn more about Judge Calibration →](/docs/agent-evaluation/judge-calibration) · [Judge Alignment →](/docs/agent-evaluation/judge-alignment) ### Signal-Based Triage Queue Prioritize the annotation queue by a per-item quality signal (an agent error, a production thumbs-down, a low score, or any custom field), so reviewers see the most-suspect items first instead of in arrival order. ```yaml triage: enabled: true signal_field: quality_score invert_signal: true assignment_strategy: priority ``` [Learn more about the Triage Queue →](/docs/agent-evaluation/triage-queue) ### Trajectory Editing for SFT/DPO The new `trajectory_edit` and `trajectory_correction` schemas let annotators rewrite agent trace steps. The exporter turns each original/corrected pair into supervised fine-tuning targets (`trajectory_sft.jsonl`) and DPO preference pairs (`trajectory_dpo.jsonl`). [Learn more about Trajectory Editing →](/docs/agent-evaluation/trajectory-correction) ### eval_trace Display A three-pane agent-trace display (Reasoning, Function Calls, and Final Answer) built for continuous evaluation, where traces arrive over a webhook, a Langfuse poller, or a watched directory and are judged as they land. [Learn more about eval_trace →](/docs/agent-evaluation/eval-trace) ### Workflow and Assignment - **Heterogeneous coverage**: per-item annotator caps, inter-annotator agreement reporting, and adjudication routing for tasks where items need different numbers of annotators. - **Reclaim abandoned assignments**: recover assignments left by Prolific or QC-blocked workers, with configurable retention and idempotent reclaim. - **Custom Batch assignment strategy**: assign predefined batches of items to specific annotators. - **Reverse-proxy URL prefixes**: serve Potato under a sub-path behind a reverse proxy. ### Licensing Potato is now released under **GPL-3.0-or-later**, relicensed from PolyForm Shield. You can use, modify, and redistribute it, including commercially, as long as derivative works stay under the GPL. See the [About page](/about#license) for details. ### Performance and Stability - **Roughly 3× faster boot.** The ML stack is no longer eager-loaded at startup: import time dropped from about 6.5s to 2s, a 50k-item boot from about 10s to 5.7s, and resident memory from about 750MB to 365MB. - **Schema rename.** `annotation_type: highlight` is now `span`, with a migration in place. Update old configs by renaming the type; existing `span` configs are unaffected. - A broad QA-hardening wave touched route registration, training phases, Prolific, persistence, export and survey handling, webhooks, solo mode, and active learning. --- ## Potato 2.5.0 *Released 2026* A qualitative-coding wave that makes Potato credible for qualitative-research workflows alongside its NLP and ML annotation surface. - **Inter-annotator agreement**: Cohen's kappa (pairwise) and Fleiss' kappa (N raters) in addition to Krippendorff's alpha, surfaced via the admin agreement API. See the [inter-annotator agreement guide](/docs/guides/inter-annotator-agreement). - **New exporters**: `codebook` (per-schema CSV with code hierarchy, color, description, and use count) and `quotation_report` (per-span CSV with text, offsets, source document, and coder). - **Code analytics**: admin endpoints for pairwise code co-occurrence and a codes-by-attribute crosstab over existing instance metadata. --- ## Potato 2.4.5 *Released 2026* A stability and tooling release. - **Validated refinement**: a pluggable framework for iteratively improving solo-mode annotation guidelines, with guardrails against guideline contradictions and metadata leakage. - **Config validator**: a new `python -m potato.validate_cli` CLI that checks config keys against the known-config schema. - **Security**: a fix for a path-traversal sibling-prefix bypass in path validation ([GHSA-q9m2-fhv9-3jcf](https://github.com/davidjurgens/potato/security/advisories/GHSA-q9m2-fhv9-3jcf)). - Fixes across save/navigation state-sync, Prolific integration, multi-phase navigation, and quality-control feedback. --- ## Potato 2.3.0 *Released March 9, 2026* Potato 2.3 is the largest release in Potato's history, introducing agentic annotation, Solo Mode, Best-Worst Scaling, SSO/OAuth authentication, Parquet export, 15 new demo projects, and security hardening. ### Agentic Annotation A complete system for evaluating AI agents through human annotation. Includes 12 trace format converters, 3 specialized display types, and 9 pre-built annotation schemas. **12 Trace Format Converters**: Import agent traces from OpenAI, Anthropic, SWE-bench, OpenTelemetry, MCP, CrewAI/AutoGen/LangGraph, LangChain, LangFuse, ReAct, WebArena/VisualWebArena, ATIF, and raw browser recordings. Auto-detection available. Conversion runs as a CLI step before the server starts: ```bash python -m potato.trace_converter \ --input traces.json \ --input-format react \ --output data/traces.jsonl ``` The converted file is then an ordinary data file, rendered by one of the agent display types: ```yaml data_files: - "data/traces.jsonl" instance_display: fields: - key: steps type: agent_trace label: "Agent trajectory" ``` **3 Display Types:** - **Agent Trace Display**: Color-coded step cards with collapsible observations, JSON pretty-printing, and timeline sidebar for tool-using agents - **Web Agent Trace Display**: Full screenshots with SVG overlays showing click targets, text inputs, and scroll actions; filmstrip navigation for browsing agents - **Interactive Chat Display**: Live chat mode (annotator interacts with agent via proxy) and trace review mode for conversational agents **Per-Turn Ratings**: Rate individual steps alongside the overall trace for fine-grained evaluation. **9 Pre-Built Schemas**: `agent_task_success`, `agent_step_correctness`, `agent_error_taxonomy`, `agent_safety`, `agent_efficiency`, `agent_instruction_following`, `agent_explanation_quality`, `agent_web_action_correctness`, `agent_conversation_quality`. **Agent Proxy System**: OpenAI, HTTP, and echo proxies for live agent evaluation. [Learn more about Agentic Annotation →](/docs/features/agentic-annotation) --- ### Solo Mode A workflow where a single human annotator collaborates with an LLM to label an entire dataset, achieving 95%+ agreement with multi-annotator pipelines while requiring only 10-15% of total human labels. Solo Mode runs as a phase state machine rather than a fixed sequence. It moves through setup, prompt review, an optional edge-case loop, parallel annotation, disagreement resolution, active annotation with periodic and rule review, autonomous labeling and final validation. Five of those phases can return to prompt review when the prompt turns out to be the problem, and final validation falls back to active annotation when the sample does not clear the threshold. ```yaml solo_mode: enabled: true labeling_models: - endpoint_type: openai model: "gpt-4o" api_key: ${OPENAI_API_KEY} thresholds: end_human_annotation_agreement: 0.92 minimum_validation_sample: 50 state_dir: "solo_state" ``` **Multi-Signal Instance Prioritization**: `instance_selection` weights seven signals (low confidence, diversity, LLM-predicted, disagreement, random, cartography, edge-case rule) to choose what the human sees next. [Learn more about Solo Mode →](/docs/features/solo-mode) --- ### Best-Worst Scaling Efficient comparative annotation where annotators select the best and worst items from tuples. Automatic tuple generation with balanced incomplete block designs and three scoring methods (Counting, Bradley-Terry, Plackett-Luce). ```yaml annotation_schemes: - annotation_type: bws name: fluency description: "Which sentence reads most fluently?" tuple_size: 4 best_description: "Most Fluent" worst_description: "Least Fluent" ``` [Learn more about Best-Worst Scaling →](/docs/annotation-types/best-worst-scaling) --- ### SSO & OAuth Authentication Production-ready authentication with Google OAuth (domain restriction), GitHub OAuth (organization restriction), and generic OIDC (Okta, Azure AD, Auth0, Keycloak). Supports auto-registration, mixed mode, and session management. ```yaml authentication: method: "oauth" providers: google: client_id: ${GOOGLE_CLIENT_ID} client_secret: ${GOOGLE_CLIENT_SECRET} allowed_domain: "umich.edu" user_identity_field: "email" auto_register: true secret_key: ${POTATO_SECRET_KEY} ``` [Learn more about SSO & OAuth →](/docs/deployment/sso-oauth) --- ### Parquet Export Export annotations to Apache Parquet format, producing three structured files: `annotations.parquet`, `spans.parquet`, and `items.parquet`. Supports snappy, gzip, zstd, lz4, and brotli compression, incremental export, and date/annotator partitioning. Compatible with pandas, DuckDB, PyArrow, Polars, and Hugging Face Datasets. ```yaml export_annotation_format: - parquet ``` [Learn more about Parquet Export →](/docs/features/parquet-export) --- ### 15 New Demo Projects New demos in `project-hub/` covering agentic annotation (5 demos), Solo Mode (3 demos), Best-Worst Scaling (3 demos), authentication (2 demos), and export workflows (2 demos). Start any demo with `potato start config.yaml`. --- ### Security Hardening - Cryptographically secure session tokens with configurable expiration - CSRF protection enabled by default - Rate limiting on authentication endpoints - Input sanitization for user-provided content - Dependency audit with all packages updated - Content Security Policy headers --- ### Other Improvements - Custom trace converters for unsupported agent frameworks - Hybrid Solo Mode with multi-annotator verification sampling - BWS admin dashboard tab with score convergence charts - Incremental Parquet export with date partitioning --- ### v2.2 vs v2.3 Comparison | Feature | v2.2 | v2.3 | |---------|------|------| | Agentic Annotation | Not available | 12 converters, 3 displays, 9 schemas | | Solo Mode | Not available | Human-LLM workflow driven by a phase state machine | | Best-Worst Scaling | Not available | BWS with 3 scoring methods | | Authentication | Username only | + Google OAuth, GitHub OAuth, OIDC | | Parquet Export | Not available | 3-file Parquet with 6 compression options | | Demo Projects | 125+ | 140+ (15 new) | | Security | Basic | CSRF, rate limiting, CSP, secure sessions | --- ## Potato 2.2.0 *Released February 20, 2026* Potato 2.2 is a major feature release with 9 new annotation schemas, a pluggable export system, MACE competence estimation, 55 validated survey instruments, and remote data sources. ### New Annotation Schemas (9) **Event Annotation**: N-ary event structures with trigger spans and typed argument roles. Annotate events like ATTACK, HIRE, and TRAVEL with constrained entity arguments and hub-spoke arc visualization. ```yaml annotation_schemes: - annotation_type: event_annotation name: events description: "Mark the event triggers and their arguments" span_schema: entities event_types: - type: "ATTACK" trigger_labels: ["EVENT_TRIGGER"] arguments: - role: "attacker" entity_types: ["PERSON", "ORGANIZATION"] required: true ``` [Learn more about Event Annotation →](/docs/annotation-types/event-annotation) **Entity Linking**: Link span annotations to external knowledge bases (Wikidata, UMLS, custom REST APIs). Add an `entity_linking:` block to any span schema to enable KB search and linking. [Learn more about Entity Linking →](/docs/annotation-types/entity-linking) **Triage**: Prodigy-style accept/reject/skip interface for rapid data screening. Customizable labels, keyboard shortcuts, and auto-advance for high-throughput annotation. [Learn more about Triage →](/docs/annotation-types/triage) **Pairwise Comparison**: Compare two items with binary (click preferred tile) or scale (slider) modes. Supports `items_key`, `allow_tie`, `scale:` block with configurable range. [Learn more about Pairwise Comparison →](/docs/annotation-types/pairwise-comparison) **Conversation Trees**: Annotate hierarchical conversation structures with per-node ratings, path selection, and branch comparison. [Learn more about Conversation Trees →](/docs/annotation-types/conversation-trees) **Coreference Chains**: Group coreferring text mentions into chains with visual indicators. Supports entity types, singleton control, and multiple highlight modes. [Learn more about Coreference Chains →](/docs/annotation-types/coreference) **Segmentation Masks**: New `fill`, `eraser`, and `brush` tools for pixel-level image segmentation. **Bounding Box for PDF/Documents**: Draw boxes on PDF pages for document annotation tasks. **Discontinuous Spans**: `allow_discontinuous: true` enables selecting non-contiguous text segments as a single span. --- ### Intelligent Annotation **MACE Competence Estimation**: Variational Bayes EM algorithm that jointly estimates true labels and annotator competence scores (0.0-1.0). Works with radio, likert, select, and multiselect schemas. ```yaml mace: enabled: true trigger_every_n: 10 min_annotations_per_item: 3 ``` [Learn more about MACE →](/docs/features/mace) **Option Highlighting**: LLM-based highlighting of likely correct options for discrete annotation tasks. Highlights top-k options with a star indicator while dimming less-likely options. ```yaml ai_support: option_highlighting: enabled: true top_k: 3 dim_opacity: 0.4 ``` [Learn more about Option Highlighting →](/docs/features/option-highlighting) **Diversity Ordering**: Embedding-based clustering and round-robin sampling to ensure annotators see diverse content rather than similar items in sequence. ```yaml assignment_strategy: diversity_clustering diversity_ordering: enabled: true prefill_count: 100 ``` [Learn more about Diversity Ordering →](/docs/features/diversity-ordering) --- ### Export System A new pluggable export CLI (`python -m potato.export`) converts annotations to 6 industry-standard formats: COCO, YOLO, Pascal VOC, CoNLL-2003, CoNLL-U, and Segmentation Masks. ```bash python -m potato.export --config config.yaml --format coco --output ./export/ ``` [Learn more about Export Formats →](/docs/features/export-formats) --- ### Remote Data Sources Load annotation data from URLs, S3, Google Drive, Dropbox, Hugging Face, Google Sheets, and SQL databases via the new `data_sources:` config block. Includes partial loading, caching, and credential management. [Learn more about Remote Data Sources →](/docs/features/remote-data-sources) --- ### Survey Instruments 55 validated questionnaires across 8 categories (Personality, Mental Health, Affect, Self-Concept, Social Attitudes, Response Style, Short-Form, Demographics). Use in prestudy/poststudy phases with `instrument: "tipi"`. [Learn more about Survey Instruments →](/docs/features/survey-instruments) --- ### Other Improvements - Video object tracking with keyframe interpolation - External AI config file support - Form layout grid improvements - Format handlers for PDF, Word, code, and spreadsheets --- ## Potato 2.1.0 *Released February 5, 2026* Potato 2.1 introduces the instance display system, visual AI support, span linking, multi-field span annotation, and layout customization. ### Instance Display System A new `instance_display` config block that separates content display from annotation. Display any combination of images, videos, audio, text, and dialogues alongside any annotation schemes. ```yaml instance_display: fields: - key: image_url type: image display_options: max_width: 600 zoomable: true - key: description type: text annotation_schemes: - annotation_type: radio name: category description: "What category best describes this image?" labels: [nature, urban, people] ``` Supports 24 display types including `text`, `html`, `image`, `video`, `audio`, `depth_map`, `dialogue`, `conversation_tree`, `pairwise`, `code`, `spreadsheet`, `document`, `pdf`, `gallery`, and five agent-trace displays. [Learn more about Instance Display →](/docs/core-concepts/instance-display) --- ### Multi-Field Span Annotation Span annotation schemes now support a `target_field` option to annotate across multiple text fields in the same instance. ```yaml annotation_schemes: - annotation_type: span name: source_entities description: "Label the entities in the source document" labels: [PERSON, ORGANIZATION] - annotation_type: span name: summary_entities description: "Label the entities in the summary" labels: [PERSON, ORGANIZATION] ``` [Learn more about Span Annotation →](/docs/annotation-types/span-annotation) --- ### Span Linking A new `span_link` annotation type for creating typed relationships between annotated spans. Supports directed and undirected links, n-ary relationships, visual arc display, and label constraints. ```yaml annotation_schemes: - annotation_type: span name: entities description: "Highlight the named entities in the text" labels: - name: "PERSON" color: "#3b82f6" - name: "ORGANIZATION" color: "#22c55e" - annotation_type: span_link name: relations description: "Link the entities that stand in a relation" span_schema: entities link_types: - name: "WORKS_FOR" directed: true allowed_source_labels: ["PERSON"] allowed_target_labels: ["ORGANIZATION"] color: "#dc2626" ``` [Learn more about Span Linking →](/docs/annotation-types/span-linking) --- ### Visual AI Support Four new vision endpoints for AI-powered image and video annotation assistance: - **YOLO**: Fast local object detection - **Ollama Vision**: Local vision-language models (LLaVA, Qwen-VL) - **OpenAI Vision**: GPT-4o cloud vision - **Anthropic Vision**: Claude with vision Features include object detection, pre-annotation, classification, hints, scene detection, keyframe detection, and object tracking. [Learn more about Visual AI Support →](/docs/features/visual-ai-support) --- ### Layout Customization Create sophisticated custom visual layouts using HTML templates and CSS. Potato generates an editable layout file, or you can provide a fully custom template with grid layouts, color-coded options, and section styling. ```yaml task_layout: layouts/custom_task_layout.html ``` Three example layouts included: content moderation, dialogue QA, and medical review. [Learn more about Layout Customization →](/docs/features/layout-customization) --- ### Label Rationales A fourth AI capability that generates balanced explanations for why each label might apply, helping annotators understand different classification perspectives. ```yaml ai_support: features: rationales: enabled: true ``` [Learn more about AI Support →](/docs/features/ai-support) --- ### Other Improvements - 50+ new tests for improved reliability - Responsive design improvements - Enhanced project-hub organization with layout examples - Bug fixes across annotation types ### v2.0 vs v2.1 Comparison | Feature | v2.0 | v2.1 | |---------|------|------| | Instance Display | Via annotation hacks | Dedicated `instance_display` block | | Span Targets | Single text field | Multi-field with `target_field` | | Span Linking | Not available | Full `span_link` type | | Visual AI | Not available | YOLO, Ollama Vision, OpenAI Vision, Anthropic Vision | | Layout Customization | Basic auto-generated | Auto-generated + custom templates | | AI Capabilities | 3 (hints, keywords, suggestions) | 4 (+ rationales) | --- ## Potato 2.0 Potato 2.0 is a major release that adds AI assistance, active learning, training phases, multi-phase workflows, and a MySQL backend. ### AI Support Integrate Large Language Models to assist annotators with intelligent hints, keyword highlighting, and label suggestions. **Supported providers:** - OpenAI (GPT-4, GPT-3.5) - Anthropic (Claude 3, Claude 3.5) - Google (Gemini) - Ollama (local models) - vLLM (self-hosted) ```yaml ai_support: enabled: true endpoint_type: openai ai_config: model: gpt-4 api_key: ${OPENAI_API_KEY} features: hints: enabled: true label_suggestions: enabled: true ``` [Learn more about AI Support →](/docs/features/ai-support) --- ### Audio Annotation Full-featured audio annotation with waveform visualization powered by Peaks.js. Create segments, label time regions, and annotate speech with keyboard shortcuts. **Key features:** - Waveform visualization - Segment creation and labeling - Per-segment annotation questions - 15+ keyboard shortcuts - Server-side waveform caching ```yaml annotation_schemes: - annotation_type: audio_annotation name: speakers description: "Mark each speaker turn in the audio" mode: label labels: - Speaker A - Speaker B ``` [Learn more about Audio Annotation →](/docs/features/audio-annotation) --- ### Active Learning Automatically prioritize annotation instances based on model uncertainty. Train classifiers on existing annotations and focus annotators on the most informative examples. **Capabilities:** - Multiple classifier options (LogisticRegression, RandomForest, SVC, MultinomialNB) - Various vectorizers (TF-IDF, Count, Hashing) - Model persistence across restarts - LLM-enhanced selection - Multi-schema support ```yaml active_learning: enabled: true schema_names: - sentiment min_instances_for_training: 30 update_frequency: 50 classifier: type: LogisticRegression ``` [Learn more about Active Learning →](/docs/features/active-learning) --- ### Training Phase Qualify annotators with practice questions before the main task. Provide immediate feedback and ensure quality through configurable passing criteria. **Features:** - Practice questions with known answers - Immediate feedback and explanations - Configurable passing criteria - Retry options - Progress tracking in admin dashboard ```yaml phases: training: enabled: true data_file: "data/training.json" passing_criteria: min_correct: 8 total_questions: 10 ``` [Learn more about Training Phase →](/docs/features/training-phase) --- ### Enhanced Admin Dashboard Comprehensive monitoring and management interface for annotation tasks. **Dashboard tabs:** - **Overview**: High-level metrics and completion rates - **Annotators**: Performance tracking, timing analysis - **Instances**: Browse data with disagreement scores - **Configuration**: Real-time settings adjustment ```yaml admin_api_key: ${ADMIN_API_KEY} ``` [Learn more about Admin Dashboard →](/docs/features/admin-dashboard) --- ### Database Backend MySQL support for large-scale deployments with connection pooling and transaction support. ```yaml database: type: mysql host: localhost database: potato_db username: ${DB_USER} password: ${DB_PASSWORD} ``` Potato automatically creates required tables on first startup. --- ### Annotation History Complete tracking of all annotation changes with timestamps, user IDs, and action types. Enables auditing and behavioral analysis. ```json { "history": [ { "timestamp": "2024-01-15T10:30:00Z", "user": "annotator_1", "action": "create", "schema": "sentiment", "value": "Positive" } ] } ``` --- ### Multi-Phase Workflows Build complex annotation workflows with multiple sequential phases: 1. **Consent** - Informed consent collection 2. **Pre-study** - Demographics and screening 3. **Instructions** - Task guidelines 4. **Training** - Practice questions 5. **Annotation** - Main task 6. **Post-study** - Feedback surveys ```yaml phases: consent: enabled: true data_file: "data/consent.json" prestudy: enabled: true data_file: "data/demographics.json" training: enabled: true data_file: "data/training.json" poststudy: enabled: true data_file: "data/feedback.json" ``` [Learn more about Multi-Phase Workflows →](/docs/features/surveyflow) --- ## v2.0 Configuration Changes ### New Configuration Structure Potato 2.0 uses a cleaner configuration format: **v1 (old):** ```yaml data_files: - data.json id_key: id text_key: text output_file: annotations.json ``` **v2 (new):** ```yaml data_files: - "data/data.json" item_properties: id_key: id text_key: text output_annotation_dir: "output/" export_annotation_format: "jsonl" ``` ### Security Requirement Configuration files must now be located within the `task_dir`: With the config beside the data, the task directory is the current one: ```yaml task_dir: "." ``` With the config in a subdirectory, name the project directory it belongs to: ```yaml task_dir: "my_project/" ``` --- ## Quick Comparison | Feature | v1 | v2.0 | v2.1 | v2.2 | v2.3 | |---------|----|----|------|------|------| | AI/LLM Support | No | Yes | Yes + Visual AI + Rationales | + Option Highlighting | + Solo Mode | | Agentic Annotation | No | No | No | No | 12 converters, 3 displays | | Best-Worst Scaling | No | No | No | No | Yes (3 scoring methods) | | Audio Annotation | Basic | Full waveform | Full waveform | Full waveform | Full waveform | | Active Learning | No | Yes | Yes | Yes + Diversity Ordering | + Solo Mode integration | | Instance Display | No | No | Yes | Yes | Yes | | Span Linking | No | No | Yes | Yes | Yes | | Event Annotation | No | No | No | Yes | Yes | | Entity Linking | No | No | No | Yes | Yes | | Pairwise/Triage/Coreference/Trees | No | No | No | Yes | Yes | | Layout Customization | No | Auto-generated | Auto + Custom templates | Auto + Custom templates | Auto + Custom templates | | Training Phase | No | Yes | Yes | Yes | Yes | | Admin Dashboard | Basic | Enhanced | Enhanced | Enhanced + MACE | + BWS tab, Solo Mode | | Database Backend | File only | File + MySQL | File + MySQL | File + MySQL | File + MySQL | | Export CLI | No | No | No | Yes (COCO, YOLO, CoNLL, etc.) | + Parquet | | Authentication | Username | Username | Username | Username | + Google/GitHub OAuth, OIDC | | Survey Instruments | No | No | No | 55 validated questionnaires | 55 validated questionnaires | | Remote Data Sources | No | No | No | S3, GDrive, HuggingFace, etc. | S3, GDrive, HuggingFace, etc. | --- ## Migration Guide ### Updating Your Configuration (v1 to v2) 1. **Data configuration** ```yaml # Old id_key: id text_key: text # New item_properties: id_key: id text_key: text ``` 2. **Output configuration** ```yaml # Old output_file: annotations.json # New output_annotation_dir: "output/" export_annotation_format: "jsonl" ``` 3. **Config file location** Ensure your config file is inside the project directory. ### Starting the Server ```bash # v2 command python -m potato start config.yaml -p 8000 # Or shorthand potato start config.yaml ``` --- ## Getting Started Ready to try Potato? Start with the [Quick Start Guide](/docs/getting-started/quick-start) or explore specific features: **v2.3 Features:** - [Agentic Annotation](/docs/features/agentic-annotation) - Evaluate AI agents with trace converters and step-level displays - [Solo Mode](/docs/features/solo-mode) - Human-LLM collaborative labeling - [Best-Worst Scaling](/docs/annotation-types/best-worst-scaling) - Comparative annotation with scoring - [SSO & OAuth](/docs/deployment/sso-oauth) - Google, GitHub, and OIDC authentication - [Parquet Export](/docs/features/parquet-export) - Columnar data export **v2.2 Features:** - [Event Annotation](/docs/annotation-types/event-annotation) - N-ary event structures - [Entity Linking](/docs/annotation-types/entity-linking) - Knowledge base linking - [Triage](/docs/annotation-types/triage) - Rapid data screening - [Coreference Chains](/docs/annotation-types/coreference) - Entity coreference - [Conversation Trees](/docs/annotation-types/conversation-trees) - Hierarchical dialogue annotation - [MACE](/docs/features/mace) - Annotator competence estimation - [Option Highlighting](/docs/features/option-highlighting) - AI-assisted option guidance - [Diversity Ordering](/docs/features/diversity-ordering) - Embedding-based item ordering - [Export Formats](/docs/features/export-formats) - Export CLI with 6 formats - [Remote Data Sources](/docs/features/remote-data-sources) - Cloud data loading - [Survey Instruments](/docs/features/survey-instruments) - 55 validated questionnaires **v2.1 Features:** - [Instance Display](/docs/core-concepts/instance-display) - Multi-modal content display - [Visual AI Support](/docs/features/visual-ai-support) - AI for image and video annotation - [Span Linking](/docs/annotation-types/span-linking) - Entity relationship annotation **Core Features:** - [AI Support](/docs/features/ai-support) - Intelligent annotation assistance - [Active Learning](/docs/features/active-learning) - Smart instance prioritization - [Audio Annotation](/docs/features/audio-annotation) - Waveform-based annotation - [Training Phase](/docs/features/training-phase) - Annotator qualification - [Admin Dashboard](/docs/features/admin-dashboard) - Monitoring and management --- # Configuration Basics Section: Getting Started Source: https://www.potatoannotator.com/docs/getting-started/configuration-basics > Learn Potato's YAML configuration format: task settings, data file paths, annotation schemes, output formats, and the user keys every project needs. Potato uses YAML configuration files to define annotation tasks. This guide covers the essential configuration options. ![What goes in a Potato config — One config.yaml file groups five sections that define the whole task](/images/blog/config-anatomy.svg "What goes in a Potato config") ## Configuration File Structure A basic Potato configuration has these main sections: ```yaml # Task settings annotation_task_name: "My Annotation Task" port: 8000 # Data configuration data_files: - data.json item_properties: id_key: id text_key: text # Output settings output_annotation_dir: "annotation_output/" export_annotation_format: "jsonl" # Annotation schemes annotation_schemes: - annotation_type: radio name: my_annotation description: "Pick the label that fits this item" labels: - Label 1 - Label 2 # User settings user_config: allow_all_users: true ``` ## Essential Settings ### Task and Server Configuration ```yaml annotation_task_name: "My Task" # Display name for your task port: 8000 # Port to run the server on ``` ### Data Configuration ```yaml data_files: - data.json # Path to your data file(s) - more_data.json # You can specify multiple files item_properties: id_key: id # Field containing unique ID text_key: text # Field containing text to annotate ``` Supported data formats: - JSON (`.json`) - JSON Lines (`.jsonl`) - CSV (`.csv`) - TSV (`.tsv`) ### Output Configuration ```yaml output_annotation_dir: "annotation_output/" # Directory for annotation files export_annotation_format: "jsonl" # Formats: jsonl, csv, tsv, parquet, and the other registered exporters ``` Annotations are always stored as `//user_state.json` while people work. `export_annotation_format` names the format the periodic auto-export writes on top of that, so leave it out when you only want the raw state files. There is no exporter called `json`; a config that asks for one is read as `jsonl` with a warning. ### Annotation Schemes Define one or more annotation schemes: ```yaml annotation_schemes: - annotation_type: radio # Type of annotation name: sentiment # Internal name description: "Select the sentiment" # Instructions labels: # Options for annotators - Positive - Negative - Neutral ``` ## Available Annotation Types | Type | Description | |------|-------------| | `radio` | Single choice selection | | `multiselect` | Multiple choice selection | | `likert` | Rating on a scale | | `text` | Free text input | | `number` | Numeric input | | `span` | Text span highlighting | | `slider` | Continuous range selection | | `multirate` | Rate multiple items | | `select` | Dropdown single selection | | `pairwise` | Pairwise comparison | | `bws` | Best-worst scaling | | `soft_label` | Soft label distribution | | `confidence` | Annotation with confidence | | `constant_sum` | Constant sum allocation | | `range_slider` | Range slider selection | | `semantic_differential` | Semantic differential scale | | `hierarchical_multiselect` | Hierarchical multi-level selection | | `card_sort` | Card sorting | | `rubric_eval` | Rubric-based evaluation | | `extractive_qa` | Extractive question answering | | `error_span` | Error span highlighting | | `triage` | Triage classification | | `coreference` | Coreference annotation | | `span_link` | Span linking | | `ranking` | Ordering a set of items | Potato registers 61 annotation types in total. The table above covers the ones most projects start from; see [Annotation Types](/docs/annotation-types/overview) for the rest. Entity linking is built from `span` plus `span_link` rather than from a type of its own. ## User Configuration ### Allow all users ```yaml user_config: allow_all_users: true ``` ### Restrict to specific users ```yaml user_config: allow_all_users: false users: - user1@example.com - user2@example.com ``` ## Task Directory The `task_dir` setting defines the root directory for relative paths: ```yaml task_dir: ./my-task/ data_files: - data/input.json # Resolves to ./my-task/data/input.json ``` ## Full Example Here's a complete configuration for a sentiment analysis task: ```yaml # config.yaml annotation_task_name: "Sentiment Analysis" port: 8000 task_dir: ./ # Data data_files: - data/tweets.json item_properties: id_key: id text_key: text # Output output_annotation_dir: "annotation_output/" export_annotation_format: "jsonl" # Annotation annotation_schemes: - annotation_type: radio name: sentiment description: "What is the sentiment expressed in this tweet?" labels: - name: Positive key_value: "1" - name: Negative key_value: "2" - name: Neutral key_value: "3" sequential_key_binding: true # Users user_config: allow_all_users: true # Assignment per_annotator_quota: default: 100 num_annotators_per_item: 2 ``` Two annotators per item is the smallest setting that yields an agreement score, so it suits a pilot where you still want to check that the label set is understood. The quota of 100 bounds one person's session; raise it once you know how long an item takes. ## Next Steps - Explore [Data Formats](/docs/core-concepts/data-formats) in detail - Learn about [Annotation Schemes](/docs/core-concepts/annotation-schemes) - Customize the interface with [UI Configuration](/docs/core-concepts/ui-configuration) - Use the [Preview CLI](/docs/tools/preview-cli) to validate your configuration --- # Guides Section: Guides Source: https://www.potatoannotator.com/docs/guides/overview > Tool-agnostic explainers on annotation practice: scheme design, inter-annotator agreement, crowdsourcing, and agent evaluation, each with a worked example. These are explainers rather than reference pages. Each one answers a question about annotation practice that is not specific to any tool, then uses Potato as the worked example. They fall into a few groups. The first cover decisions you make before any data is labelled: what to annotate, how to write guidelines, what shape the data should be in. A larger group covers particular kinds of annotation, from named entities and coreference through audio, video, image segmentation and 3D point clouds. A third covers measurement: agreement, adjudication, sample size, and what to do when annotators disagree. The rest cover evaluating LLM and agent output, and running a study with paid annotators. If you are new to annotation, start with [What Is Data Annotation?](/docs/guides/what-is-data-annotation). If you have annotations already and need to report on their quality, start with [Inter-Annotator Agreement Explained](/docs/guides/inter-annotator-agreement). --- # What Is Data Annotation? Section: Guides Source: https://www.potatoannotator.com/docs/guides/what-is-data-annotation > An introduction to data annotation, the main task types (classification, span labeling, ranking, free text), and how to run an annotation project in Potato. **Data annotation is the process of attaching labels to raw data, text, images, audio, video, or model outputs, so that the data can be measured, compared, or used to train and evaluate machine learning models.** A label might be a sentiment category on a tweet, a highlighted name in a sentence, a 1–5 quality rating on a chatbot reply, or a bounding box around a pedestrian in a photo. Annotation is sometimes called data labeling, tagging, or coding (the term used in the social sciences). See [Data annotation](https://en.wikipedia.org/wiki/Data_annotation) and the related idea of a [labeled training set](https://en.wikipedia.org/wiki/Labeled_data) on Wikipedia. ## What annotation is for Supervised machine learning learns from examples that already carry the right answer. The quality of those answers sets a ceiling on model quality, so careful annotation is often where the biggest gains are. Annotation is also how you *evaluate* a model, because deciding whether a system's output is correct usually takes a person reading it. ## The main types of annotation task Most projects fall into a few families. Each maps to one or more annotation controls in Potato (see [Annotation Schemes](/docs/core-concepts/annotation-schemes)). - **Classification**: pick one or more categories for a whole item, such as marking a review positive, negative, or neutral. See [Text classification](/docs/guides/text-annotation). - **Span labeling**: mark a region *inside* an item, such as a name in a sentence or a region of an audio clip. See [Span Annotation](/docs/guides/span-annotation). - **Rating and scoring**: place an item on a scale, such as a 1–5 quality judgment. See [Rating Scales](/docs/guides/rating-scales). - **Ranking and comparison**: order items or pick the better of two. See [Pairwise and Best–Worst Scaling](/docs/guides/pairwise-and-best-worst). - **Structured annotation**: link spans into relations, build coreference chains, or annotate events. See [Relation and Event Extraction](/docs/guides/relation-and-event-extraction). - **Free-text**: write an explanation, a correction, or a transcription. ## A minimal example A sentiment task in Potato is a few lines of YAML. The `annotation_schemes` block defines the labels an annotator sees: ```yaml annotation_schemes: - annotation_type: radio name: sentiment description: "What is the overall sentiment of this review?" labels: - Positive - Negative - Neutral ``` Those eight lines are the whole interface definition. You supply the data, run `potato start config.yaml`, and annotators label in the browser. ## How a project usually goes 1. **Define the task.** Write down the question and the label set. 2. **Write guidelines.** Give annotators rules and examples. See [Writing Annotation Guidelines](/docs/guides/writing-annotation-guidelines). 3. **Pilot.** Label a small batch, find disagreements, refine the guidelines. 4. **Annotate with overlap.** Have several people label the same items so you can measure agreement. 5. **Measure agreement.** See [Inter-Annotator Agreement](/docs/guides/inter-annotator-agreement). 6. **Adjudicate and export.** Resolve disagreements and export for training or analysis. ## Frequently asked questions The questions below cover how data annotation relates to data labeling and what an annotator does. ### What is the difference between data annotation and data labeling? In most usage there is no difference. Data labeling, tagging, and coding, the term used in the social sciences, are other names for attaching labels to data so that it can be measured, compared, or used to train and evaluate machine learning models. ### What does a data annotator do? A data annotator reads each item, such as a tweet, a photo, or a chatbot reply, and attaches the label the task asks for according to written guidelines. In a research study several annotators label the same items independently so that agreement between them can be measured, and disagreements are resolved before the labels are exported for training or analysis. ## Further reading - [Choosing an Annotation Scheme](/docs/guides/choosing-an-annotation-scheme) - [Quick Start](/docs/getting-started/quick-start), get Potato running in five minutes - [Annotation Schemes reference](/docs/core-concepts/annotation-schemes) - [Sentiment Analysis](/showcase/sentiment-analysis), the three-way sentiment task above as a runnable design with sample data --- # Choosing an Annotation Scheme Section: Guides Source: https://www.potatoannotator.com/docs/guides/choosing-an-annotation-scheme > How to map your research question to the right Potato annotation type, radio, multiselect, span, likert, slider, pairwise, best-worst, multirate, rubric, and more. **The annotation scheme is the shape of the question you ask annotators. Pick it by asking what kind of answer you need: one category, several categories, a region of the item, a position on a scale, or a comparison between items.** Potato supports 61 annotation types; this guide narrows them down. For the full list of options and their settings, see the [Annotation Schemes reference](/docs/core-concepts/annotation-schemes). ## A decision guide | If you need… | Use this type | Example | |---|---|---| | Exactly one category | `radio` | Sentiment: positive / negative / neutral | | Several categories at once | `multiselect` | Topics present in an article | | A category from a long list | `select` (dropdown) | Country, language, ICD code | | A region inside the text/audio | `span` | Named entities, error spans | | A position on a scale | `likert` | Agreement, fluency, quality | | A continuous value | `slider` | Confidence 0–100 | | The better of two items | `pairwise` | Which model reply is better? | | Best and worst of a set | `best_worst_scaling` | Rank translations by fluency | | The same scale across many items | `multirate` | Rate each retrieved document | | Multiple weighted criteria | `rubric_eval` | MT-Bench-style LLM scoring | | A written answer | `text` | Justification, correction | ## Worked example: single vs. multiple labels If categories are mutually exclusive, use `radio` so annotators must pick one: ```yaml annotation_schemes: - annotation_type: radio name: stance description: "What stance does the post take?" labels: [Supports, Opposes, Neutral] sequential_key_binding: true ``` If an item can have several labels at once, use `multiselect` and set limits: ```yaml annotation_schemes: - annotation_type: multiselect name: topics description: "Select every topic the article covers." labels: [Politics, Technology, Health, Sports, Business] ``` `sequential_key_binding: true` lets annotators press number keys instead of clicking, which speeds up large jobs. ## Combine schemes for richer tasks You can stack several schemes on one screen, for example, a category plus a free-text reason. Pair them with conditional logic so the reason only appears when it is needed: ```yaml annotation_schemes: - annotation_type: radio name: quality description: "Is this answer acceptable?" labels: [Good, Bad] - annotation_type: text name: reason description: "If Bad, briefly explain why." label_requirement: required: false ``` ## Further reading - [Text Annotation](/docs/guides/text-annotation) and [Span Annotation](/docs/guides/span-annotation) - [Rating Scales](/docs/guides/rating-scales) and [Pairwise and Best–Worst Scaling](/docs/guides/pairwise-and-best-worst) - [Writing Annotation Guidelines](/docs/guides/writing-annotation-guidelines) --- # Designing Data Formats for Annotation Section: Guides Source: https://www.potatoannotator.com/docs/guides/data-formats-for-annotation > How to structure input data (JSON, JSONL, CSV) for an annotation project, what fields Potato expects, and how to plan for clean export to training pipelines. **Good annotation starts with well-structured input. Each item needs a stable unique identifier and the content to be labeled; everything else is optional context.** Getting this right at the start saves painful re-runs later, because annotations are keyed to your item IDs. Common interchange formats are [JSON](https://en.wikipedia.org/wiki/JSON), [JSON Lines](https://jsonlines.org/) (one object per line, ideal for large datasets), and [CSV](https://en.wikipedia.org/wiki/Comma-separated_values). Potato reads all three. For the full reference see [Data Formats](/docs/core-concepts/data-formats). ## The minimum each item needs - **A unique ID** that never changes. Annotations are stored against this ID, so if you renumber items mid-project you lose the link to existing labels. - **The content to annotate**: a text field, an image URL, an audio path, a time-aligned transcript, or a structured trace. A JSONL file for a text task looks like this: ```json {"id": "rev_001", "text": "The battery lasts all day. Highly recommend."} {"id": "rev_002", "text": "Stopped working after a week."} ``` You tell Potato which keys to use: ```yaml item_properties: id_key: id text_key: text data_files: - "data/reviews.jsonl" ``` ## Carry context, but keep it separate from labels Extra fields, a source URL, a timestamp, a model name, can ride along on each item and be shown to annotators without becoming labels. Keep them clearly named so the export is easy to read later. ## Transcript files count as input too If your content is speech, you do not have to flatten it into a text field first. Potato reads 21 transcript and subtitle formats, including Whisper and WhisperX JSON, Deepgram, AssemblyAI, AWS Transcribe, SRT, WebVTT, TTML, Praat TextGrid, and ELAN EAF. A data file can point straight at the files your ASR tool wrote: ```json {"id": "int_001", "conversation": {"audio": "media/int_001.mp3", "transcript": "media/int_001.srt"}} ``` Detection is based on the file contents rather than the extension, so the same transcript works inlined or as a sidecar. See [Transcript Formats](/docs/annotation-types/transcript-formats) for the full list and the normalized turn model everything gets converted into. ## Plan the export before you label Decide early how labeled data will feed your pipeline. Potato exports to JSON, JSONL, and CSV, and to ML-native formats such as [CoNLL](https://en.wikipedia.org/wiki/CoNLL) for sequence labeling, Hugging Face Datasets, and COCO/YOLO for vision. Choosing the target format up front tells you which fields and ID scheme to use now. See [Exporting Annotations for ML](/docs/guides/exporting-annotations-for-ml). ```yaml output_annotation_dir: "annotation_output/" export_annotation_format: "jsonl" ``` ## Further reading - [Data Formats reference](/docs/core-concepts/data-formats) - [Instance Display](/docs/core-concepts/instance-display), how content is shown - [What Is Data Annotation?](/docs/guides/what-is-data-annotation) --- # Writing Effective Annotation Guidelines Section: Guides Source: https://www.potatoannotator.com/docs/guides/writing-annotation-guidelines > How to write an annotation codebook that produces consistent labels, clear definitions, worked examples, edge-case rules, and a pilot-and-revise loop. **Annotation guidelines (a "codebook") are the rulebook annotators follow. What drives label quality is whether the guidelines make two careful people choose the same label, not how detailed they are.** A short, sharp codebook with good examples beats a long vague one. This connects to [content analysis](https://en.wikipedia.org/wiki/Content_analysis) in the social sciences, where codebook design has been studied for decades, and it is what ultimately determines your [inter-rater reliability](https://en.wikipedia.org/wiki/Inter-rater_reliability). ## What a good codebook contains - **A one-sentence definition per label**, plus a line on what the label is *not*. Negative definitions prevent the most common disagreements. - **Worked examples**, including borderline ones. Real items from your data beat invented ones. - **Tie-breaker rules.** When two labels could apply, say which wins. - **A "when in doubt" default** and an escape hatch (a "can't tell" option) so annotators don't guess silently. ## Put the rules where annotators see them Potato can show instructions and per-label tooltips right next to the controls, so the guideline lives at the point of decision rather than in a separate document: ```yaml annotation_schemes: - annotation_type: radio name: toxicity description: "Is this comment toxic? Toxic = rude, disrespectful, or likely to make someone leave a conversation." labels: [Toxic, Not toxic, Can't tell] ``` The "Can't tell" option matters: it separates genuine ambiguity from forced guesses, which keeps your agreement metrics honest. ## Pilot, then revise Guidelines are never right on the first try. Run a small pilot, look at every item where annotators disagreed, and decide whether the *guideline* was unclear or the *item* was genuinely ambiguous. Fix the guideline and re-pilot. Two or three rounds usually stabilizes the label set. Potato's [gold standards and attention checks](/docs/guides/gold-standards-and-attention-checks) let you encode the settled cases as checks so future annotators stay calibrated. ## Further reading - [Inter-Annotator Agreement](/docs/guides/inter-annotator-agreement) - [Gold Standards and Attention Checks](/docs/guides/gold-standards-and-attention-checks) - [Adjudication and Disagreement](/docs/guides/adjudication-and-disagreement) --- # Build or Buy an Annotation Tool: Spreadsheets, Custom Apps and Platforms Section: Guides Source: https://www.potatoannotator.com/docs/guides/build-or-buy-annotation-tool > When a spreadsheet is enough for labelling, when a home-built app stops paying off, and how to pick an open-source or hosted annotation platform for NLP or GenAI evaluation. **Use a spreadsheet when one person is labelling a few hundred items with one label each. Once several people label the same items, you need them to work independently and you need an agreement number, and from then on an existing annotation tool costs less than building one.** A hosted platform is worth paying for when you also want it to supply annotators, train models, or satisfy a procurement requirement. An annotation tool shows each item to an annotator, records the [label](https://en.wikipedia.org/wiki/Labeled_data), and keeps track of who labelled what. [Inter-rater reliability](https://en.wikipedia.org/wiki/Inter-rater_reliability), usually reported as Cohen's kappa or Krippendorff's alpha, measures whether independent annotators agree, and it is what reviewers ask for when a paper reports human labels. See [Inter-Annotator Agreement Explained](/docs/guides/inter-annotator-agreement). ## When a spreadsheet is enough A spreadsheet is the right tool when all of these hold: - one annotator, usually you - one label per item, from a short fixed list - a few hundred items - no agreement to report Put the labels in a dropdown with data validation, so they cannot drift into `Positive`, `positive` and `pos` across a long afternoon. ## Where spreadsheets break - **Annotators see each other's labels.** In a shared sheet the second annotator can read the first annotator's column, and agreement measured that way is not agreement between independent judgments. - **Assignment is manual.** Who labels which rows, and how many labels each row needs, is tracked by hand. - **A cell holds one value for the whole row.** Highlighting a phrase, drawing a box, or ranking four model responses does not fit in a cell. - **Nothing records the process.** There is no record of how long an annotator spent on an item, or which version of the instructions they had seen. ## Building your own labelling app Coding agents have made this common. Ask Claude Code, Codex or Cursor for a Streamlit or Flask page that shows one item at a time and saves each answer to a CSV, and you have a working tool in minutes. For one person, that is often enough. The work arrives with the second annotator and the second study. A multi-annotator study needs logins, assignment of items to people, instructions and a training round, attention checks, agreement, a completion code for [Prolific or MTurk](/docs/guides/crowdsourcing-prolific-mturk), and export in the format the next script expects. Each of those is code someone has to write, test and debug, in a tool that serves one project. If you were going to ask a coding agent to build the tool, ask it to design the study instead. [potato-skill](/docs/getting-started/quick-start-with-a-coding-agent) has the agent work through the design with you and build the result as a Potato task, which already has every item on that list. The task is a folder with a config and data, so a colleague can run it too. ## Open-source or hosted Self-hosted open-source tools, including Potato, Label Studio's community edition, Doccano and CVAT, are free to use, keep the data on machines you control, and leave you to run the server. Hosted platforms run the server for you, and some also supply a workforce, model-assisted labelling or model training, usually for a per-seat or per-item price. Four questions narrow it down: 1. **Where must the data live?** If it cannot leave your institution, self-host. 2. **Who are the annotators?** Your own team, crowdworkers you recruit, or a vendor's workforce. Only the last needs a platform that supplies people. 3. **Do you need agreement statistics in the tool?** Many tools leave this to you, or reserve it for a paid tier. 4. **Can you leave?** Check which formats the tool imports and exports before your data is in it. [Annotation Tools Compared](/docs/guides/annotation-tools-compared) goes through 14 tools on these points. ## Evaluating generative AI output Human evaluation of LLM and agent output is the same decision with extra requirements. Raters should not know which model wrote a response, response order should be randomised, and you usually want agreement between human raters and, if you use one, between the humans and an LLM judge. A spreadsheet handles none of that well, and a home-built app has to implement each piece. Potato covers [pairwise comparison](/docs/guides/pairwise-model-comparison), [rubric scoring](/docs/guides/rubric-based-llm-evaluation), [agent trajectories](/docs/guides/evaluating-ai-agents) and [LLM-judge calibration](/docs/agent-evaluation/judge-calibration). ## A minimal Potato task One label per row becomes this config: ```yaml annotation_task_name: "Sentiment Analysis" task_dir: "." data_files: - "data/data.json" item_properties: id_key: id text_key: text output_annotation_dir: "annotation_output/" annotation_schemes: - annotation_type: radio name: sentiment description: "What is the sentiment of this text?" labels: - Positive - Negative - Neutral ``` Start it with `potato start config.yaml -p 8000` and send annotators the URL. [Quick Start](/docs/getting-started/quick-start) walks through it. ## Side by side | | Spreadsheet | Home-built app | Potato | Hosted platform | |---|---|---|---|---| | Several annotators per item | By hand | You write it | Built in | Built in | | Annotators work independently | No, in a shared sheet | You write it | Yes | Yes | | Agreement statistics | You compute them | You write it | Built in | Varies by vendor | | Spans, boxes, rankings | No | You write each one | Built in | Built in | | Data stays on your machines | Depends on where the sheet lives | Yes | Yes | No | | Price | Free | Your time | Free | Subscription or per item | ## Frequently asked questions The questions below cover spreadsheets, building or buying a platform for GenAI evaluation, choosing between platforms, and self-hosting. ### Should I label data in a spreadsheet or an annotation tool? Use a spreadsheet when one person labels a few hundred items with one label each and there is no agreement to report. Once several people label the same items, an annotation tool costs less. A shared sheet lets the second annotator read the first annotator's column, assignment is tracked by hand, and a cell cannot hold a highlighted phrase, a box, or a ranking. ### Should I build or buy a GenAI annotation platform? Use an existing platform rather than build one. Human evaluation of LLM and agent output needs raters who do not know which model wrote a response, randomized response order, and agreement between the raters and against any LLM judge, and a home-built app has to implement each of those. A hosted platform is worth paying for when it also supplies annotators or trains models. Otherwise a self-hosted open-source tool such as Potato covers pairwise comparison, rubric scoring, agent trajectories, and LLM-judge calibration. ### How do I choose a GenAI annotation platform? Score the candidates on whether they can run a blind evaluation, report agreement, and export what your analysis needs, then treat everything else as preference. A platform that presents responses in a fixed order, or shows the rater which model produced each one, will give you ratings that measure position and brand rather than quality. A platform that reports no agreement statistic cannot tell you whether the ratings mean anything, and one that reports agreement between human raters but not between a rater and an LLM judge cannot tell you whether the judge can be trusted on the rest of the corpus. Check the export format against the analysis you plan before you commit, because a platform that exports only aggregate scores makes per-rater modeling impossible afterwards. ### Which annotation tools can be self-hosted? Potato, Label Studio's community edition, Doccano, and CVAT are open source and run on your own server. Self-hosting keeps the data on machines you control and leaves you to run the server, while hosted platforms run it for you, usually for a per-seat or per-item price. ## Further reading - [Quick Start with a Coding Agent](/docs/getting-started/quick-start-with-a-coding-agent) - [Annotation Tools Compared](/docs/guides/annotation-tools-compared) - [How Many Annotators Do You Need?](/docs/guides/how-many-annotators) - [Choosing an Open-Source Annotation Tool in 2026](/blog/choosing-an-annotation-tool-2026) --- # Text Annotation Section: Guides Source: https://www.potatoannotator.com/docs/guides/text-annotation > A complete guide to text annotation, classification, multi-label tagging, rating, and free-text, and how to build each kind of text task in Potato with copy-paste config. **Text annotation means labeling written language: sorting documents into categories, tagging the topics in an article, rating a passage for quality, or writing a correction. It is the most common annotation task in [natural language processing](https://en.wikipedia.org/wiki/Natural_language_processing), and it is what Potato was first built for.** This guide covers the whole-document text tasks; for marking regions *inside* text, see [Span Annotation](/docs/guides/span-annotation). ## The text tasks at a glance - **Document classification**: one label for the whole text ([text classification](https://en.wikipedia.org/wiki/Document_classification)). - **Multi-label tagging**: several labels at once, such as topics or content warnings. - **Rating and scoring**: a position on a scale, such as quality or [sentiment](https://en.wikipedia.org/wiki/Sentiment_analysis) intensity. - **Free-text**: a written answer, paraphrase, or correction. ## Classification: one label per document The workhorse of text annotation. Use `radio` when the categories are mutually exclusive: ```yaml annotation_schemes: - annotation_type: radio name: sentiment description: "What is the overall sentiment of this review?" labels: [Positive, Negative, Neutral] sequential_key_binding: true ``` `sequential_key_binding` maps the labels to keys `1`, `2`, `3`, so annotators keep their hands on the keyboard. On a job of thousands of items this is a large speed-up. See the live [sentiment analysis](/showcase/sentiment-analysis) design for a working example. ## Multi-label: several tags at once When more than one label can apply, use `multiselect`. Bound the selection count to match your guidelines: ```yaml annotation_schemes: - annotation_type: multiselect name: content_warnings description: "Select every content warning that applies." labels: [Violence, Profanity, Sexual content, Self-harm, None] ``` Content moderation is a classic multi-label text task; the [toxicity detection](/showcase/toxicity-detection) design combines a category with a highlighted span. ## Rating text on a scale To capture *degree* rather than category, use a [Likert scale](https://en.wikipedia.org/wiki/Likert_scale): ```yaml annotation_schemes: - annotation_type: likert name: helpfulness description: "How helpful is this answer?" size: 5 min_label: "Not helpful" max_label: "Very helpful" ``` See [Rating Scales](/docs/guides/rating-scales) for scale-design pitfalls such as acquiescence bias and how many points to use. ## Free-text and corrections Sometimes the most useful label is a sentence the annotator writes, a justification, a rewrite, or a transcription. Combine it with a category and show it only when relevant: ```yaml annotation_schemes: - annotation_type: radio name: factuality description: "Is the claim supported by the source?" labels: [Supported, Contradicted, Not enough info] - annotation_type: text name: evidence description: "Quote the sentence that supports your choice." label_requirement: required: false ``` ## Getting consistent text labels Text is ambiguous, so consistency comes from the surrounding process, not the interface: 1. Write tight [guidelines](/docs/guides/writing-annotation-guidelines) with a "can't tell" option. 2. Have multiple annotators overlap on the same items. 3. Track [inter-annotator agreement](/docs/guides/inter-annotator-agreement) and adjudicate disagreements. 4. Speed up large jobs with [LLM pre-annotation](/docs/guides/llm-pre-annotation) and verify the suggestions by hand. ## Further reading - [Span Annotation](/docs/guides/span-annotation), marking regions inside text - [Choosing an Annotation Scheme](/docs/guides/choosing-an-annotation-scheme) - [Annotation Schemes reference](/docs/core-concepts/annotation-schemes) - [Text Annotation Tools Compared](/docs/guides/text-annotation-tools-compared) --- # Span Annotation Section: Guides Source: https://www.potatoannotator.com/docs/guides/span-annotation > A complete guide to span annotation, highlighting regions of text, overlapping and nested spans, label colors, BIO/IOB tagging, and building span tasks in Potato. **Span annotation means marking a region *inside* an item rather than labeling the whole item. The annotator highlights a stretch of text (or a segment of audio) and assigns it a label.** It is the foundation for named entity recognition, error marking, extractive question answering, and audio event detection, they are all span tasks with different label sets. A span is a labeled sub-sequence: a start position, an end position, and a category. In machine learning this is usually framed as [sequence labeling](https://en.wikipedia.org/wiki/Sequence_labeling), where every token gets a tag. ## A basic span task Define the labels and let annotators highlight. Colors and tooltips make the interface fast and self-explanatory: ```yaml annotation_schemes: - annotation_type: span name: entities description: "Highlight each named entity and choose its type." labels: [PERSON, ORGANIZATION, LOCATION, DATE, MISC] sequential_key_binding: true allow_discontinuous: false ``` The [named entity recognition](/showcase/named-entity-recognition) design is exactly this task, ready to run. ## Overlapping and nested spans By default a character belongs to at most one span. Some tasks need more: - **Overlapping spans**: two annotations cover some of the same text, e.g. a sentiment span over an entity span. - **Nested spans**: one span sits inside another, e.g. "[University of [Michigan]]" where the location is nested in the organization. Set `allow_overlapping: true` when your guidelines call for it. Decide this early, because it changes how annotators think about boundaries. ## BIO/IOB tagging, what your export looks like Span annotations are usually exported for training as token tags in the **BIO** scheme (also called [IOB](https://en.wikipedia.org/wiki/Inside%E2%80%93outside%E2%80%93beginning_(tagging))): `B-` marks the first token of an entity, `I-` marks tokens inside it, and `O` marks tokens outside any entity. ``` Barack B-PERSON Obama I-PERSON visited O Paris B-LOCATION ``` Potato exports spans to [CoNLL](https://en.wikipedia.org/wiki/CoNLL), which uses this tagging directly and which spaCy's `spacy convert` reads. See [Exporting Annotations for ML](/docs/guides/exporting-annotations-for-ml). ## Getting boundaries right The hardest part of span work is agreeing where a span starts and ends. A few rules that help: - Decide whether to include surrounding punctuation, titles ("Dr."), and trailing possessives, and write it down. - Measure agreement at the span level, not just the document level, so boundary disagreements show up. See [Inter-Annotator Agreement](/docs/guides/inter-annotator-agreement). - Use tooltips to keep the boundary rule in front of the annotator. ## Span tasks beyond NER The same mechanism powers many tasks: - **Error spans**: mark mistakes in a translation or a model output, MQM-style. See [Detecting Hallucinations](/docs/guides/detecting-hallucinations). - **Extractive QA**: highlight the answer to a question in a passage. - **Audio event detection**: mark *when* a sound occurs on a waveform. See [Audio Annotation](/docs/guides/audio-annotation). - **Relations and coreference**: link spans together. See [Relation and Event Extraction](/docs/guides/relation-and-event-extraction) and [Coreference Resolution](/docs/guides/coreference-resolution). ## Further reading - [Named Entity Recognition](/docs/guides/named-entity-recognition) - [Text Annotation](/docs/guides/text-annotation) - [Annotation Schemes reference](/docs/core-concepts/annotation-schemes) --- # Named Entity Recognition Section: Guides Source: https://www.potatoannotator.com/docs/guides/named-entity-recognition > What named entity recognition (NER) is, common label sets, and how to build an NER annotation task in Potato with colored span labels and tooltips. **Named entity recognition (NER) is the task of finding and classifying named things in text, people, organizations, locations, dates, and more. It is a [span annotation](/docs/guides/span-annotation) task with an entity-typed label set.** NER is a building block for search, knowledge graphs, redaction, and information extraction. See [Named-entity recognition](https://en.wikipedia.org/wiki/Named-entity_recognition) for background. ## Choosing a label set Start from a standard scheme and trim it to your domain: - **CoNLL-2003**: `PER`, `ORG`, `LOC`, `MISC`. A good minimal default. - **OntoNotes**: 18 types including dates, money, and percentages, for richer needs. - **Domain-specific**: biomedical (genes, diseases), legal (statutes, parties), or finance. Fewer, well-defined types give higher agreement. Add types only when a real downstream use needs them. ## Building the task in Potato ```yaml annotation_schemes: - annotation_type: span name: entities description: "Highlight each named entity and select its type." labels: [PERSON, ORGANIZATION, LOCATION, DATE, MISC] allow_discontinuous: false sequential_key_binding: true ``` The [named entity recognition showcase](/showcase/named-entity-recognition) runs this configuration with sample data. For benchmark and domain variants, see [CoNLL-2003 with a triage step](/showcase/conll2003-ner-triage) and [clinical NER on i2b2 notes](/showcase/clinical-ner-i2b2). ## Boundary rules that prevent disagreement Most NER disagreement is about *where* an entity starts and ends, not *what* it is. Decide and document: - Do titles count? ("**Dr.** Jane Smith" vs. "Dr. **Jane Smith**".) - Do you include "the" in "**the** United Nations"? - How do you tag nested entities like "Bank of **England**"? If you need them, set `allow_overlapping: true`. ## From labels to a model Export to [CoNLL](https://en.wikipedia.org/wiki/CoNLL), which represents entities with [BIO/IOB tags](https://en.wikipedia.org/wiki/Inside%E2%80%93outside%E2%80%93beginning_(tagging)); spaCy's `spacy convert` turns it into training data. See [Exporting Annotations for ML](/docs/guides/exporting-annotations-for-ml). ## Further reading - [Span Annotation](/docs/guides/span-annotation) - [Entity Linking](/docs/guides/entity-linking), connecting entities to a knowledge base - [Relation and Event Extraction](/docs/guides/relation-and-event-extraction) --- # Coreference Resolution Section: Guides Source: https://www.potatoannotator.com/docs/guides/coreference-resolution > What coreference annotation is, how to group mentions into entity chains, and how to set up a coreference task in Potato. **Coreference resolution is the task of grouping all the mentions in a text that refer to the same thing. "Marie Curie … she … the physicist" is one chain pointing at one person.** It turns scattered mentions into entities, which is essential for summarization, question answering, and knowledge extraction. See [Coreference](https://en.wikipedia.org/wiki/Coreference) for background. ## What annotators do 1. Mark each mention (a name, a pronoun, or a noun phrase) as a span. 2. Group mentions that refer to the same entity into a **chain**. 3. Repeat for every distinct entity in the passage. The output is a set of chains, each a list of spans that co-refer. Chains can cross sentence boundaries, which is what makes the task harder than plain [span annotation](/docs/guides/span-annotation). ## Setting it up in Potato Potato has a coreference annotation type that lets annotators mark mentions and link them into chains. The [coreference showcase](/showcase/coreference) is a ready-to-run example. For corpus-scale versions, see [OntoNotes coreference](/showcase/ontonotes-coreference-resolution), the multilingual [CorefUD](/showcase/corefud-multilingual-coreference), and [MAVEN-ERE](/showcase/maven-ere-event-coreference), which links mentions of the same *event* rather than the same entity. ```yaml annotation_schemes: - annotation_type: span name: mentions description: "Mark every mention (names, pronouns, noun phrases), then group mentions that refer to the same entity into a chain." labels: [Entity] allow_discontinuous: true ``` Allow overlapping spans, because mentions frequently nest ("[[his] mother]"). ## Common pitfalls - **Singletons.** Decide whether to mark entities mentioned only once. It affects your counts and metrics. - **Generic vs. specific.** "Doctors recommend rest", is "doctors" an entity to track? Write a rule. - **Split antecedents.** "Alice and Bob … they" refers to both; decide how to represent it. Because chains are structured, measure agreement carefully, see [Inter-Annotator Agreement](/docs/guides/inter-annotator-agreement) and adjudicate with care. ## Further reading - [Span Annotation](/docs/guides/span-annotation) - [Relation and Event Extraction](/docs/guides/relation-and-event-extraction) - [Entity Linking](/docs/guides/entity-linking) --- # Relation and Event Extraction Section: Guides Source: https://www.potatoannotator.com/docs/guides/relation-and-event-extraction > How to annotate relations between entities and structured events with triggers and arguments, using span linking and event annotation in Potato. **Relation extraction connects two entities with a typed link ("Marie Curie, *won*, Nobel Prize"). Event extraction goes further: it marks an event trigger and the roles around it (who did what, to whom, when).** Both turn flat text into structured knowledge. See [Relationship extraction](https://en.wikipedia.org/wiki/Relationship_extraction) and [Event extraction](https://en.wikipedia.org/wiki/Event_extraction) for background. ## Relations: linking spans A relation annotation has three parts: a head span, a tail span, and a relation type. In Potato, annotators first mark the spans, then draw a directed, typed link between them (span linking). ```yaml annotation_schemes: - annotation_type: span name: entities description: "Mark the entities involved." labels: [PERSON, ORGANIZATION, AWARD] - annotation_type: span_link name: relations description: "Draw a link between two entities and choose the relation." span_schema: entities link_types: - name: WON directed: true allowed_source_labels: [PERSON] allowed_target_labels: [AWARD] - name: AFFILIATED_WITH directed: true allowed_source_labels: [PERSON] allowed_target_labels: [ORGANIZATION] ``` `span_schema` names the span scheme whose spans can be linked, and each entry in `link_types` constrains which labels may sit at each end, so an annotator cannot draw a relation the schema does not allow. Relations are directional, so decide and document the direction of each type (is it "founded_by" pointing from company to person, or the reverse?). ## Events: triggers and arguments An event is anchored by a **trigger** word (often a verb like "acquired") and has **arguments** filling typed **roles** (Buyer, Seller, Price, Date). Potato's event annotation type lets annotators mark the trigger and attach role-labeled spans. The [event annotation showcase](/showcase/event-annotation) demonstrates a working trigger-and-argument setup. ## Why these tasks are hard - The label space is large, so agreement drops. Keep relation and role types few and well-defined. - Annotators must first get the spans right; errors compound. Consider annotating entities in a first pass and relations in a second. - Structured output makes agreement metrics non-trivial, measure at the level you care about (correct trigger, correct argument, correct full event). ## Further reading - [Span Annotation](/docs/guides/span-annotation) - [Coreference Resolution](/docs/guides/coreference-resolution) - [Entity Linking](/docs/guides/entity-linking) --- # Entity Linking Section: Guides Source: https://www.potatoannotator.com/docs/guides/entity-linking > How to annotate entity linking, connecting mentions in text to entries in a knowledge base like Wikidata, and set up a linking task in Potato. **Entity linking connects a mention in text to a specific entry in a knowledge base, resolving "Paris" to the city in France rather than the person, or to the city in Texas.** Where [named entity recognition](/docs/guides/named-entity-recognition) finds *that* something is an entity, entity linking decides *which* real-world entity it is. See [Entity linking](https://en.wikipedia.org/wiki/Entity_linking) and the common targets [Wikidata](https://en.wikipedia.org/wiki/Wikidata) and [Wikipedia](https://en.wikipedia.org/wiki/Wikipedia) for background. This is closely related to [word-sense disambiguation](https://en.wikipedia.org/wiki/Word-sense_disambiguation). ## What annotators do 1. A mention span is identified (often pre-marked from an NER pass). 2. The annotator searches the knowledge base and selects the matching entry. 3. If no entry fits, they mark it **NIL** (not in the knowledge base). The NIL case is essential, without it, annotators force-fit mentions to wrong entries and corrupt the data. ## Setting it up in Potato Potato supports entity linking with a typeahead search against a knowledge base (Wikidata, UMLS, or a custom list), so annotators pick from real candidates rather than typing IDs. The [entity linking showcase](/showcase/entity-linking) is a working example. ```yaml annotation_schemes: - annotation_type: span name: mentions description: "Mark the mention to link." labels: [Entity] - annotation_type: text name: kb_id description: "Search the knowledge base and enter the matching ID, or write NIL if none fits." ``` ## Quality considerations - **Candidate quality.** A good typeahead with descriptions reduces wrong picks far more than longer guidelines. - **Ambiguity defaults.** Tell annotators what to do when two entries seem equally valid. - **Granularity.** Link to the company or the subsidiary? The film or the franchise? Decide once. ## Further reading - [Named Entity Recognition](/docs/guides/named-entity-recognition) - [Coreference Resolution](/docs/guides/coreference-resolution) - [Span Annotation](/docs/guides/span-annotation) --- # How to Annotate Threaded Conversations Section: Guides Source: https://www.potatoannotator.com/docs/guides/annotating-threaded-conversations > Render reply structure from reply_to, annotate whole threads and individual comments at once, link spans across comments, and import a ConvoKit corpus both ways. **A forum thread flattened into a transcript loses the thing most conversational research is about: who was replying to whom.** Rendering reply structure is what makes a thread annotatable as a thread rather than as a list. ## Render the structure ```yaml instance_display: fields: - key: conversation type: dialogue label: "Thread" display_options: indent_replies: true ``` Nesting derives from each turn's `reply_to`. Turn identity is read from `turn_id`, `step_id` or plain `id`, so forum exports, chat logs and mailing lists work without reshaping the data. **Use stable ids.** Falling back to position means a re-fetched or re-ordered thread re-points every existing annotation, which is the kind of corruption that is invisible until someone checks. ## One field, several questions at once The useful property is that a single conversation field can carry, simultaneously: - **Whole-thread schemes** — did this thread go off the rails? - **Per-comment radio, likert, select or text** — rate each comment - **Spans inside comments** — mark the sentence that did it - **`span_link` between spans in *different* comments** — link a claim to the reply that rebuts it That last one is what most conversational annotation actually needs and what flat renderings cannot express. A rebuttal is a relation between two positions in two different turns. For branching conversations, `conversation_tree` gains per-node widgets keyed by node id, so a branching tree and a flat thread can refer to the same messages. ## ConvoKit, both directions [ConvoKit](https://convokit.cornell.edu/) (Cornell) is the standard corpus format for conversational research in CS and computational linguistics. ```bash potato convokit conversations-gone-awry-corpus # by name potato convokit ./my-corpus/ # by directory potato convokit corpus.zip # by zip ``` Import at conversation or utterance granularity. Export back with `--format convokit`, either as `info..jsonl` overlays that drop into an existing corpus, or as a full dump. Two properties worth knowing: - **Each turn carries the real utterance id**, so per-comment annotations round-trip by direct lookup rather than by position. - **No `convokit` dependency.** The format is read and written with the standard library, so importing a corpus does not pull in a research stack. Every format variant in the wild is read, down to the pre-rename `user`/`root`/`users.json` layout. ## Measuring agreement Thread-level labels use ordinary categorical agreement. Per-comment labels are the interesting case: agreement should be computed **per comment**, not pooled across the thread, or a long thread of easy comments will swamp the disagreement on the one comment that mattered. Spans inside comments use [span agreement](/docs/guides/agreement-for-spans), with the caveat that offsets must be computed against the same normalisation the server applies. Potato had a bug here where the server collapsed whitespace and the client did not, drifting every offset in a dialogue. ## Further reading - [Dialogue annotation](/docs/annotation-types/dialogue-annotation) - [Conversation trees](/docs/annotation-types/conversation-trees) - [Span linking](/docs/annotation-types/span-linking) - [ConvoKit integration](https://github.com/davidjurgens/potato/blob/master/docs/integrations/convokit.md) --- # Rating Scales and Likert Design Section: Guides Source: https://www.potatoannotator.com/docs/guides/rating-scales > How to design rating scales for annotation, Likert vs. sliders, how many points to use, avoiding acquiescence bias, and building rating tasks in Potato. **A rating scale captures *degree*, how positive, how fluent, how helpful, rather than a category. The two common forms are the discrete [Likert scale](https://en.wikipedia.org/wiki/Likert_scale) (e.g. 1–5) and the continuous [slider](https://en.wikipedia.org/wiki/Visual_analogue_scale).** Small design choices in a scale change your data more than people expect. ## Likert: discrete points Use a Likert scale when you want comparable, easy-to-aggregate ratings: ```yaml annotation_schemes: - annotation_type: likert name: fluency description: "How fluent is this translation?" size: 5 min_label: "Not fluent at all" max_label: "Perfectly fluent" ``` Design decisions that matter: - **How many points?** Five is a safe default. Seven gives more resolution if annotators can use it. An even number removes the neutral midpoint and forces a lean, useful when "neutral" is a cop-out, risky when neutrality is real. - **Label the ends, and ideally every point.** Labeled points are interpreted more consistently than bare numbers. - **Keep the direction consistent** across all your scales so annotators don't flip them by habit. ## Sliders: continuous values Use a `slider` when the underlying quantity really is continuous, such as a confidence percentage or an emotion intensity: ```yaml annotation_schemes: - annotation_type: slider name: confidence description: "How confident are you in your label?" min_value: 0 max_value: 100 starting_value: 50 step: 1 ``` Continuous scales give resolution but lower agreement, because people don't share a fine-grained sense of "67 vs. 72". Bin the output if you need agreement. ## Biases to design around - **Acquiescence bias**: a tendency to agree. Mix in reverse-worded items so agreement isn't the default. See [acquiescence bias](https://en.wikipedia.org/wiki/Acquiescence_bias). - **Central tendency**: clustering on the middle. Clear endpoint labels and, where appropriate, an even number of points push against it. - **Anchoring**: the first few items set a reference. A short calibration set at the start helps. ## Beyond a single scale - Rate many items on the same scale at once with `multirate` (e.g. each retrieved document). See [RAG Evaluation](/docs/guides/rag-evaluation). - Score several criteria at once with `rubric_eval`, which shares one scale across every criterion. See [Rubric-Based LLM Evaluation](/docs/guides/rubric-based-llm-evaluation). - Audio quality ratings such as [MOS](https://en.wikipedia.org/wiki/Mean_opinion_score) use the same Likert mechanism, see [Audio Annotation](/docs/guides/audio-annotation). ## Further reading - [Choosing an Annotation Scheme](/docs/guides/choosing-an-annotation-scheme) - [Pairwise and Best–Worst Scaling](/docs/guides/pairwise-and-best-worst), when comparisons beat ratings - [Inter-Annotator Agreement](/docs/guides/inter-annotator-agreement) --- # Pairwise and Best–Worst Scaling Section: Guides Source: https://www.potatoannotator.com/docs/guides/pairwise-and-best-worst > When to use comparative judgments instead of ratings, pairwise comparison and best-worst scaling (MaxDiff), and how to set them up in Potato. **People are bad at giving stable absolute scores but good at comparing. Comparative annotation exploits this: instead of "rate this 1–5", you ask "which is better, A or B?".** The two main forms are pairwise comparison and best–worst scaling, and both are standard formats for collecting preference data. See [Pairwise comparison](https://en.wikipedia.org/wiki/Pairwise_comparison) and [MaxDiff](https://en.wikipedia.org/wiki/MaxDiff) for background. ## Pairwise comparison Show two items and ask which wins. It is simple, high-agreement, and the format used to collect human preference data for [reinforcement learning from human feedback](https://en.wikipedia.org/wiki/Reinforcement_learning_from_human_feedback). ```yaml annotation_schemes: - annotation_type: pairwise name: preference description: "Which response better answers the question?" mode: binary allow_tie: true sequential_key_binding: true ``` Allowing ties keeps annotators from inventing a difference where none exists. To capture *how much* better, switch `mode` to a scale (e.g. "A much better … B much better"). The [pairwise preference showcase](/showcase/pairwise-preference) is a working example. Many pairwise judgments can be turned into a single ranking with a model such as the [Elo rating system](https://en.wikipedia.org/wiki/Elo_rating_system) or the [Bradley–Terry model](https://en.wikipedia.org/wiki/Bradley%E2%80%93Terry_model). ## Best–worst scaling (MaxDiff) Show a small set (often four items) and ask for the **best** and the **worst**. Each judgment is more informative than a single pairwise vote, because it fixes both ends of the set at once. ```yaml annotation_schemes: - annotation_type: bws name: fluency description: "Pick the most and least fluent translation." tuple_size: 4 best_description: "Most fluent" worst_description: "Least fluent" ``` Best–worst scaling produces reliable interval-like scores from simple choices and is widely used to build calibrated rankings from many annotators. ## When to prefer comparisons over ratings - Your construct is hard to anchor absolutely (humor, helpfulness, aesthetic quality). - You need high agreement and your Likert scale is noisy. - You are building preference data to train or align a model. The cost is that you get *relative* information; you may need a model (Elo, Bradley–Terry) to recover absolute scores. ## Further reading - [Rating Scales](/docs/guides/rating-scales), the absolute-scoring alternative - [Pairwise Model Comparison](/docs/guides/pairwise-model-comparison), comparing AI outputs - [RLHF Preference Data](/docs/guides/rlhf-preference-data) --- # Audio Annotation Section: Guides Source: https://www.potatoannotator.com/docs/guides/audio-annotation > How to set up audio annotation in Potato, from clip classification and tagging to waveform event spans, transcription, MOS ratings, emotion, and diarization. **Audio annotation covers everything from labeling a whole clip ("is this speech or music?") to marking the exact moment a sound occurs on the waveform.** Potato displays an interactive waveform with playback and time markers, so the same tool handles classification, tagging, time-aligned event detection, transcription, quality ratings, and speaker work. For the feature reference see [Audio Annotation](/docs/features/audio-annotation). This guide maps each common audio task to a Potato setup and a runnable showcase design. ## Clip-level classification Label the whole clip with one category. Clip-level tasks include [acoustic scene classification](/showcase/acoustic-scene-classification), [environmental sound classification](/showcase/environmental-sound-classification), [keyword spotting](/showcase/keyword-spotting), and [respiratory sound classification](/showcase/respiratory-sound-classification). ```yaml annotation_schemes: - annotation_type: radio name: scene description: "What environment was this recorded in?" labels: [Street, Park, Office, Home, Vehicle] ``` ## Multi-label tagging When several sounds or tags apply at once, as in [music tagging](/showcase/music-tagging) and [AudioSet-style event classification](/showcase/audioset-event-classification), use `multiselect`. ```yaml annotation_schemes: - annotation_type: multiselect name: tags description: "Select every instrument you can hear." labels: [Guitar, Drums, Piano, Vocals, Bass, Synth] ``` ## Sound event detection, spans on the waveform To mark *when* a sound starts and ends, use a span over the audio timeline. The task is [sound event detection](/showcase/sound-event-detection), the audio version of [span annotation](/docs/guides/span-annotation). ```yaml annotation_schemes: - annotation_type: span name: events description: "Mark the start and end of each sound event and label it." labels: [Speech, Music, Dog bark, Siren, Silence] ``` ## Transcription from scratch For [audio transcription](/showcase/audio-transcription), pair playback with a free-text field. Annotators can scrub the waveform while they type. ```yaml annotation_schemes: - annotation_type: text name: transcript description: "Type what is said in this clip." ``` ## Annotating a transcript you already have Transcribing from scratch is the expensive case. Most projects start with a transcript that already exists, from Whisper, a cloud ASR API, or captions downloaded alongside a video. Potato reads 21 transcript and subtitle formats and renders their turns as speaker bubbles synced to the audio, so annotators label the segments rather than re-creating them. ```yaml instance_display: fields: - key: conversation type: audio_dialogue label: "Transcript" span_target: true display_options: show_timestamps: true allow_speaker_assignment: auto ``` Which one you want depends on what you are measuring. Transcribe from scratch when the transcript is the deliverable, or when the audio is bad enough that ASR output would anchor annotators to the wrong words. Start from an existing transcript when the content is what you care about, or when you are checking ASR quality and want the errors visible. See [Annotating Whisper Transcripts](/docs/guides/annotating-whisper-transcripts) and [Annotating YouTube Subtitles](/docs/guides/annotating-youtube-subtitles). ## Quality ratings: MOS and intelligibility Subjective audio quality is measured with a [mean opinion score](https://en.wikipedia.org/wiki/Mean_opinion_score), a 1–5 Likert rating averaged across listeners. This covers [speech quality (MOS)](/showcase/speech-quality-mos) and [speech intelligibility](/showcase/speech-intelligibility-rating). ```yaml annotation_schemes: - annotation_type: likert name: mos description: "Rate the overall quality of this audio." size: 5 min_label: "Bad" max_label: "Excellent" ``` See [Rating Scales](/docs/guides/rating-scales) for scale-design tips. ## Emotion and sentiment [Speech emotion recognition](/showcase/speech-emotion-recognition) and [audio sentiment analysis](/showcase/audio-sentiment-analysis) combine a category (the emotion) with dimensional ratings (arousal, valence) using `radio` plus `slider` or `likert`. ## Speaker diarization [Speaker diarization](/showcase/speaker-diarization) answers "who spoke when". Annotators mark time spans and link each to a speaker, which is span annotation plus a linking step. If you already ran diarization upstream, with WhisperX or a cloud API, Potato reads the speaker labels directly and you only need annotators to correct the mistakes. From version 2.9, `potato transcripts --transcribe --diarize` can also transcribe and diarize on your own machine, with no PyTorch and no Hugging Face token. See [Transcribing locally](/docs/annotation-types/transcript-formats#transcribing-locally). Undiarized turns render as *Unassigned* with a picker, and on messy audio a person listening often beats an automatic system. ## Practical tips - Keep clips short enough to judge in one or two plays; long clips lower agreement. - For event detection, agree on how precise boundaries must be and measure agreement at the span level, see [Inter-Annotator Agreement](/docs/guides/inter-annotator-agreement). - Normalize loudness across clips so quality ratings aren't driven by volume. ## Further reading - [Audio Annotation feature reference](/docs/features/audio-annotation) - [Annotating Whisper Transcripts](/docs/guides/annotating-whisper-transcripts) - [Annotating YouTube Subtitles](/docs/guides/annotating-youtube-subtitles) - [Transcript Formats](/docs/annotation-types/transcript-formats) - [Video Annotation](/docs/guides/video-annotation) - [Span Annotation](/docs/guides/span-annotation) - [Audio Annotation Tools Compared](/docs/guides/audio-annotation-tools-compared) --- # Video Annotation Section: Guides Source: https://www.potatoannotator.com/docs/guides/video-annotation > How to annotate video in Potato, frame-by-frame navigation, temporal segment labeling, per-frame classification, and tracking objects across frames. **Video annotation adds a time axis to image work. The same clip can be labeled as a whole, segmented into time intervals ("the goal happens from 0:12 to 0:15"), or annotated frame by frame.** Potato provides frame navigation and temporal controls so annotators can move through a clip precisely. Video tasks are central to [activity recognition](https://en.wikipedia.org/wiki/Activity_recognition) and [object tracking](https://en.wikipedia.org/wiki/Video_tracking). ## Clip-level classification The simplest task: one label for the whole clip. ```yaml annotation_schemes: - annotation_type: radio name: action description: "What is the main action in this clip?" labels: [Walking, Running, Sitting, Jumping, Other] ``` ## Temporal segments, when something happens To mark intervals on the timeline, use a span over the video's time axis, just like [sound event detection](/docs/guides/audio-annotation) does for audio. ```yaml annotation_schemes: - annotation_type: span name: events description: "Mark the start and end of each event and label it." labels: [Goal, Foul, Substitution, Replay] ``` ## Per-frame annotation and tracking For frame-level work, classifying individual frames or tracking an object across frames, annotators step through the video and annotate at each frame. Decide a sampling rate (every frame, every Nth frame, or keyframes only); labeling every frame is expensive, so most projects subsample. ## Keeping video annotation consistent - **Boundary precision.** Agree how exact segment start/end must be; frame-level precision is costly. - **Occlusion and exit.** Write rules for when a tracked object is hidden or leaves the frame. - **Workload.** Video is the most time-consuming modality, pilot to estimate cost before scaling, and consider [LLM/vision pre-annotation](/docs/guides/llm-pre-annotation) to seed labels. ## Further reading - [Audio Annotation](/docs/guides/audio-annotation), the same temporal-span ideas - [Image Annotation](/docs/guides/image-annotation) - [Span Annotation](/docs/guides/span-annotation) - [Video Annotation Tools Compared](/docs/guides/video-annotation-tools-compared) --- # Image Annotation Section: Guides Source: https://www.potatoannotator.com/docs/guides/image-annotation > How to annotate images in Potato, classification, multi-label tagging, bounding boxes, polygons, and landmarks, and export to COCO/YOLO. **Image annotation ranges from a single label per image to precise regions drawn on the pixels, boxes, polygons, and points.** The right level depends on what your model needs: a classifier needs labels, a detector needs [bounding boxes](https://en.wikipedia.org/wiki/Minimum_bounding_box), and a segmentation model needs polygons. For the feature reference see [Image Annotation](/docs/features/image-annotation). ## Whole-image classification and tagging Use `radio` for one label, `multiselect` for several. The [image classification showcase](/showcase/image-classification) is a working example. ```yaml annotation_schemes: - annotation_type: multiselect name: contents description: "Select everything visible in the image." labels: [Person, Vehicle, Animal, Building, Vegetation] ``` ## Regions: boxes, polygons, landmarks For localization, annotators draw on the image: - **Bounding boxes** for [object detection](https://en.wikipedia.org/wiki/Object_detection). - **Polygons** for [image segmentation](https://en.wikipedia.org/wiki/Image_segmentation), where object shape matters. - **Landmarks / keypoints** for poses and faces. Potato's image annotation supports these region types with per-class colors, the same way [span annotation](/docs/guides/span-annotation) works for text. ## Boundary and labeling rules - **Tightness.** Do boxes hug the object or include a margin? Be consistent. - **Occlusion and truncation.** Decide how to box a partly hidden object. - **Small objects and crowds.** Set a minimum size and a rule for dense scenes. These rules drive your agreement far more than the drawing tool does. ## Export for vision models Potato exports image annotations to [COCO](https://cocodataset.org/) and YOLO formats, which detection and segmentation training pipelines read directly. See [Exporting Annotations for ML](/docs/guides/exporting-annotations-for-ml). ## Further reading - [Image Annotation feature reference](/docs/features/image-annotation) - [Video Annotation](/docs/guides/video-annotation) - [LLM and Vision Pre-Annotation](/docs/guides/llm-pre-annotation) - [Image Annotation Tools Compared](/docs/guides/image-annotation-tools-compared) --- # How to Annotate Whisper Transcripts Section: Guides Source: https://www.potatoannotator.com/docs/guides/annotating-whisper-transcripts > How to take Whisper or WhisperX output and turn it into a running annotation project: which output file to keep, when you need diarization, how to assign speakers, and how to export with the timings intact. **Whisper gives you timed segments, and an annotation tool should read them as they are. Keep the `.json` output rather than the `.txt`, run WhisperX if you need speaker labels, then point Potato at the folder.** No conversion script, no reformatting step. For the format-by-format detail, see [Transcript Formats](/docs/annotation-types/transcript-formats). [Whisper](https://github.com/openai/whisper) is OpenAI's open-source [speech recognition](https://en.wikipedia.org/wiki/Speech_recognition) model, and it has become the default first step for anyone with a pile of audio to work through. What it produces is a set of timed segments, which is most of what an annotation project needs. The gap is everything between "I have transcripts" and "annotators are labeling them." > **Note:** This guide starts from transcripts you already have, produced by an ASR pipeline you ran yourself. From version 2.9, Potato can instead run Whisper and speaker diarization on your own machine with `potato transcripts --transcribe --diarize`, described in [Transcribing locally](/docs/annotation-types/transcript-formats#transcribing-locally). Either way the transcription finishes before annotation starts, so the upstream choices below decide what your annotators can label. ## Which Whisper output file should I keep? Whisper writes several files and the choice matters more than it looks: ```bash whisper interview_01.mp3 --model medium --output_format json --word_timestamps True ``` | File | Contains | Use it? | |------|----------|---------| | `.json` | Segments with start and end times, optionally per-word timings | **Yes, this one** | | `.srt` / `.vtt` | Segments with timings, no metadata | Yes, works fine | | `.tsv` | Start and end in milliseconds plus text | Yes | | `.txt` | Text only, **no timings** | No, nothing to sync to audio | If the `.txt` is all you kept, the alignment is gone and you cannot recover it without re-running the model. That is the most common reason a transcript arrives in Potato as one undifferentiated block of text. ## Does Whisper label speakers? No. Plain Whisper transcribes; it does not tell you who is talking. Every turn arrives unassigned, which is fine for content annotation and painful for anything speaker-related. [Speaker diarization](https://en.wikipedia.org/wiki/Speaker_diarisation) is a separate step. Three ways to handle it: **Run [WhisperX](https://github.com/m-bain/whisperX)**, which wraps Whisper with [pyannote](https://github.com/pyannote/pyannote-audio) diarization: ```bash whisperx interview_01.mp3 --model medium --diarize --output_format json ``` Its output carries a `speaker` field per segment (`SPEAKER_00`, `SPEAKER_01`, and so on) and Potato reads it directly. **Use a cloud API with diarization enabled.** Deepgram with `diarize=true`, AssemblyAI with `speaker_labels`, AWS Transcribe, and Rev.ai all produce speaker labels, and Potato reads all four natively. ![Three transcript turns with hatched grey backgrounds, each labeled Unassigned with a dropdown caret.](/images/docs/transcript-unassigned-turns.png "Undiarized turns render as Unassigned with a speaker picker") **Let annotators do it.** When turns arrive undiarized, each bubble renders as *Unassigned* with a picker. On messy audio, overlapping speech, or a room with background noise, a person listening is often more accurate than automatic diarization. For a small corpus this is a reasonable trade rather than a fallback. ## How do I build the data file? Point the converter at your Whisper output folder: ```bash potato transcripts ./whisper_out --media-dir ./audio -o data/interviews.json ``` Transcripts pair to media by basename, so `interview_01.json` finds `interview_01.mp3`. Item ids come from the filename, and Whisper's doubled `interview_01.mp3.json` extension is handled without producing an id of `interview_01.mp3`. Check what it understood before you commit to it: ```bash potato transcripts ./whisper_out --dry-run ``` ``` Scanned 3 file(s): interview_01.json Whisper JSON 42 turns 891.4s undiarized interview_02.json WhisperX JSON 51 turns 1120.8s 2 speaker(s): SPEAKER_00, SPEAKER_01 interview_03.json Whisper JSON 38 turns 754.2s undiarized 3 item(s), 131 turn(s). ``` A file reporting `plain text` or zero turns is your problem file. Almost always it is a `.txt` that got mixed in with the `.json` output. ## Can I skip the conversion step? Yes. A data file can point straight at the transcript files on disk: ```json {"id": "int_001", "conversation": {"audio": "media/int_001.mp3", "transcript": "media/int_001.srt"}} ``` Potato reads the file, detects the format from its contents, and normalizes it when the instance renders. Your transcripts stay as files you can diff and re-export instead of being baked into a data blob. ## What does the config look like? The generated data file has this shape: ```json { "id": "interview_01", "conversation": { "audio": "audio/interview_01.mp3", "turns": [ {"turn_id": "t0", "speaker": "SPEAKER_00", "start": 0.0, "end": 6.5, "text": "Welcome back."} ] } } ``` And the matching config, which `potato transcripts --emit-config` will print for you: ```yaml annotation_task_name: "Interview Annotation" task_dir: . data_files: - data/interviews.json item_properties: id_key: id text_key: conversation instance_display: fields: - key: conversation type: audio_dialogue label: "Transcript" span_target: true display_options: show_timestamps: true allow_speaker_assignment: auto annotation_schemes: - annotation_type: span name: topics description: "Highlight topic mentions" labels: - name: policy - name: personal ``` ![Transcript turns rendered as colored speaker bubbles with per-turn play buttons, timestamps, and an inline label question under each turn.](/images/docs/transcript-speaker-bubbles.png "The audio_dialogue display, with a per-turn question attached to every turn") The `audio_dialogue` display renders turns as speaker bubbles synced to the audio, with a play button on each turn that plays only that turn. `span_target: true` lets spans run across turn boundaries, and the offsets stay stable when a speaker gets reassigned. If you would rather have annotators check the transcription itself than label its content, `speech_transcript` gives you segment cards with error tags and a correction box. For overlap and interruption work, `voice_interaction` gives you a dual-track timeline. Both read the same files. ## How do annotators assign speakers? Define a roster so speakers get stable names, colors, and sides: ```yaml display_options: allow_speaker_assignment: auto speakers: - id: interviewer name: "Interviewer" color: "#7c3aed" side: left - id: participant name: "Participant" color: "#059669" side: right ``` `auto` turns the picker on when there are undiarized turns to label. Set it to `true` to allow reassignment even when the source did label speakers, which is what you want if you are correcting diarization errors rather than filling in blanks. Assignments persist with the annotations, keyed to a stable turn id, so they survive a reload. ## How do I get the annotations back out? JSON, JSONL, and CSV export work as usual. When the time alignment needs to survive, export tiered annotations to ELAN EAF or Praat TextGrid: ```bash python -m potato.export --config config.yaml --format eaf --output ./out/ python -m potato.export --config config.yaml --format textgrid --output ./out/ ``` Both round-trip. Potato reads EAF and TextGrid as input too, so you can annotate in Potato, refine in [ELAN](https://archive.mpi.nl/tla/elan) or [Praat](https://www.fon.hum.uva.nl/praat/), and read the result back in. ## Further reading - [Transcript Formats](/docs/annotation-types/transcript-formats), every supported format and how it is detected - [Annotating YouTube Subtitles](/docs/guides/annotating-youtube-subtitles), the caption-file version of this workflow - [Audio Annotation](/docs/guides/audio-annotation), for waveform work that does not start from a transcript - [Transcript Format Ingestion](/showcase/transcript-formats), a runnable showcase design with six formats side by side - [Speaker Diarization](/showcase/speaker-diarization), a runnable showcase design - [Inter-Annotator Agreement](/docs/guides/inter-annotator-agreement), for measuring agreement on turn-level labels --- # How to Annotate Image Segmentation Masks Section: Guides Source: https://www.potatoannotator.com/docs/guides/image-segmentation-annotation > Polygons, brush masks and click-to-segment, when each is the right tool, how to write boundary guidelines annotators can follow, and how to measure agreement on a mask. **Segmentation labels which pixels belong to an object, and almost all of its difficulty is in the boundary rule rather than the drawing tool.** Two annotators given the same image and no rule about shadows will disagree systematically, and no amount of tooling fixes that. ## Polygon or mask | | Polygon | Brush mask | |---|---|---| | Holes and disconnected parts | Awkward | Natural | | Thin structures (wires, hair, legs) | Poor | Good | | Format support | Wide: COCO, YOLO, CVAT, LabelMe | Narrower: COCO RLE, DAVIS, mask PNG | | Editability | Move a vertex | Repaint a region | | Storage | Small | Large unless sparse | Polygons round-trip through more formats, which matters if the labels have to leave the tool. Masks are the honest answer when the object genuinely has holes or wisps. If you are unsure, look at what the model consuming the labels needs. A [semantic segmentation](https://en.wikipedia.org/wiki/Image_segmentation) model consumes masks either way; the question is only what the annotators find least error-prone. ## Write the boundary rule first This is the part that determines your agreement number. Decide, and give examples for: - **Shadow and reflection.** In or out? Consistently. - **Transparent and translucent parts.** A glass, a windscreen, a plastic bag. - **Thin structures.** Where does a wire stop being worth tracing? - **Occlusion.** Mask only the visible part, or infer the whole object? - **Contact.** Two touching instances: where exactly is the split? - **Motion blur.** Where is the boundary of a blurred edge? A single worked example per rule is worth more than a paragraph of prose. If annotators disagree after this, you will see it as **low σ with high detection agreement** — they are finding the same objects and drawing them differently, which is a guidelines problem rather than a carelessness problem. ## Start from a model Click-to-segment gives a first pass in roughly 130 ms, in the browser, with no GPU: ```yaml annotation_schemes: - annotation_type: image_annotation name: objects description: "Outline every object." source_field: image tools: [sam, polygon, brush, eraser] labels: - {name: object, color: "#6e56cf"} ``` ```bash potato download-models onnxruntime potato download-models mobile_sam ``` Positive clicks add to a mask, negative clicks cut away, a box constrains the search, and an existing mask can be refined. **The correction is the annotation value.** If your annotators accept the first mask every time, you have collected a record of the model agreeing with itself — see [telling review from rubber-stamping](/docs/guides/detecting-rubber-stamped-prelabels), because agreement statistics will not tell you. ## Measuring agreement on masks Do not use mean IoU between annotators. It answers "is this task easy", not "do these annotators agree" — see [measuring agreement on bounding boxes](/docs/guides/measuring-agreement-on-bounding-boxes) for why. Use [STAPLE](https://pubmed.ncbi.nlm.nih.gov/15250643/), which estimates a latent consensus boundary along with a **sensitivity** and **specificity** per annotator: - Low sensitivity means under-segmenting, drawing inside the true boundary. - Low specificity means over-segmenting, drawing generously around it. They have opposite fixes, which is why a single accuracy number is not enough to give an annotator feedback. STAPLE also weights by demonstrated reliability rather than counting votes. Two careful annotators outnumbered three-to-two by noisy ones score Dice 0.846 under majority vote and 1.000 under STAPLE. ## Export COCO carries polygons and RLE masks, including `iscrowd` crowd regions. DAVIS carries per-instance masks. YOLO carries polygons but not masks. Check the [format matrix](/docs/vision-spatial/cv-formats) before committing to a mask workflow if the labels have to end up somewhere specific. ## Further reading - [Interactive segmentation](/docs/vision-spatial/segmentation) - [Mask consensus with STAPLE](/docs/measurement/mask-consensus-staple) - [Geometry primitives](/docs/vision-spatial/geometry-primitives) - [Image annotation guide](/docs/guides/image-annotation) --- # How to Annotate YouTube Subtitles Section: Guides Source: https://www.potatoannotator.com/docs/guides/annotating-youtube-subtitles > How to download YouTube captions with yt-dlp and annotate them in Potato, what auto-generated captions can support, and what to check before sharing media. **Download captions with [`yt-dlp`](https://github.com/yt-dlp/yt-dlp) and point Potato at the files. Auto-generated captions have no punctuation and no speaker labels, and their line breaks are meaningless, which constrains what you can sensibly annotate.** Potato reads every caption format `yt-dlp` emits. See [Transcript Formats](/docs/annotation-types/transcript-formats) for the full list. Video platforms are a large and convenient source of naturalistic speech data. [Closed captioning](https://en.wikipedia.org/wiki/Closed_captioning) makes that speech text, and the text is small, easy to move around, and easy to annotate. The catch is that not all captions are the same kind of object, and a scheme designed for one kind breaks on the other. ## How do I download the captions? `yt-dlp` fetches subtitles with or without the media: ```bash # Human-written subtitles, if the uploader provided any yt-dlp --write-subs --sub-langs en --skip-download # Auto-generated captions yt-dlp --write-auto-subs --sub-langs en --sub-format vtt --skip-download ``` Potato reads `vtt`, `srt`, `json3`, `ttml`, and the `srv1`, `srv2`, and `srv3` variants, so whichever `--sub-format` you pick will parse. Choose based on what you want to keep: `json3` preserves per-word timings, while [WebVTT](https://en.wikipedia.org/wiki/WebVTT) and [SubRip](https://en.wikipedia.org/wiki/SubRip) are easier to read and edit by hand. ## What is the difference between human subtitles and auto-captions? They look alike in a text editor and behave very differently as annotation input: | | Human subtitles | Auto-captions | |---|---|---| | Punctuation | Yes | No, or unreliable | | Sentence boundaries | Meaningful | Arbitrary | | Speakers | Sometimes, via `` tags | Never | | Word timings | No | Yes, in `json3` | | Verbatim | Often condensed for reading | Closer to what was said | Neither is strictly better. Human subtitles read well but get edited for readability, so they drop filler, compress repetition, and sometimes rewrite a sentence entirely. Auto-captions are closer to verbatim, which is what you want for anything about how people actually talk, but they arrive as an unpunctuated stream. ## What can I annotate on auto-captions? **Do not design a sentence-level annotation scheme over auto-captions.** Their cue boundaries fall wherever the caption window filled up rather than where a sentence ended, which makes a cue a display unit with no linguistic status. Ask annotators to rate sentences and each of them will pick different sentence boundaries, so your agreement numbers end up measuring that variation instead of the judgment you cared about. Two workable options: - **Annotate the caption windows as they are.** Label each cue with whatever applies to it, and accept that the unit is arbitrary but consistent. Fine for topic, presence of a phenomenon, or anything you can judge on a fragment. - **Re-segment upstream.** Run a punctuation-restoration model or an ASR system that punctuates, then re-time. More work, but you get real units. If you have human subtitles, this problem mostly goes away. The cue boundaries were placed by someone reading along, so they usually respect clauses. ## What about the video? Captions are small and easy to redistribute. Video usually is not. Three options: ```bash # Media hosted somewhere your annotators can reach potato transcripts ./captions --media-url-prefix https://cdn.example.org/video -o data/talks.json # Media downloaded locally as audio only yt-dlp -f 'ba' -x --audio-format mp3 -o './audio/%(id)s.%(ext)s' potato transcripts ./captions --media-dir ./audio -o data/talks.json ``` The third option is no media at all. Transcripts without media annotate fine; annotators read instead of listening. If that is your plan, say so in the instructions, because annotators otherwise assume the missing player is a bug and file it as one. > **Warning:** Check your rights before redistributing anything you did not create. Downloading captions or media for your own research is a different question from serving that media to a pool of annotators, and platform terms, copyright, and your institution's rules all bear on it. Resolve this before you build the pipeline, not after. ## How do I set up the task? The converter writes a data file with the transcript already normalized: ```bash potato transcripts ./captions --media-dir ./audio -o data/talks.json ``` ```json { "id": "talk_01", "conversation": { "audio": "audio/talk_01.mp3", "turns": [ {"turn_id": "t0", "speaker": null, "start": 0.0, "end": 4.2, "text": "so the thing about caption windows is"} ] } } ``` `speaker: null` is what an auto-caption gives you. Those turns render as *Unassigned* with a picker, so annotators can attribute them while they listen. ![Three auto-caption turns with hatched grey backgrounds, each labeled Unassigned with a dropdown caret, and an inline question attached to each turn.](/images/docs/transcript-unassigned-turns.png "Auto-captions carry no speakers, so each cue is Unassigned until an annotator picks one") ```yaml annotation_task_name: "Talk Annotation" task_dir: . data_files: - data/talks.json item_properties: id_key: id text_key: conversation instance_display: fields: - key: conversation type: audio_dialogue label: "Captions" span_target: true display_options: show_timestamps: true allow_speaker_assignment: auto annotation_schemes: - annotation_type: radio name: cue_topic description: "What is this caption window about?" labels: [setup, argument, example, aside] turn_level: true turn_binding: field: conversation ``` `turn_level: true` attaches the question to every cue rather than to the video as a whole, and `turn_binding.field` names the display field those cues come from. The cue is the right unit to ask about when the cues are all you have. Run it: ```bash potato start config.yaml -p 8000 ``` You can also skip the converter entirely and point a data file at the caption files on disk. See [Annotating Whisper Transcripts](/docs/guides/annotating-whisper-transcripts), where the rest of this workflow is the same. ## Further reading - [Transcript Formats](/docs/annotation-types/transcript-formats), every supported caption and transcript format - [Annotating Whisper Transcripts](/docs/guides/annotating-whisper-transcripts), the ASR-output version of this workflow - [Transcript Format Ingestion](/showcase/transcript-formats), a runnable showcase design with six formats side by side - [Video Annotation](/docs/guides/video-annotation), for annotating the video rather than its captions - [Designing Data Formats for Annotation](/docs/guides/data-formats-for-annotation) - [Timed Text Markup Language](https://en.wikipedia.org/wiki/Timed_Text_Markup_Language), the broadcast caption standard Potato also reads --- # How to Label Objects by Typing Their Name Section: Guides Source: https://www.potatoannotator.com/docs/guides/open-vocabulary-object-detection > Open-vocabulary detection lets an annotator type a phrase and get every match boxed. How Grounding DINO works, what the licence allows, and where the approach breaks down. **Open-vocabulary detection means the annotator types a phrase and every match in the image comes back boxed, without the class having been in any training set.** It is the fastest way to bootstrap a detection dataset for classes no off-the-shelf detector covers. ## How it works [Grounding DINO](https://arxiv.org/abs/2303.05499) takes an image and a caption, and scores 900 candidate boxes against every word of that caption. Give it "cat . bus . stop sign" and it returns boxes with per-phrase confidences. That is a different mechanism from a fixed-class detector, and it changes how you should write your prompts: - **Phrases beat labels.** "traffic cone" is a phrase the model has seen in captions; `cone_orange_road` is not. - **Context helps.** "person riding a bicycle" can localise better than "person" on a busy street. - **Terse is worse.** The model was trained on natural language, so write natural language. ## The licence matters Text-prompt labelling does not require SAM 3. Grounding DINO is **Apache-2.0**, so nothing about the labelling model constrains what you do with the dataset. SAM 3 comes under Meta's SAM License, which permits commercial use, places acceptable-use limits on the model, and has to be accepted before the weights are fetched. Check the licence before you build a dataset rather than after. If your labelling model's licence is non-commercial and your dataset is not, you have a problem that is expensive to discover late. Potato's [model zoo](/docs/vision-spatial/model-zoo) states each model's licence, marks the non-commercial ones in `download-models --list`, and refuses to fetch a model whose licence requires acknowledgement without `--accept-licence`. ## Configuration ```yaml annotation_schemes: - annotation_type: image_annotation name: objects description: "Find and label the objects." source_field: image tools: [bbox, polygon, sam] labels: [cat, bus, stop sign] text_prompt: phrases: [cat, bus, stop sign] box_threshold: 0.3 text_threshold: 0.25 segment: false ``` The two thresholds do different jobs and should be tuned separately: - **`box_threshold`** decides whether a box is offered at all. Raise it if annotators are drowning in false positives. - **`text_threshold`** decides which phrase labels an offered box. Raise it if boxes are appearing under the wrong label. Setting `segment: true` hands each accepted box to the SAM decoder, turning it into a mask. ## Quantization is a real choice If you are deploying one of these yourself, do not accept the default export. Measured against the 686 MB full-precision Grounding DINO export: | Export | Box IoU vs full precision | Size | |---|---|---| | `q4f16` | 0.972 | 151 MB | | `int8` | 0.874 | 201 MB | `q4f16` keeps substantially more geometry **and** is smaller. Taking int8 because it is the conventional choice ships a worse model in a larger file. ## Where it breaks down - **Fine-grained classes.** "Golden retriever" versus "labrador" is beyond it; it will box the dog. - **Domain-specific vocabulary.** Medical, industrial and scientific terms are usually not in the training captions. - **Counting.** It finds instances; it does not reliably find *all* instances in a crowded scene. - **Negations and relations.** "The cup that is not on the table" is not a query it answers. For those cases the suggestions are still a useful starting point, but expect to draw. ## Suggestions are not annotations Keep them apart all the way through. A box the annotator accepted is ground truth; a box they did not is evidence about the model, and folding the two together destroys the only thing the second is for. If you want to know whether the accepting is real review, that is a [timing question](/docs/guides/detecting-rubber-stamped-prelabels), not an accuracy question. ## Further reading - [Text prompting reference](/docs/vision-spatial/text-prompting) - [Interactive segmentation](/docs/vision-spatial/segmentation) - [Model zoo and licences](/docs/vision-spatial/model-zoo) --- # How to Annotate Video Object Tracking Section: Guides Source: https://www.potatoannotator.com/docs/guides/video-object-tracking > Track an object through a clip with model-assisted mask propagation, handle occlusion honestly, and measure temporal agreement with a tolerance sweep. **Tracking annotation means labelling the same object across frames, and the expensive part is not the first mask — it is the hundreds that follow.** Model-assisted propagation removes most of that, provided the model is honest about when it has lost the object. ## The workflow 1. Mask the object on one frame. 2. Propagate forward. Each result arrives as a keyframe. 3. Scrub the keyframes, correct where the mask has drifted. Correcting a frame re-anchors the track from that point. ```yaml annotation_schemes: - annotation_type: video_annotation name: tracks description: "Draw the object once, then press Track forward." source_field: video_url mode: tracking labels: - {name: subject, color: "#d1495b"} tracking_options: interpolation: linear auto_advance_frames: 5 video_fps: 12 frame_stepping: true ``` ## Occlusion should return nothing The single most useful property of a propagation model in an annotation loop is that it **declines** when the object is hidden, returning an empty frame rather than its best guess. A plausible wrong mask on an occluded frame is worse than a gap: the annotator has to notice it, decide it is wrong, and remove it. An empty frame is immediately readable as "the object is not visible here", which is also the correct label. [SAM 2](https://arxiv.org/abs/2408.00714) does this, and its memory modules are what make propagation hold up across an occlusion rather than losing the object at the far side. Measured against known ground truth, per-frame IoU stays at 0.974 to 0.979 with no decay across the sequence. ## Held versus blended On frames between keyframes, a mask is either **held** (repeated from the last keyframe) or **blended** (interpolated). Held is the honest choice, and held frames should be visually marked. A blended mask looks smoother and is wrong in a way nobody can see. If your consumer needs interpolated geometry, interpolate at export time where it is a stated transformation, rather than in the interface where it is invisible. ## Polygon interpolation is harder than it looks Interpolating between two polygon keyframes fails in two ways naive implementations get wrong: - The two keyframes may have **different vertex counts**. - The annotator may have started tracing at **a different point** around the shape. Arc-length resampling with rotational alignment handles both. Without it, an interpolated polygon can turn inside out between keyframes. ## Where propagation should run On the server, if the model has memory. The cost is per frame rather than per prompt, and a hundred frames of a video model in a browser tab is minutes of frozen UI. Browser-side carry-forward that re-prompts frame to frame is a different, lighter thing — useful, but not the same capability, and worth not conflating in your own documentation. ## Measuring agreement Two annotators will not choose the same frame for an event boundary, so temporal agreement depends on a matching tolerance. **Report the sweep**, not one threshold: - High at 2 s, low at 0.25 s: they agree an event happened, not precisely when. Usually a guidelines problem. - Low at both: they disagree about whether it happened. For the masks themselves, use the [mask agreement](/docs/measurement/mask-consensus-staple) machinery per frame. ## A bug worth checking for in your own tooling HTML video reports `currentTime` truncated to six decimal places. Converting that to a frame index with a naive multiply-and-floor gives an off-by-one after every seek: frame 14 of a 12 fps clip reads back as 13. This was live in Potato before 2.8 and affected every annotator on every clip. If you have frame-indexed video annotations from any tool, it is worth verifying against a known frame. ## Further reading - [Video tracking reference](/docs/vision-spatial/video-tracking) - [DAVIS showcase design](/showcase/davis-video-object-segmentation) - [Temporal agreement](/docs/measurement/temporal-agreement) - [Video annotation guide](/docs/guides/video-annotation) --- # How to Annotate 3D Point Clouds Section: Guides Source: https://www.potatoannotator.com/docs/guides/point-cloud-annotation > Label lidar and photogrammetry data with oriented 3D cuboids. Formats, level of detail, why rotation should be a quaternion, and how to measure agreement on a 3D box. **Annotating a [point cloud](https://en.wikipedia.org/wiki/Point_cloud) means placing oriented 3D boxes on sparse, unevenly sampled data where a distant object may be a few dozen returns.** The hard parts are not the drawing: they are keeping the viewer interactive, giving the annotator a way to check their work, and choosing a rotation representation that survives a round trip. ## Formats | Format | Notes | |---|---| | KITTI velodyne `.bin` | Raw float32 x, y, z, intensity. No header, so the layout is assumed | | [PCD](https://pointclouds.org/documentation/tutorials/pcd_file_format.html) | ascii, binary, and binary_compressed (LZF) | | [PLY](https://en.wikipedia.org/wiki/PLY_(file_format)) | ascii and both binary endiannesses | | [LAS](https://en.wikipedia.org/wiki/LAS_file_format) | 1.0 to 1.4 | | `.xyz` / `.pts` | `x y z [r g b]` per line | | LAZ | Compressed LAS. Convert first: `laszip -i scan.laz -o scan.las` | Parse server-side rather than in the browser. Four formats means four sets of endianness and record-layout bugs, and re-parsing a two-million-point scan on every page load. ## Keeping it interactive A raw scan is too large to render naively. Two things make it workable: - **Octree level of detail.** Points load by region and by distance to the camera. This is on by default in Potato (`lod: true`). - **A decimation cap** (`max_points`). Treat it as a viewer setting, not a data setting. That distinction bit once and is worth repeating: per-point segment labels used to be stored as indices into the *decimated* cloud, so lowering `max_points` silently re-pointed existing labels at different points. Indices should always refer to the source file. ## Give annotators a 2D check A car at 40 metres is a few dozen returns. Whether the box is tight, the right length, and correctly rotated are close to unanswerable from the cloud alone, and obvious in a camera image. So: **edit in 3D, verify in 2D**. Supply calibration per item and project every box into each camera view. Without that, 3D labelling has no feedback loop — a box that looks correct from the orbit camera can be metres off along the view axis with nothing to say so. Orthographic slab panels give the other half: three axis-aligned views for precise adjustment, fully keyboard-driven. ## Store rotation as a quaternion Most formats store a single yaw angle. Storing a full [quaternion](https://en.wikipedia.org/wiki/Quaternion) internally looks like over-engineering until a round trip loses something. It makes KITTI import lossless, including the roughly 0.85° camera-to-lidar mounting tilt a yaw-only field discards without comment. Export back to KITTI still has to drop pitch and roll — the right behaviour is to report **how much** orientation was discarded rather than flattening the box silently. Cross that gap in the format-specific code, loudly, rather than in the storage layer, silently. ## Frames are where this goes wrong Three conventions that are easy to get wrong and hard to notice: - **Reference frame versus camera frame.** Using a camera's transform where the rectified reference frame is wanted shifts every box by that camera's stereo baseline. Systematic, a few centimetres, and easily mistaken for annotator sloppiness. - **Location is the bottom face in KITTI**, not the centre. Reading it as the centre puts every object half its own height underground. - **Dimension order and axis assignment.** A round-trip test cannot catch a 90° rotation error here, because the inverse conversion makes the same wrong assumption and the two agree with each other. Only a comparison against the reference implementation's own corner formula catches it. ## Measuring agreement Use **exact rotated 3D IoU**. An axis-aligned approximation is fine for level automotive data and wrong for drone, handheld and indoor scans, where it reports disagreement that is an artefact of the measure rather than of the annotators. As with 2D, split the question: did annotators find the same objects, label them the same, and place them the same. See [measuring agreement on bounding boxes](/docs/guides/measuring-agreement-on-bounding-boxes). ## Config ```yaml annotation_schemes: - annotation_type: spatial_annotation name: objects description: "Put a 3D box around every vehicle and pedestrian." source_field: point_cloud calibration_field: calibration tools: [cuboid_3d, point_3d] labels: - {name: car, color: "#FF6B6B", key_value: "1"} - {name: pedestrian, color: "#FFD93D", key_value: "2"} color_mode: height lod: true max_points: 400000 fit_box_height: true ``` Working example: the [KITTI showcase design](/showcase/kitti). ## When another tool is the right answer Segments.ai, Kognic, Deepen AI, Supervisely and Xtreme1/BasicAI are specialists, and for production autonomous-driving pipelines they are ahead: sequence workflows with track propagation, radar and multi-LiDAR, auto-fit cuboid models, and in Deepen's case a full targetless calibration product. Supervisely and Xtreme1 are self-hostable and Supervisely handles far larger clouds. Potato does **not** do point-cloud sequences: annotation is per frame, and track propagation across a sweep is not implemented. Choose it for 3D when you want reliability statistics over spatial labels, or when 3D is one part of a multimodal study. ## Further reading - [Point clouds reference](/docs/vision-spatial/point-clouds) - [Calibration](/docs/vision-spatial/calibration) - [Depth map annotation](/docs/guides/depth-map-annotation) --- # How to Annotate Depth Maps Section: Guides Source: https://www.potatoannotator.com/docs/guides/depth-map-annotation > Depth files do not carry their own unit, zero means no-return rather than close, and the interesting range is never the full range. What to get right before annotating depth. **A depth map opened as an image is a black rectangle, and if it were not, you would still be looking at colours with no idea what distance any of them means.** Three properties separate depth from an image, and all three are silent when you get them wrong. ## The file does not know its own unit Millimetres for RealSense, Azure Kinect and NYU-Depth. 1/256 m for the KITTI completion benchmark. Metres for anything written from a float array. Nothing in the file records which. `depth_scale` is metres per stored unit, and getting it wrong is invisible: the window rescales and the picture looks the same. The only way to catch it is a **readout in metres** checked against something whose distance you know. ## Zero is not a distance Zero is the near-universal "no return" code — the sensor did not measure this pixel. Read as depth, it paints a bright wall across every hole in coverage. Carry non-measurements as NaN and paint them a colour that appears in no colormap. Potato uses magenta, so a hole cannot be misread as near or far depth, and reports what fraction of the map is holes. That fraction matters more than it sounds. Stereo matching needs texture to find correspondences, so a rig facing a blank wall returns no depth over much of the frame. An annotator who cannot see which pixels are missing will read them as geometry and label them. ## The interesting range is never the full range A 16-bit depth map whose content sits between 1200 and 1800 renders with **4 grey levels of separation out of 255** under a naive 8-bit cast. Under a percentile window it renders with **254**. Default the near/far window to the 2nd and 98th percentile of the *valid* pixels — valid meaning after the no-return pixels are excluded, or the holes will dominate the percentiles. ## Configuration ```yaml instance_display: fields: - key: depth type: depth_map label: "Depth" ``` Formats read directly: 16-bit PNG, 16-bit TIFF, NPY and NPZ, PFM, EXR. ## Unprojection Given camera intrinsics, a depth map becomes a point cloud, and the same [3D cuboid tools](/docs/guides/point-cloud-annotation) apply. For RGB-D datasets this is usually the more productive way to annotate: judging a 3D extent is easier in 3D than by reading a colormap. ## What to annotate on depth - **Quality judgements** on estimated depth, against the RGB frame. - **Region labels** where depth is wrong, which is the supervision signal for depth completion. - **3D boxes**, after unprojection. For a quality task, show the RGB frame alongside. Depth errors are usually obvious only in comparison. ## Further reading - [Depth maps reference](/docs/vision-spatial/depth-maps) - [Point cloud annotation](/docs/guides/point-cloud-annotation) - [Media ingest](/docs/vision-spatial/media-ingest) --- # How to Annotate Robot Episodes Section: Guides Source: https://www.potatoannotator.com/docs/guides/robot-episode-annotation > Segment a robot demonstration into phases, judge the outcome, and draw a dense reward curve. Why partial is a real outcome and why the time-series lanes matter more than the video. **A robot demonstration is several synchronized camera streams plus several numeric time series, and every question worth asking about one is temporal.** When did the grasp start, when did it fail, how close to done was it at each moment. So the interface has to be a timeline, not a form. ## Put the signals next to the video Camera views above, the arm's own signals as lanes beneath, one shared time axis, and every annotation layer drawn onto it. This is not a layout preference. A missed grasp is usually visible in the **gripper and wrist-force traces several frames before it is obvious on camera**. Published work supports this: [ATLAS](https://arxiv.org/abs/2604.26637) (TU Wien) reports that showing proprioceptive time series alongside video reduces boundary error against vision-only tools. Two practical consequences: - **Do not hide the lanes behind a tab.** If the annotator has to go looking, they will not. - **Downsample min/max-preserving.** A one-frame force spike drawn into a 300-pixel-wide lane disappears entirely under mean downsampling, and that spike is often the event. ## Segment by sub-goal A phase boundary is where the robot's *objective* changes — reach, grasp, transport, place, retract — not where the video cuts or where the arm happens to pause. Phase taxonomies are task-family specific. Manipulation and navigation need entirely different sets, so this belongs in your config rather than in the tool: ```yaml phases: - {name: reach, color: "#4ECDC4", key_value: "1"} - {name: grasp, color: "#FFD93D", key_value: "2"} - {name: transport, color: "#6C8AE4", key_value: "3"} - {name: place, color: "#95E1A3", key_value: "4"} - {name: retract, color: "#C9A0DC", key_value: "5"} ``` ## Three outcomes, not two `success`, `partial`, `failure`. **Partial is the modal result in real robot data.** Forcing it into a binary destroys the signal that makes the dataset worth annotating: an episode that got the object off the table but dropped it is not the same as one that never moved, and a policy trained as if they were the same learns the wrong thing. Pair it with a failure-cause taxonomy — missed grasp, object slipped, collision, wrong object, wrong placement, timeout — so the failures are analysable rather than just counted. ## Hindsight relabelling An episode that failed at the task it was recorded for is frequently a **successful demonstration of some other task**. Collect what the episode actually shows, and a discarded recording becomes usable training data. This is cheap to collect at annotation time and impossible to recover later, so ask for it even if you are not sure you will use it. ## The reward layer is expensive A dense progress-reward curve is minutes per episode where phases are seconds. Enable it only when you need it, and expect it to dominate your annotation budget. ## Measuring agreement Three layers, three measures: | Layer | Measure | |---|---| | Phase boundaries | Temporal IoU | | Outcome labels | Krippendorff's α | | Reward curves | ICC plus Pearson | **State coverage alongside each.** A correlation computed over 5% of a timeline says nothing about the other 95%, and a coefficient without its coverage will be read as though it covered everything. For boundaries specifically, report a [tolerance sweep](/docs/measurement/temporal-agreement) rather than one threshold. ## Formats LeRobot v2, RLDS/TFDS, HDF5 (RoboMimic and ALOHA), and ROS bags all import. Export a per-frame JSONL sidecar rather than rewriting the dataset, so a read-only public release can carry your annotations without being republished. ## Config ```yaml annotation_schemes: - annotation_type: episode_annotation name: episode_review description: "Mark the phases, judge the outcome, and draw the progress." source_field: episode layers: [phases, outcome, reward] outcomes: [success, partial, failure] failure_causes: [missed grasp, object slipped, collision, timeout] reward_range: [0.0, 1.0] series_shown: [gripper, wrist_force] ``` Working example: the [LeRobot showcase design](/showcase/lerobot-episode-annotation). ## Related tools [ATLAS](https://arxiv.org/abs/2604.26637) is purpose-built for single-annotator boundary precision with ROS bags. [ELAN](https://archive.mpi.nl/tla/elan) and BORIS remain the standards for behavioural coding. [Rerun](https://rerun.io/) and [Foxglove](https://foxglove.dev/) are the best robotics visualisation tools and now carry structured annotations. Potato's distinct contribution here is being multi-annotator and web-deployed with agreement statistics. ## Further reading - [Robot episodes reference](/docs/vision-spatial/robot-episodes) - [Temporal agreement](/docs/measurement/temporal-agreement) - [World-model evaluation](/docs/guides/world-model-evaluation) --- # Inter-Annotator Agreement Explained Section: Guides Source: https://www.potatoannotator.com/docs/guides/inter-annotator-agreement > How inter-annotator agreement works, when to reach for percent agreement, Cohen's or Fleiss' kappa, or Krippendorff's alpha, and what Potato reports. **Inter-annotator agreement (IAA) measures how often independent annotators give the same label. It is the standard evidence that an annotation task is well-defined and the resulting labels are trustworthy.** Low agreement usually means the *guidelines* are unclear, not that the annotators are careless. The general topic is [inter-rater reliability](https://en.wikipedia.org/wiki/Inter-rater_reliability), the extent to which independent raters assign the same value to the same unit. Potato computes agreement live in the admin dashboard and reports it per annotation scheme, so you can see which question your annotators split on rather than one number for the whole study. See [Quality Control](/docs/features/quality-control) for the dashboard itself. ## Why raw percent agreement isn't enough **Percent agreement** is the simplest measure, the fraction of items annotators labeled identically. Some of that agreement happens by chance, which is what makes the raw fraction misleading. If two annotators both pick "positive" 90% of the time, they will agree on most items even when they label at random. Chance-corrected coefficients subtract the agreement you would expect from the label distribution alone. A chance-corrected coefficient has the form: ``` P_observed − P_expected κ = ───────────────────────── 1 − P_expected ``` where `P_observed` is the actual agreement and `P_expected` is the agreement expected by chance. A value of 1 is perfect agreement; 0 is chance-level. ## The three coefficients - **[Cohen's kappa](https://en.wikipedia.org/wiki/Cohen%27s_kappa)**: two annotators, categorical labels. The classic choice for a pair. - **[Fleiss' kappa](https://en.wikipedia.org/wiki/Fleiss%27_kappa)**: more than two annotators, categorical labels, when different items may be judged by different raters. - **[Krippendorff's alpha](https://en.wikipedia.org/wiki/Krippendorff%27s_alpha)**: the most general option. It works with any number of annotators, handles missing data, and supports nominal, ordinal, interval, and ratio data. Potato computes it for every scheme type it can classify. Use kappa for simple categorical pairs; reach for Krippendorff's alpha when you have many annotators, incomplete overlap, or ordered/continuous labels (where "off by one" should count less than "off by four"). ## Fleiss' kappa or Krippendorff's alpha with three annotators? **If all three annotators labeled every item and the labels are unordered categories, either coefficient is defensible. If any item is missing a label, or the labels are ordered, use Krippendorff's alpha.** The two differ in what they assume: - **Missing labels.** Fleiss' kappa expects the same number of ratings on every item. If one of the three annotators skipped an item, you have to drop that item. Krippendorff's alpha uses every rating it has. - **Level of measurement.** Fleiss' kappa treats labels as unordered, so on a 1–5 scale a 4 against a 5 counts as the same disagreement as a 1 against a 5. Krippendorff's alpha takes a distance metric (nominal, ordinal, interval or ratio) and scores near-misses as smaller disagreements. - **Small samples.** Krippendorff's alpha includes a correction for small samples, so on a pilot of a few dozen items the two can come apart even when the data suit both. On complete, nominal data from a reasonable number of items, the two usually land close together. Report the one your field expects, and name the coefficient and the metric in the paper, because "agreement was 0.72" means different things under each. Potato reports several coefficients at once rather than making you pick one up front. A categorical scheme gets percent agreement, Cohen's kappa, Fleiss' kappa and nominal alpha side by side, and an ordered scheme gets linearly and quadratically weighted kappa, Spearman's rho and ordinal alpha. Before version 2.9, a labeled Likert scale's ordinal alpha ranked the word labels alphabetically, which made it nominal in practice, so recompute ordinal agreement on labeled scales if you reported it from an earlier version. You can also check a table of labels in the browser with the [agreement calculator](/tools/inter-annotator-agreement), which computes nominal, ordinal and interval alpha alongside Cohen's and Fleiss' kappa. ## Interpreting the number There is no universal cutoff. The thresholds below are the ones conventionally used with alpha in content analysis, where researchers rely on data at 0.80 or above and set aside data under 0.67: - **≥ 0.80**: good enough to rely on. - **0.67–0.80**: usable for tentative conclusions; investigate disagreements. - **< 0.67**: revisit the guidelines before trusting the labels. Treat these as prompts to investigate rather than pass/fail gates. Always look at *which* items and *which* labels drive disagreement. ## Measuring it in Potato Agreement needs the same item labeled by more than one person, so have annotators overlap on a shared subset before you turn reporting on. `min_overlap` sets how many annotators an item needs before it counts toward the number: ```yaml agreement_metrics: enabled: true min_overlap: 2 # an item counts once this many annotators have answered it refresh_interval: 60 # seconds between dashboard updates ``` Nothing is reported for a scheme until `min_overlap` annotators have answered the same item, so a single-annotator pilot shows a message rather than a number. For span and structured tasks, measure agreement at the level you care about, because document-level metrics hide boundary disagreements. Potato reports token-level kappa, exact and partial span F1, unitized alpha and [Mathet's gamma](https://aclanthology.org/J15-3003/) for span schemes, which lets you separate a disagreement about the label from a disagreement about where the span starts. ## When agreement is low 1. Read the disagreed items and decide whether the *guideline* is ambiguous or the *item* is genuinely hard. 2. Tighten definitions and add the hard cases as examples. See [Writing Annotation Guidelines](/docs/guides/writing-annotation-guidelines). 3. Re-pilot. If agreement stays low on truly subjective tasks, consider capturing the disagreement itself rather than forcing a single answer. ## Frequently asked questions The questions below cover the range of Krippendorff's alpha, how it is calculated, and agreement with only two annotators. ### Can Krippendorff's alpha be less than -1? No. Alpha falls below 0 when annotators disagree more than chance predicts, and it approaches -1 without reaching it. Two annotators who disagree on every one of N items, using two labels equally often, give an alpha of exactly -1 + 1/N, which is -0.8 for 5 items and -0.95 for 20. [Krippendorff (2011)](https://www.asc.upenn.edu/sites/default/files/2021-03/Computing%20Krippendorff's%20Alpha-Reliability.pdf) gives the range that matters for reliability as 0 to 1 and attributes values below 0 to systematic disagreement or sampling error. A clearly negative alpha points to systematic disagreement, such as annotators reading a scale in opposite directions, and calls for a look at the guidelines rather than a report of the number. ### How do I calculate Krippendorff's alpha? Alpha is 1 minus the ratio of observed disagreement to the disagreement expected by chance. Observed disagreement comes from pairs of labels given to the same item, and expected disagreement comes from pairs drawn from all labels in the study, with a distance metric (nominal, ordinal, interval, or ratio) deciding how much each mismatch counts. Potato computes it per annotation scheme in the admin dashboard once `min_overlap` annotators have labeled the same item, and the [agreement calculator](/tools/inter-annotator-agreement) computes nominal, ordinal, and interval alpha from a table of labels in the browser. ### Can inter-annotator agreement be calculated with only two annotators? Yes. Cohen's kappa is designed for exactly two annotators with categorical labels, and Krippendorff's alpha works with two as well as with more. Both need the two annotators to have labeled the same items independently, and with few items the estimate is unstable, so report the number of items alongside the coefficient. ## Further reading - [Quality Control feature reference](/docs/features/quality-control) - [Gold Standards and Attention Checks](/docs/guides/gold-standards-and-attention-checks) - [Adjudication and Disagreement](/docs/guides/adjudication-and-disagreement) --- # Gold Standards and Attention Checks Section: Guides Source: https://www.potatoannotator.com/docs/guides/gold-standards-and-attention-checks > How to use gold-standard items and attention checks to catch low-quality annotators and keep a project calibrated, with Potato configuration. **Gold standards and attention checks are items where you already know the correct answer. Mixing them into the stream lets you measure each annotator's accuracy and catch people who are rushing, confused, or gaming the task.** They are the front line of [annotation quality control](/docs/features/quality-control), especially in crowdsourcing. ## Gold standards A **gold-standard** item has an expert-verified answer. Sprinkle them in and compare each annotator's response against the known answer to get a per-person accuracy score. Gold items can be silent (used only for scoring) or give immediate feedback (used for training). ```yaml gold_standards: enabled: true items_file: "gold_standards.json" mode: mixed # silent scoring + occasional feedback frequency: 20 # roughly one gold item per 20 ``` Build your gold set from the unambiguous cases your [guidelines](/docs/guides/writing-annotation-guidelines) settled. Don't use genuinely ambiguous items as gold, you'll punish good annotators for reasonable choices. ## Attention checks An **attention check** is an item with an obvious, instruction-embedded answer ("Select 'Disagree' for this item"). It catches annotators who aren't reading. Potato can also flag suspicious timing, answers submitted faster than a human could read. ```yaml attention_checks: enabled: true items_file: "attention_checks.json" frequency: 10 ``` ## Using the signal - **Set a passing accuracy threshold.** Annotators below it can be retrained or excluded. - **Combine with a [training phase](/docs/features/training-phase).** Require a passing score on gold items before live work begins. - **Don't over-check.** Too many checks annoy good annotators and inflate cost. A small, steady rate is enough. For estimating annotator competence and inferring labels from disagreement statistically, see [Adjudication and Disagreement](/docs/guides/adjudication-and-disagreement) and Potato's [MACE](/docs/features/mace) support. ## Further reading - [Quality Control feature reference](/docs/features/quality-control) - [Inter-Annotator Agreement](/docs/guides/inter-annotator-agreement) - [Running a Study on Prolific and MTurk](/docs/guides/crowdsourcing-prolific-mturk) --- # Adjudication and Resolving Disagreement Section: Guides Source: https://www.potatoannotator.com/docs/guides/adjudication-and-disagreement > What to do when annotators disagree, adjudication workflows, aggregation by majority vote, and statistical models like MACE that weight annotators by competence. **Disagreement is normal and informative. Resolving it means turning several annotators' labels into one trusted label, by expert review, by aggregation, or by a statistical model that weights annotators by how reliable they are.** Forcing a single answer too early throws away signal about which items are genuinely hard. ## Three ways to resolve 1. **Majority vote.** Simple and transparent: take the most common label. Works well when annotators are roughly equal and the task is clear, but it treats a careless annotator the same as a careful one. 2. **Expert adjudication.** Route disagreed items to an expert who makes the final call. Most accurate, most expensive. Use it for the items that matter and where aggregation is unreliable. 3. **Statistical aggregation.** Models like [MACE](/docs/features/mace) (Multi-Annotator Competence Estimation) infer each annotator's reliability from their pattern of agreement and produce a weighted "best guess" label plus a competence score per annotator. This down-weights spammers automatically without hand-checking every item. See the underlying idea of [latent-variable models for crowdsourced labels](https://en.wikipedia.org/wiki/Crowdsourcing). ## A practical workflow - Collect overlapping annotations (several people per item). - Aggregate with majority vote or MACE to get a draft label and flag low-agreement items. - Send only the flagged items to expert adjudication. - Feed what you learn back into the [guidelines](/docs/guides/writing-annotation-guidelines). Potato supports an adjudication workflow where a reviewer sees all annotators' labels side by side and records the resolved answer. ## When disagreement is the data For subjective tasks, humor, offense, emotion, persistent disagreement can reflect real differences between people, not error. In those cases, consider keeping the full distribution of labels (sometimes called *soft labels* or *perspectivist* annotation) instead of collapsing to one answer. Potato supports capturing distributions rather than forcing consensus. ## Further reading - [MACE feature reference](/docs/features/mace) - [Inter-Annotator Agreement](/docs/guides/inter-annotator-agreement) - [How Many Annotators Do You Need?](/docs/guides/how-many-annotators) --- # How Many Annotators Do You Need? Section: Guides Source: https://www.potatoannotator.com/docs/guides/how-many-annotators > How to decide annotator count and overlap for an annotation project, balancing agreement, cost, and statistical confidence, with Potato overlap settings. **There is no single right number, but the decision comes down to three levers: how many people label each item (overlap), how clear the task is, and your budget. Clear tasks need little overlap; subjective tasks need more.** This guide gives rules of thumb and the settings to implement them. ## Overlap vs. coverage Every annotation budget is split between two goals: - **Coverage**: labeling more distinct items (each once). - **Overlap**: labeling the same items multiple times, which buys you agreement estimates and the ability to aggregate. You cannot maximize both. A common pattern: fully overlap a *subset* to measure [agreement](/docs/guides/inter-annotator-agreement), then single-annotate the rest once you trust the task. ## Rules of thumb - **Objective tasks** (clear categories, high agreement): 1 annotator for most items, with 2–3× overlap on a 5–10% sample to monitor quality. - **Moderately subjective tasks**: 3 annotators per item, resolved by majority vote or [MACE](/docs/guides/adjudication-and-disagreement). - **Highly subjective tasks** (offense, emotion, preference): 5+ annotators per item, and consider keeping the full label distribution rather than collapsing it. More annotators reduce the variance of an item's aggregate label, with diminishing returns, going from 1 to 3 helps far more than 7 to 9. ## Setting overlap in Potato Potato's [task assignment](/docs/features/task-assignment) controls how many annotators see each item and how items are distributed. ```yaml automatic_assignment: on: true instance_per_annotator: 50 # how many items each person labels labels_per_instance: 3 # how many annotators label each item (overlap) ``` ## Quality checks alongside overlap Headcount doesn't help if some annotators are unreliable. Pair overlap with [gold standards and attention checks](/docs/guides/gold-standards-and-attention-checks) so you can weight or exclude low-quality work before aggregating. ## Further reading - [Task Assignment feature reference](/docs/features/task-assignment) - [Inter-Annotator Agreement](/docs/guides/inter-annotator-agreement) - [Running a Study on Prolific and MTurk](/docs/guides/crowdsourcing-prolific-mturk) --- # Agreement for Spans and Structured Outputs Section: Guides Source: https://www.potatoannotator.com/docs/guides/agreement-for-spans > Why Cohen's and Fleiss' kappa break down for span, NER, and structured annotation, and what to use instead: F1-as-agreement, exact vs partial match, and Krippendorff's unitized alpha. **Chance-corrected agreement like Cohen's kappa assumes every annotator labels the same fixed set of items from the same fixed set of categories. Span annotation breaks that assumption: annotators can disagree on where a span starts, where it ends, and whether it exists at all. For spans, the standard reliability measure is pairwise F1, not kappa, and you have to decide up front whether a partial boundary overlap counts as agreement.** This guide explains why the usual metrics fail here and what to report instead. ## Why kappa doesn't fit spans A [chance-corrected coefficient](/docs/guides/inter-annotator-agreement) needs three things: a fixed list of items, a fixed list of labels, and the ability to compute how often annotators would agree *by chance*. Span tasks supply none of them cleanly. There is no predetermined list of "items" to label, an annotator invents the spans as they read, so two people can produce different numbers of spans over the same document. And there is no meaningful negative class: the "items nobody marked" are every possible substring, an astronomically large and ill-defined set. That last point is what breaks chance correction. [Hripcsak and Rothschild (2005)](https://doi.org/10.1197/jamia.M1733) showed that when the negative class is very large or undefined, as in information retrieval and span extraction, the chance of two annotators agreeing on the same arbitrary span is effectively zero, so chance correction barely changes anything and the assumptions behind kappa no longer hold. Their result is the standard justification for a cleaner alternative: the [F-measure](https://en.wikipedia.org/wiki/F-score) itself is the appropriate agreement statistic. Treat one annotator's spans as the reference and the other's as predictions, compute F1, and average over all annotator pairs. Because F1 is symmetric, the pair order doesn't matter. ## Exact match or partial match: decide before you measure The number you report depends entirely on what counts as a hit, and there is no universal answer, so state your choice. - **Exact match**: two spans agree only if both boundaries are identical. Strict, and the right call when boundaries carry meaning (legal citations, chemical names). - **Partial / overlap match**: two spans agree if they overlap at all, or past some threshold. More forgiving, and reasonable when the *presence* of an entity matters more than its exact extent. - **Boundary vs. label**: for typed spans (NER), separate two questions: did annotators mark the same extent, and did they give it the same type? Reporting them together hides which one is actually causing your disagreement. [Artstein and Poesio (2008)](https://aclanthology.org/J08-4004/) is the standard survey of agreement for computational linguistics and works through this "unitizing" problem, disagreement about how to segment the text into units, in detail. It is the reference to cite when you need to defend a methodology choice. ## When you do want a chance-corrected number If you can reduce the task to a fixed set of units, chance correction becomes valid again. Two common reductions: - **Token-level labeling**: recast the span task as a label per token (the BIO scheme). Now every token is a fixed item with a small label set, and Fleiss' kappa or [Krippendorff's alpha](https://en.wikipedia.org/wiki/Krippendorff%27s_alpha) applies directly. The catch is that token-level agreement looks inflated: most tokens are the easy "outside" class, so a high number can hide real boundary disagreement. - **Unitized alpha**: [Krippendorff (2004)](https://us.sagepub.com/en-us/nam/content-analysis/book258450) developed a variant of alpha for exactly the case where annotators segment a continuum themselves. It is the principled option when you want a single chance-corrected reliability figure for segmentation, at the cost of more setup. A practical middle path: report token-level kappa *and* span-level F1 together. The first tells you about label consistency, the second about boundary consistency, and the gap between them tells you which problem to fix. ## Doing it in Potato Potato computes [Krippendorff's alpha](/docs/features/quality-control) automatically for categorical schemes, but for a `span` scheme, the document-level number hides boundary disagreement, so measure at the level you actually care about. The reliable recipe is to have annotators overlap on a shared subset, export their spans, and compute pairwise F1 yourself under your chosen match rule. ```yaml annotation_schemes: - name: pii_spans annotation_type: span description: "Highlight every span that reveals personal information." labels: - name: person - name: location - name: org # Overlap a subset so agreement is measurable automatic_assignment: on: true instance_per_annotator: 100 labels_per_instance: 3 ``` The [export](/docs/features/export-formats) keeps each annotator's spans with their character offsets and labels, which is everything you need to compute exact-match or overlap F1 offline, and to split boundary agreement from type agreement. If your spans are typed, run the F1 twice, once ignoring type (boundary agreement) and once requiring type to match (full agreement). ## Further reading - [Inter-Annotator Agreement Explained](/docs/guides/inter-annotator-agreement), for the chance-corrected metrics that *do* apply to categorical tasks. - [Span Annotation](/docs/guides/span-annotation) and [Named Entity Recognition](/docs/guides/named-entity-recognition), for designing the task itself. - [Coreference Resolution](/docs/guides/coreference-resolution), where agreement has its own specialized metrics (MUC, B³, CEAF). - [Exporting Annotations for ML](/docs/guides/exporting-annotations-for-ml), for getting the spans out to compute F1. --- # Aggregating Crowd Labels: Beyond Majority Vote Section: Guides Source: https://www.potatoannotator.com/docs/guides/aggregating-crowd-labels > How to combine many noisy annotations into one label using annotator models like Dawid-Skene and MACE, when to trust them, and how Potato estimates competence and infers labels. **When several people label the same item, majority vote is the obvious way to combine their answers and usually the wrong one. Models that estimate each annotator's reliability recover better labels, flag spammers, and tell you how confident to be. But every one of them assumes a single correct answer, so on subjective tasks you have to decide first whether the disagreement is error to remove or signal to keep.** ## What majority vote misses Collect three labels for an item and take the majority. It works when annotators are roughly equal and mostly right. It breaks the moment they aren't. Majority vote counts a careful expert and a bot clicking randomly as one vote each, throws away the vote split (a 2-1 win and a 3-0 sweep come out the same), and gives you no way to tell a genuinely hard item from a lazy annotator. This is the [truth inference](https://en.wikipedia.org/wiki/Crowdsourcing) problem: recover the latent true label and each annotator's reliability at the same time, from nothing but the label matrix. ## Confusion-matrix models: Dawid and Skene The foundational method is nearly 50 years old. [Dawid and Skene (1979)](https://doi.org/10.2307/2346806) modeled each annotator with a confusion matrix, the probabilities that they label a true-positive item positive, negative, and so on, and used [expectation-maximization](https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm) to estimate those matrices and the true labels jointly. An annotator who confuses two categories gets a confusion matrix that says so, and their vote on that distinction is downweighted accordingly. Almost every modern aggregation model is a descendant of this idea. ## MACE: competence and spam detection [Hovy et al. (2013)](https://aclanthology.org/N13-1132/) introduced MACE (Multi-Annotator Competence Estimation), which adds an explicit spamming model: each annotator is treated as either *knowing* the answer or *guessing*, and MACE estimates the probability they were guessing on each item. That gives you a single competence score per annotator between 0 and 1, plus a per-item entropy that flags ambiguous items. It is fast, good at catching random clickers, and the model Potato ships. ## Bayesian models and the survey evidence The space has grown well past these two. [Paun et al. (2018)](https://aclanthology.org/Q18-1040/) compared a family of Bayesian annotation models on real datasets and found they consistently beat majority vote, especially when annotators vary a lot in quality, while also giving calibrated uncertainty you can propagate downstream. On the engineering side, [Zheng et al. (2017)](https://doi.org/10.14778/3055540.3055547) benchmarked 17 truth-inference methods and asked whether the problem is solved. No single method wins everywhere, but nearly all of them beat majority vote, and the gap grows as label quality drops. ## The assumption all of these share Every model above assumes there is one true label and disagreement is error. For objective tasks that is fine. For subjective ones it is wrong: on offensiveness, emotion, or moral judgment, two annotators can disagree because they genuinely read the text differently, and [Plank (2022)](https://aclanthology.org/2022.emnlp-main.731/) argues that this human label variation is often signal, not noise. Aggregate it away and you have thrown out the variation you were measuring. (We go deeper on this in [Disagreement is signal, not noise](/blog/disagreement-is-signal-not-noise).) This is where knowing *who* annotated starts to matter. [NUTMEG (Ivey, Gauch, and Jurgens, 2025)](https://arxiv.org/abs/2507.18890) is a Bayesian model built for this tension: it uses annotator background information to separate legitimate, systematic disagreement from noise, removing careless labels from training data while preserving the disagreement that tracks who the annotator is. That only works if you collected the background in the first place. If you run a prestudy demographics survey (see [collecting annotator demographics responsibly](/blog/collecting-annotator-demographics-responsibly) and Potato's [survey instruments](/docs/features/survey-instruments)), you have the annotator metadata that a NUTMEG-style model needs; without it, you are stuck treating every disagreement as either all error or all signal. ## Doing it in Potato Potato runs MACE over your multi-annotator data and reports competence and inferred labels in the admin dashboard. It works on categorical schemes (`radio`, `likert`, `select`, `multiselect`) and needs real overlap, several annotators per item, to have anything to estimate. ```yaml mace: enabled: true trigger_every_n: 10 # re-estimate after every 10 new annotations min_annotations_per_item: 3 # ignore items with fewer than 3 labels min_items: 5 # wait for at least 5 eligible items ``` After it runs, each annotator gets a competence score (near 1.0 is reliable, below 0.5 is likely a spammer) and each item gets a predicted label plus an entropy value. Low entropy means the model is confident; entropy near its maximum means no consensus, which usually flags a hard or under-specified item rather than a bad annotator. Full options are in the [MACE feature reference](/docs/features/mace). Two practical notes. First, MACE needs several labels per item, so plan the [overlap](/docs/guides/how-many-annotators) before the study rather than after. Second, MACE gives you a single label; if your task is subjective, consider keeping the distribution instead with a [`soft_label`](/docs/annotation-types/soft-label) scheme, and reach for adjudication only where you truly need one answer. ## When to aggregate and when to keep the spread A rough decision rule: - **Objective task, real answer key exists** → aggregate to one label. Use MACE or majority vote and move on. - **Objective-ish, but some annotators are unreliable** → aggregate with a competence model (MACE), not plain majority vote, so bad raters don't sway the result. - **Subjective task, disagreement is meaningful** → keep the full distribution (`soft_label`), and if you have annotator metadata, model the disagreement rather than deleting it. ## Further reading - [MACE Competence Estimation](/docs/features/mace), the feature reference with API endpoints and interpretation. - [Adjudication and Disagreement](/docs/guides/adjudication-and-disagreement), for resolving the cases you decide to collapse. - [Inter-Annotator Agreement Explained](/docs/guides/inter-annotator-agreement), for measuring how much your annotators diverge before you aggregate. - [How Many Annotators Do You Need?](/docs/guides/how-many-annotators), for the overlap that makes aggregation possible. --- # Statistical Power and Sample Size for Annotation Studies Section: Guides Source: https://www.potatoannotator.com/docs/guides/statistical-power-annotation > How many items you need for a result to mean something, why that is a different question from how many annotators per item, and how to avoid underpowered, over-claimed annotation and evaluation studies. **"How many annotators?" and "how many items?" are two different questions that get confused constantly. Annotator overlap controls how reliable each item's label is; the number of items controls whether a difference you observe is real or noise. A study can have five annotators per item and still be too small to support its conclusion.** This guide is about the second axis, statistical power, and how to keep an annotation or evaluation study from over-claiming. ## The two budgets Every annotation project spends effort along two independent axes, and it helps to name them separately: - **Overlap** (annotators per item): buys you label *reliability*, the confidence that a single item's label is right. This is the subject of [How Many Annotators Do You Need?](/docs/guides/how-many-annotators). - **Sample size** (number of items): buys you *statistical power*, the ability to detect a real difference between conditions, models, or groups. They trade against a fixed budget but solve different problems. Ten annotators labeling 50 items gives you very reliable labels for a sample too small to compare anything. One annotator labeling 5,000 items gives you noisy labels but enough of them to detect a real effect. Which one you need depends on the question you are asking. ## What statistical power is [Statistical power](https://en.wikipedia.org/wiki/Power_of_a_test) is the probability that your study detects an effect that is genuinely there. Low power means that even when model A really is better than model B, your experiment often fails to show it, and, less obviously, that the "significant" results you *do* get are more likely to be flukes with inflated effect sizes. The convention is to aim for 80% power, which requires deciding in advance the smallest effect worth detecting and sizing the study to catch it. This step gets skipped often. [Card et al. (2020)](https://aclanthology.org/2020.emnlp-main.745/) ran power analyses across common NLP setups and found that many published comparisons are badly underpowered: to reliably detect the small differences that typical papers claim, especially in human evaluation, you often need hundreds to thousands of items, far more than studies actually use. Their practical takeaway is to run the power calculation *before* collecting data, not to reverse-engineer significance after. ## Getting the significance test right Having enough items is necessary but not sufficient; you also have to test correctly. [Dror et al. (2018)](https://aclanthology.org/P18-1128/) is the standard reference here, and its advice is concrete: - **Match the test to the data.** NLP metrics are usually not normally distributed, so lean on nonparametric options, bootstrap and permutation tests, rather than assuming a t-test applies. - **Correct for multiple comparisons.** Testing many models, metrics, or subgroups inflates false positives; adjust (Bonferroni, or better, [Benjamini-Hochberg](https://en.wikipedia.org/wiki/False_discovery_rate)) when you run many tests. - **Report effect size and a confidence interval, not just a p-value.** With enough items, a difference can be statistically significant and practically meaningless. The effect size and interval tell the reader whether to care. ## A workable recipe 1. State the smallest difference that would matter (say, a 2-point difference in win rate). 2. Run a [power analysis](https://en.wikipedia.org/wiki/Power_of_a_test) for that effect at 80% power to get a target number of items. 3. Decide overlap separately, based on how subjective the labels are (see the [annotator-count guide](/docs/guides/how-many-annotators)). 4. After collection, use a bootstrap or permutation test, correct for the number of comparisons, and report effect sizes with intervals. The order matters: sizing the study after seeing the data is how underpowered results get dressed up as findings. ## Doing it in Potato Power is a design decision, not a config key, but Potato's job is to give you clean data to run the analysis on. Set overlap for reliability and instance counts for sample size in [task assignment](/docs/features/task-assignment): ```yaml automatic_assignment: on: true instance_per_annotator: 400 # sample size: items each annotator sees labels_per_instance: 3 # overlap: reliability per item ``` The two knobs are independent on purpose. The [export](/docs/features/export-formats) keeps every annotator's individual label with their ID and timestamp, which is what lets you bootstrap resample, by item and by annotator, when you compute significance offline. Preserving the per-annotator labels rather than only the aggregate is what makes a proper power-aware analysis possible; collapse to one gold label too early and you lose the variance the bootstrap needs. ## Further reading - [How Many Annotators Do You Need?](/docs/guides/how-many-annotators), the reliability side of the budget. - [Inter-Annotator Agreement Explained](/docs/guides/inter-annotator-agreement), for measuring reliability once you have overlap. - [Human Evaluation of Generated Text](/docs/guides/human-evaluation-generated-text), where underpowered comparisons are especially common. - [Exporting Annotations for ML](/docs/guides/exporting-annotations-for-ml), for getting per-annotator labels out to test. --- # Documenting Datasets and Models: Datasheets, Data Statements, and Model Cards Section: Guides Source: https://www.potatoannotator.com/docs/guides/documenting-datasets-and-models > A reference to the three standard documentation frameworks for annotated data and the models built on it, what each covers, when to use which, and how reproducibility reporting ties them together. **Three documentation standards have become the norm for machine-learning data: data statements and datasheets for the dataset, model cards for what you train on it. They overlap heavily and none is optional if you want the data trusted and reused. This guide is a reference to what each covers and when to reach for it.** For a narrative walk-through of writing one, see the companion post on [documenting your annotation dataset](/blog/documenting-annotation-datasets); this page is the standards comparison. ## The cost of undocumented data An annotated dataset without documentation ages badly. Six months on, nobody can tell how it was sampled, who labeled it, or what a label was supposed to mean, so the data becomes a black box people either trust blindly or throw away. Two specific costs recur: **irreproducibility** (you cannot rebuild the dataset or explain a discrepancy without the sampling method, guideline version, and annotator pool) and **hidden bias** (labels from an undocumented, narrow pool carry blind spots that stay invisible until they surface in production). The frameworks below exist to make the *who* and *how* legible before either bites. ## The three standards Each framework targets a different artifact and audience, but they were designed to interlock. **[Data statements](https://aclanthology.org/Q18-1041/) (Bender and Friedman, 2018)** are the NLP-specific schema. They characterize a *language* dataset, curation rationale, language variety and its speakers, annotator demographics, guidelines, and intended use, so a reader can judge how results will generalize and which populations the data underrepresents. Reach for a data statement when the data is text and language variety matters. **[Datasheets for datasets](https://doi.org/10.1145/3458723) (Gebru et al., 2021)** are the general-purpose version, borrowed from electronics, where every component ships with a datasheet. They ask a standard question set across motivation, composition, collection process, preprocessing, recommended uses, and maintenance. Use a datasheet for any ML dataset, text or not; it overlaps a data statement heavily, so on a language dataset you are really choosing which question set to organize around, not doing both from scratch. **[Model cards](https://doi.org/10.1145/3287560.3287596) (Mitchell et al., 2019)** document the *model*, not the data: its intended use, and, crucially, its performance broken down across demographic and other groups rather than as one aggregate number. A model card is where a fairness problem becomes visible. The three form a chain. A datasheet or data statement documents the data; a model card documents what was built on it; and the annotator-demographics section of the first is what makes the group-wise evaluation in the last interpretable. | Framework | Documents | Best for | Key sections | |---|---|---|---| | Data statement | A language dataset | NLP / text data | Curation rationale, language variety, speaker + annotator demographics, guidelines | | Datasheet | Any ML dataset | General ML data | Motivation, composition, collection, uses, maintenance | | Model card | A trained model | Any released model | Intended use, disaggregated evaluation, limitations | ## Reproducibility Documentation and reproducibility are the same goal from two angles. [Pineau et al. (2021)](https://jmlr.org/papers/v22/20-303.html) reported on the NeurIPS reproducibility program and distilled it into a [reproducibility checklist](https://en.wikipedia.org/wiki/Reproducibility): report the exact data, the collection and preprocessing steps, the evaluation setup, and enough detail to rerun the work. For an annotation project specifically, the reproducibility-critical facts are the ones a datasheet already asks for: how items were sampled, what guideline version was used, who annotated, and how [disagreement](/docs/guides/inter-annotator-agreement) was handled. If you can answer those, the dataset is both documented and reproducible; if you cannot, that is a gap to close before release, not after. ## A release checklist Before you publish, confirm you can answer: - How were items sampled, and from where? - What language variety is this, and who wrote the source text? - Who annotated it, how many people, and what is the pool's demographic makeup? - What guidelines did they follow, and which version? - Was disagreement aggregated to a gold label or kept as a distribution? Report [agreement](/docs/guides/inter-annotator-agreement) either way. - What is this dataset for, and what should it *not* be used for? ## Doing it in Potato Most of a dataset's documentation already exists as project artifacts, so you are not starting from a blank page. The config *is* documentation: the YAML records the schemes, label sets, and task structure, version-controlled next to the data, and the instructions you wrote are the guideline section verbatim. If you ran a prestudy phase, the [annotator demographics](/blog/collecting-annotator-demographics-responsibly) are already stored per annotator, aggregate them into distributions for the annotator-demographics section. And the [export](/docs/features/export-formats) keeps the annotator and timestamp on every label, so provenance travels with the data instead of getting stripped: ```json { "id": "doc_001", "annotations": { "sentiment": "positive" }, "annotator": "user_1", "timestamp": "2024-01-15T10:30:00Z" } ``` When you publish to the Hub, generate the dataset card as the last step of the export and fill its sections from the config, the guidelines, and the prestudy demographics you already have. ## Further reading - [Documenting Your Annotation Dataset](/blog/documenting-annotation-datasets), the narrative walk-through with a worked data statement. - [Collecting Annotator Demographics Responsibly](/blog/collecting-annotator-demographics-responsibly), for the annotator-demographics section done right. - [Writing Annotation Guidelines](/docs/guides/writing-annotation-guidelines), which double as the guideline section. - [Exporting Annotations for ML](/docs/guides/exporting-annotations-for-ml), for getting labels and metadata out cleanly. --- # How to Measure Inter-Annotator Agreement on Bounding Boxes Section: Guides Source: https://www.potatoannotator.com/docs/guides/measuring-agreement-on-bounding-boxes > Raw IoU is not agreement. How to measure whether annotators agree on bounding boxes, polygons and masks with chance correction, and why detection, classification and localization need separate numbers. **Mean [IoU](https://en.wikipedia.org/wiki/Jaccard_index) between annotators measures how easy the task is, not how much they agree. On a typical detection corpus it reads around 0.95 even for annotators who never looked at the image.** Measuring agreement on spatial labels means correcting for chance, and reporting detection, classification and localization as separate numbers. ## Why raw IoU misleads Consider a corpus where each image holds one large, centred object. Two annotators who both draw a box in the middle will overlap heavily, so mean IoU comes out around 0.95. So will an annotator who drew a box in the middle without looking. The number is measuring the task, not the annotators. That is the same problem [percent agreement](/docs/guides/inter-annotator-agreement) has on categorical labels, where one dominant class inflates the score, and it has the same fix: correct for how much agreement the task produces by accident. CVAT's consensus engine and V7's consensus stage both compare annotators with raw IoU against a per-class threshold. Neither applies chance correction, and both are widely used, so a reported "0.9 agreement" from either usually cannot be compared with anything. ## Why not Krippendorff's α over 1 − IoU The obvious fix is to feed IoU distance into [Krippendorff's α](https://en.wikipedia.org/wiki/Krippendorff%27s_alpha), which accepts any distance function. It does not work well here, and the reason is worth knowing before you build it. Braylan, Alonso and Lease ([WWW 2022](https://dl.acm.org/doi/10.1145/3485447.3512242)) evaluate distance functions by whether the resulting score **ranks annotator quality correctly**. For bounding boxes, α ranks plain L2 (0.687) above IoU (0.505) and GIoU (0.507), which inverts the ordering that both their distribution-based measures and practitioners give. Structurally: IoU distance is bounded in [0, 1] and saturates. Two randomly paired boxes almost always have IoU 0, so expected disagreement collapses to about 1, and α degenerates into `1 − mean distance` with no chance correction left in it. IoU is also flat where it matters: two boxes that do not overlap score 0 whether they are touching or at opposite corners, so there is no gradient exactly where annotators disagree. ## What to report instead ### Split the question three ways | Component | Question | Measure | |---|---|---| | Detection | Did they find the same objects at all? | Nominal α over presence | | Classification | Did they give matched objects the same label? | Nominal α over labels | | Localization | Are matched objects in the same place? | σ and KS | An annotator who finds every object and mislabels them all is a different problem from one who labels correctly but boxes loosely. A combined score cannot tell you which you have, and the two have different fixes. ### σ, with an empirical chance baseline ``` σ = 1 − mean(within-item distance) / mean(between-item distance) ``` This is α's `1 − D_o/D_e` form generalized to any distance, with the chance baseline estimated by comparing annotations of *different* items. σ = 0 means annotators agree no more than they would on unrelated images. Do not clamp negative values. "Further apart on the same image than on different images" is a real state, and it almost always means the guidelines are ambiguous rather than that the annotators are careless. ### KS alongside it The two-sample [Kolmogorov–Smirnov](https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test) statistic between the within-item and between-item distance distributions. σ compares two means and can be dragged by a few outliers; KS compares whole distributions. Report both. When they disagree, a minority of items is carrying most of the disagreement, which tells you to go and look at those items. ## Masks need a different tool For pixel masks, the question "whose boundary do we record" is not the same as "do they agree". Use [STAPLE](https://pubmed.ncbi.nlm.nih.gov/15250643/) (Warfield, Zou and Wells, 2004), which runs expectation-maximization over per-pixel labels and estimates a sensitivity and specificity per annotator. Sensitivity and specificity are reported separately because they have **opposite fixes**: low sensitivity means under-segmenting, low specificity means over-segmenting, and a single accuracy number scores both identically. STAPLE also beats a majority vote when careful annotators are outnumbered. With two careful annotators against three noisy ones, majority vote scores Dice 0.846 against truth and STAPLE scores 1.000. ## 3D boxes need exact rotated IoU For [cuboids](/docs/vision-spatial/point-clouds), use exact rotated 3D IoU. An axis-aligned approximation reports disagreement that is an artefact of the measure as soon as the data has real pitch and roll, which drone, handheld and indoor scans all do. ## Doing it in Potato ```yaml agreement_metrics: enabled: true annotation_schemes: - annotation_type: image_annotation name: objects description: "Draw a tight box around every vehicle." source_field: image tools: [bbox] labels: - {name: car, color: "#FF6B6B"} - {name: truck, color: "#4ECDC4"} num_annotators_per_item: default: 2 ``` Reports appear at `/admin/iaa`. Two properties are worth knowing about: - **Undefined values explain themselves.** α is genuinely undefined on a unanimous corpus. A bare `NaN` reads as a broken computation and a `1.0` would be a lie. - **Nothing is truncated silently.** Pairwise comparison is quadratic in annotators and in instances per item, so there is a budget. When it is hit, the item is skipped whole rather than half-measured, and the report says how many items were included. ## Checklist 1. Get at least two independent annotations per item. Agreement over a single annotator is undefined rather than perfect. 2. Report detection, classification and localization separately. 3. Correct for chance against an empirically estimated baseline rather than a fixed threshold. 4. Report σ and KS together. 5. Use STAPLE for masks and rotated 3D IoU for cuboids. 6. State coverage. A number computed over a sample that presents itself as complete will be quoted as complete. ## Further reading - [Agreement over geometry](/docs/measurement/geometry-agreement) - [Mask consensus with STAPLE](/docs/measurement/mask-consensus-staple) - [Inter-annotator agreement explained](/docs/guides/inter-annotator-agreement) — the categorical case - [Agreement for spans](/docs/guides/agreement-for-spans) - [Interactive IAA calculator](/tools/inter-annotator-agreement) --- # How to Tell Review from Rubber-Stamping in Pre-Labelled Data Section: Guides Source: https://www.potatoannotator.com/docs/guides/detecting-rubber-stamped-prelabels > When annotators accept model pre-labels without reading them, agreement rises while accuracy does not. Timing records separate review from rubber-stamping. **Pre-labelling makes annotators faster and rubber-stamping frictionless, and no measure computed from the annotations themselves can tell the two apart.** The geometry is identical, and inter-annotator agreement goes *up*, because annotators who all accept the same model output are unanimous by construction. The difference appears in the timing of the work. ## Why quality measures fail here Suppose a model pre-labels every image and three annotators accept nearly everything. - **Agreement** rises, because they all record the model's output. - **Gold-standard accuracy** rises wherever the model is right, which is most of the time. - **Adjudication volume** falls, because there is nothing to adjudicate. Every dashboard looks better. The dataset is now a record of a model agreeing with itself, and its errors are systematic rather than random — which is the worst kind, because they will not average out across annotators. Pre-labelling is a standard feature of current annotation platforms, including Label Studio's [pre-annotations](https://labelstud.io/guide/predictions), CVAT's [automated annotation](https://docs.cvat.ai/docs/annotation/auto-annotation/) and V7's [Auto-Annotate](https://docs.v7labs.com/docs/auto-annotate). Wherever it is turned on, chance-corrected agreement stops meaning what its interpretation says it means, because the coefficient was designed for independent judgements and an accepted pre-label is not one. ## What to record Timing, at the level of the individual accept: | Signal | What it tells you | |---|---| | **AI-accept latency** | Time from suggestion rendered to accepted. The single most useful number. | | Time per shape | Whether a drawn annotation took a plausible amount of time | | Revision count | Whether anything was corrected after being placed | | Stroke dynamics | Whether a mask was painted or accepted whole | | Zoom behaviour | Whether the annotator ever looked closely | You do not need coordinates for any of this, and you should not collect them. A process record that cannot reconstruct the annotation is much easier to justify to an ethics board, and it answers the question just as well. ## Reading the numbers > A human cannot inspect a mask boundary and decide in 300 ms. Once, maybe, if the object was obvious. As a **median across many items**, it is not fast expertise. Read a distribution, never a threshold on one item. Three rules that hold up: 1. **Compare each annotator to the project, not to a constant.** An easy corpus really is fast. Percentile calibration within your own project is the only defensible baseline. 2. **Look for runs.** Twenty consecutive sub-500 ms accepts is a much stronger signal than twenty scattered ones. 3. **Split by whether a suggestion was present.** An annotator's own drawing speed is the natural control for their accept speed. ## What this does not do It does not detect fraud, and it does not classify an annotator. A flag is a prompt to look, not a finding. Base rates matter and are easy to forget: in a corpus where 80% of items really are trivial, fast will usually be correct, and a rule tuned to catch rubber-stamping will mostly catch competence. That is why the thresholds belong to your project rather than to a shipped model. ## Doing it in Potato ```yaml annotation_telemetry: enabled: true fidelity: events idle_ms: 120000 ``` For free-text answers the equivalent is [keystroke logging](/docs/features/keystroke-logging), which distinguishes composed from transcribed from pasted text. It captures on `beforeinput` rather than `keydown`, because paste, drop, IME, dictation and autofill all change a field without firing a keydown — so a keydown-only logger is blind to exactly the cases it exists to catch. Neither ships a pre-trained classifier. There is no labelled corpus of rubber-stamping, and fitted coefficients presented as validated would be fabricated validation. ## Ethics Annotators see a recording notice by default, and turning disclosure off with `disclose_to_annotators: false` logs a warning. Be explicit in your consent language about what is recorded (timing and process), what is not (content and coordinates), how long it is kept, and what it will be used for. See [keystroke logging ethics](/docs/features/keystroke-logging-ethics) for sample consent language and the retention questions. ## Further reading - [Drawing telemetry](/docs/measurement/drawing-telemetry) - [Keystroke logging](/docs/features/keystroke-logging) - [Gold standards and attention checks](/docs/guides/gold-standards-and-attention-checks) - [LLM pre-annotation](/docs/guides/llm-pre-annotation) --- # LLM and Vision Pre-Annotation Section: Guides Source: https://www.potatoannotator.com/docs/guides/llm-pre-annotation > How to speed up annotation with LLM pre-labeling and human verification, in-context learning, option highlighting, and vision pre-annotation, using Potato's AI support. **Pre-annotation uses a model to propose labels that a human then verifies or corrects. Checking a good suggestion is far faster than labeling from scratch, so pre-annotation can cut annotation time substantially, as long as you keep a human in the loop.** This is [human-in-the-loop](https://en.wikipedia.org/wiki/Human-in-the-loop) machine learning. Potato has built-in AI support for [OpenAI, Claude, Gemini, Ollama, and others](/docs/features/ai-support). ## How pre-annotation works 1. A model (an LLM, or a vision model for images) predicts a label for each item. 2. The prediction is shown to the annotator as a pre-filled suggestion or a highlighted option. 3. The annotator confirms or fixes it. 4. The verified label, not the raw model output, becomes your data. ## Turning it on ```yaml ai_support: enabled: true endpoint_type: openai # or anthropic, gemini, ollama, ... ai_config: model: gpt-4 api_key: ${OPENAI_API_KEY} temperature: 0.3 ``` Potato offers a few flavors: - **[In-context learning labeling](/docs/features/icl-labeling)**: the model labels items from a few examples in the prompt; the human verifies. - **[Option highlighting](/docs/features/option-highlighting)**: the model pre-selects the labels it thinks are most likely, so the annotator confirms rather than searches. - **[Visual AI support](/docs/features/visual-ai-support)**: vision models (GPT-4V, Claude, Gemini, or a detector like YOLO) propose image labels and boxes. ## The risk: automation bias The danger of pre-annotation is [automation bias](https://en.wikipedia.org/wiki/Automation_bias), annotators rubber-stamp the model's suggestions, importing its errors into your "gold" data. Guard against it: - Keep [gold standards](/docs/guides/gold-standards-and-attention-checks) running so you can detect blind acceptance. - Don't pre-fill on the items you use to measure [agreement](/docs/guides/inter-annotator-agreement); measure on un-suggested items. - Use lower-confidence suggestions as *hints*, not defaults, for hard cases. ## Pre-annotation vs. active learning Pre-annotation makes each label faster. [Active learning](/docs/guides/active-learning) makes each label more *valuable* by choosing which items to label next. They combine well. ## Further reading - [LLM Annotators vs Humans](/blog/llm-annotators-vs-humans), on deciding when to automate a labeling job. - [Codebooks for AI Annotators](/blog/codebooks-for-ai-annotators), on turning a coding scheme into a model an LLM can run. - [AI Support feature reference](/docs/features/ai-support) - [Active Learning for Annotation](/docs/guides/active-learning) - [Solo Mode](/docs/features/solo-mode), a guided human-plus-LLM workflow --- # Active Learning for Annotation Section: Guides Source: https://www.potatoannotator.com/docs/guides/active-learning > What active learning is, when it helps, and which query strategies Potato supports (uncertainty, diversity, BADGE, BALD), so you label fewer items for the same model quality. **Active learning chooses *which* items to annotate next so a model reaches the same accuracy with far fewer labels. Instead of labeling at random, you label the items the model finds most informative.** See [active learning](https://en.wikipedia.org/wiki/Active_learning_(machine_learning)) for background. For the feature reference, see [Active Learning](/docs/features/active-learning). ## The loop 1. Label a small seed set. 2. Train a quick model on what you have. 3. Score the unlabeled pool and pick the most informative items. 4. Annotate those, add them, retrain. Repeat. Each round spends the annotation budget where the model learns most. ## Query strategies Potato supports - **Uncertainty sampling**: pick items the model is least confident about (near the decision boundary). The simplest default, and often good enough. - **Diversity sampling**: pick items that are different from each other, so you don't waste budget on near-duplicates. - **[BADGE](https://arxiv.org/abs/1906.03671)**: combines uncertainty and diversity using gradient embeddings. - **[BALD](https://arxiv.org/abs/1112.5745)**: Bayesian strategy that selects items expected to most reduce model uncertainty. - **Hybrid**: blends strategies. ```yaml active_learning: enabled: true schema_names: [sentiment] query_strategy: uncertainty # or diversity, badge, bald, hybrid min_instances_for_training: 20 ``` ## When it helps It helps when labels are expensive, the pool is large, and a useful model can be trained on a small seed. It helps *less* when: - The task is so easy that random labeling already saturates quickly. - You need an unbiased held-out test set. Active learning skews what gets labeled, so keep the evaluation data randomly sampled. - Labels are cheap relative to engineering effort. ## Further reading - [Active Learning feature reference](/docs/features/active-learning) - [LLM and Vision Pre-Annotation](/docs/guides/llm-pre-annotation) - [Diversity Ordering](/docs/features/diversity-ordering) --- # Collecting RLHF and Preference Data Section: Guides Source: https://www.potatoannotator.com/docs/guides/rlhf-preference-data > How to collect human preference data for RLHF and model alignment, pairwise comparisons, rubric scoring, and justifications, with Potato. **Reinforcement learning from human feedback (RLHF) trains models to match human preferences. The core data is human judgments comparing model outputs, most often "which of these two responses is better?".** Collecting that data well is an annotation problem, and Potato handles it directly. See [reinforcement learning from human feedback](https://en.wikipedia.org/wiki/Reinforcement_learning_from_human_feedback) for background. ## The standard recipe: pairwise preference Show a prompt and two candidate responses; the annotator picks the better one. These judgments train a [reward model](https://en.wikipedia.org/wiki/Reward_model) that scores outputs, which then guides the policy model. [HH-RLHF](/showcase/hh-rlhf-pairwise-preference) is the canonical dataset in this exact shape. ```yaml annotation_schemes: - annotation_type: pairwise name: preference description: "Which response better follows the instruction and is more helpful and harmless?" mode: binary allow_tie: true - annotation_type: text name: rationale description: "One sentence on why you chose it." label_requirement: required: false ``` A short rationale is worth collecting: it lets you audit the preference data and find cases where annotators optimized the wrong thing (length, formatting) instead of quality. See [Pairwise and Best–Worst Scaling](/docs/guides/pairwise-and-best-worst) for the comparison mechanics and [Pairwise Model Comparison](/docs/guides/pairwise-model-comparison) for evaluating models head-to-head. ## Multi-dimensional preferences A single "better" judgment hides trade-offs. To collect signal on *why* one output wins, score several criteria with a rubric: ```yaml annotation_schemes: - annotation_type: rubric_eval name: quality description: "Rate the response on each dimension." scale_points: 5 criteria: - {name: Helpfulness, description: "Does it actually answer the request?"} - {name: Harmlessness, description: "Is it safe and appropriate?"} - {name: Honesty, description: "Is it accurate and non-misleading?"} ``` See [Rubric-Based LLM Evaluation](/docs/guides/rubric-based-llm-evaluation). [UltraFeedback](/showcase/ultrafeedback-rubric-evaluation) is a large preference dataset collected this way, with a separate score per criterion. ## Quality pitfalls specific to preference data - **Length and style bias.** Annotators often prefer longer or more confident answers regardless of correctness. Name this in the guidelines and watch for it. - **Position bias.** Randomize which response is shown as "A". - **Calibration drift.** Re-share anchor examples periodically so standards don't drift across a long campaign. - **Agreement.** Preference is subjective; collect overlap and track [agreement](/docs/guides/inter-annotator-agreement). ## Further reading - [Pairwise and Best–Worst Scaling](/docs/guides/pairwise-and-best-worst) - [Rubric-Based LLM Evaluation](/docs/guides/rubric-based-llm-evaluation) - [Evaluating AI Agents](/docs/guides/evaluating-ai-agents) --- # How to Evaluate AI Agents Section: Guides Source: https://www.potatoannotator.com/docs/guides/evaluating-ai-agents > An overview of evaluating AI agents and LLMs with human annotation, trajectory, step, span, and comparison-level evaluation, and which Potato tool fits each. **Evaluating an AI agent means judging not just its final answer but the *path* it took, the reasoning, tool calls, and actions along the way. Human annotation is still the gold standard for this, because many agent failures (a plausible-but-wrong step, an unsafe action) only a person can reliably catch.** Potato is an open-source tool for human annotation of LLM agent trajectories, with a purpose-built display for each level of evaluation. An [AI agent](https://en.wikipedia.org/wiki/Intelligent_agent) here means an LLM-driven system that takes multi-step actions, calling tools, browsing, or writing code, to complete a task. See the [agent evaluation overview](/agent-evaluation) and the [Agentic Annotation](/docs/features/agentic-annotation) reference. ## What are the levels of AI agent evaluation? Pick the level that matches the question you're asking: - **Trajectory level**: judge the whole run. Did it succeed? Was it efficient and safe? See [Annotating Agent Trajectories](/docs/guides/agent-trajectory-annotation). - **Step level**: judge each action. Was this tool call correct? Was this step necessary? This is the data behind [process reward models](/docs/guides/process-reward-models). - **Span level**: highlight specific problems inside outputs, such as a hallucinated claim or an unsafe instruction. See [Detecting Hallucinations](/docs/guides/detecting-hallucinations). - **Comparison level**: judge two agents or two runs head-to-head. See [Pairwise Model Comparison](/docs/guides/pairwise-model-comparison). - **Team level**: for multi-agent systems, attribute a failure to the responsible agent, step, and handoff. See [How to Evaluate Multi-Agent Systems](/docs/guides/evaluating-multi-agent-systems). ## What agent trace formats does Potato support? Potato reads agent traces from 15 formats, including OpenAI and Anthropic tool calls, [ReAct](https://arxiv.org/abs/2210.03629), LangChain, LangFuse, WebArena, SWE-bench, MCP, and OpenTelemetry, and renders them in displays tuned for the kind of agent: - **Agent trace display** for reasoning/tool traces. - **Web agent display** with screenshots and action overlays, see [Web-Agent Evaluation](/docs/guides/web-agent-evaluation). - **Coding trace display** with diffs and terminal output, see [Coding-Agent Evaluation](/docs/guides/coding-agent-evaluation). - **Live agent display** to watch and steer an agent in real time, see [Live Agent Evaluation](/docs/guides/live-agent-evaluation). - **Multimodal agent displays** for computer-use, voice, and video agents, see [Evaluating Computer-Use and Multimodal Agents](/docs/guides/evaluating-computer-use-agents). ## Which agent evaluation method should I choose? | Your question | Approach | |---|---| | "Did the agent complete the task?" | Trajectory success label | | "Where exactly did it go wrong?" | Step-level error taxonomy | | "Which version is better?" | Pairwise comparison | | "How good is it on several axes?" | Rubric evaluation | | "Is the retrieved-context answer faithful?" | RAG evaluation | | "Which agent in the team caused the failure?" | Multi-agent attribution | | "Did the computer-use agent click the right thing?" | GUI trajectory review | ## Further reading - [Agent evaluation overview](/agent-evaluation) - [Annotating Agent Trajectories](/docs/guides/agent-trajectory-annotation) - [How to Evaluate Multi-Agent Systems](/docs/guides/evaluating-multi-agent-systems) - [Evaluating Computer-Use and Multimodal Agents](/docs/guides/evaluating-computer-use-agents) - [Rubric-Based LLM Evaluation](/docs/guides/rubric-based-llm-evaluation) - [AI Agent Evaluation Tools Compared](/docs/guides/agent-evaluation-tools-compared) - [How to Measure Agreement Between an LLM Judge and Human Annotators](/docs/guides/llm-judge-human-agreement) --- # Annotating Agent Trajectories Section: Guides Source: https://www.potatoannotator.com/docs/guides/agent-trajectory-annotation > How to annotate AI agent trajectories step by step, error taxonomies, severity scoring, and trajectory-level success, using Potato's trajectory evaluation. **A trajectory is the full sequence of steps an agent took, its thoughts, tool calls, and observations. Annotating a trajectory means judging the run as a whole and marking where individual steps went wrong, with a category and a severity for each error.** Potato annotates agent trajectories step by step with custom rubrics, free and self-hosted. Those labels are the input to reward models and to targeted debugging. For the feature reference, see [Agentic Annotation](/docs/features/agentic-annotation). ## What do you collect when annotating a trajectory? - **Overall outcome**: success, partial success, or failure. - **Per-step judgments**: for each step, was it correct, unnecessary, or wrong? - **Error categories**: *why* a step was wrong (wrong tool, bad arguments, hallucination, looping, unsafe action…). - **Severity**: how bad each error was, often weighted into a score. ## How do I set up trajectory evaluation in Potato? Potato's `trajectory_eval` type renders each step as a card and attaches a per-step error taxonomy with severity weights: ```yaml annotation_schemes: - annotation_type: trajectory_eval name: step_evaluation description: "Evaluate each step for correctness and mark any errors." steps_key: steps error_types: - {name: reasoning, subtypes: [logical_error, factual_error, planning_error]} - {name: execution, subtypes: [wrong_tool, wrong_args, api_error]} - {name: safety, subtypes: [harmful_action, data_leak, scope_violation]} severities: - {name: minor, weight: -1} - {name: major, weight: -5} - {name: critical, weight: -10} show_score: true ``` The severity weights roll up into a trajectory score, so you can rank runs and track regressions across model versions. ## How do I design an agent error taxonomy? Keep the taxonomy small, close to exhaustive, and mutually exclusive. A practical starting set: - **Reasoning errors**: wrong conclusion, ignored evidence, bad plan. - **Execution errors**: wrong tool, malformed call, mishandled result. - **Safety errors**: unsafe action, out-of-scope behavior, data exposure. Add a free-text "other" so annotators aren't forced to misfile novel failures, then promote recurring "other" notes into named categories. ## Quality considerations - Agreement on *step correctness* is usually high; agreement on *error category* is lower. Measure both; see [Inter-Annotator Agreement](/docs/guides/inter-annotator-agreement). - Long trajectories are fatiguing; cap length or paginate. - The "first wrong step" is often what matters most for training; see [Process Reward Models](/docs/guides/process-reward-models). ## Further reading - [How to Get Reliable Labels on Agent Trajectories](/blog/annotating-agent-trajectories-reliably), the longer treatment of taxonomy design, step-level agreement, and adjudication. - [How to Evaluate AI Agents](/docs/guides/evaluating-ai-agents) - [Process Reward Models](/docs/guides/process-reward-models) - [Evaluating Tool Use](/docs/guides/tool-use-evaluation) --- # Process Reward Models and Step-Level Labeling Section: Guides Source: https://www.potatoannotator.com/docs/guides/process-reward-models > How to collect process reward (PRM) data by labeling agent steps as correct or incorrect, first-error and per-step modes, with Potato. **A process reward model (PRM) scores the *reasoning steps* an agent takes, not just its final answer. Training one needs step-level labels: for each step in a trajectory, was it correct?** This data is what lets a model learn to reason well, not just to land on the right answer by luck. PRMs contrast with outcome reward models, which score only the final result. Labeling at the step level catches the case where a model reaches the right answer through flawed reasoning. For the feature reference see [Process Reward Annotation](/docs/features/process-reward-annotation). ## Two labeling modes Potato's `process_reward` type supports the two standard schemes: - **First-error mode**: the annotator marks the first step that goes wrong; every step after it is automatically treated as compromised. Fast, and well-matched to how reasoning failures cascade. - **Per-step mode**: the annotator judges every step independently as correct or incorrect. More granular, more effort. ```yaml annotation_schemes: - annotation_type: process_reward name: step_rewards description: "Mark the first incorrect step. Steps after it are flagged automatically." steps_key: structured_turns mode: first_error ``` The colors make the cascade visible: green steps are good, the red step is the first error, and orange marks the now-suspect downstream steps. ## When to use which mode - **First-error** for math, coding, and chained reasoning where one mistake invalidates the rest. Cheaper and usually sufficient. - **Per-step** when steps are independent, or when you need a dense reward signal for every step. ## Quality considerations - Define "correct step" precisely: correct *and* useful, or merely not-wrong? A redundant-but-harmless step needs a rule. - Reasoning is subjective at the margins, collect overlap on a sample and check [agreement](/docs/guides/inter-annotator-agreement). - Pair with a trajectory-level outcome label so you can study where good outcomes hide bad reasoning. See [Annotating Agent Trajectories](/docs/guides/agent-trajectory-annotation). ## Further reading - [How to Get Reliable Labels on Agent Trajectories](/blog/annotating-agent-trajectories-reliably), on taxonomy design and step-level agreement. - [Process Reward Annotation feature reference](/docs/features/process-reward-annotation) - [Annotating Agent Trajectories](/docs/guides/agent-trajectory-annotation) - [Collecting RLHF and Preference Data](/docs/guides/rlhf-preference-data) --- # Evaluating Tool Use and Function Calling Section: Guides Source: https://www.potatoannotator.com/docs/guides/tool-use-evaluation > How to annotate and evaluate an agent's tool calls and function calling across trace formats (OpenAI, Anthropic, ReAct, LangChain) with Potato per-turn ratings. **When an agent calls tools, search, a calculator, an API, a database, each call is a decision you can evaluate: was this the right tool? were the arguments correct? was the result used properly?** Tool-use evaluation turns those decisions into per-step labels over an agent's trace. This is the human-judgment complement to automated [function calling](https://en.wikipedia.org/wiki/Function_(computer_programming)) benchmarks: a call can be syntactically valid yet wrong for the task. ## What to judge at each tool call - **Tool selection**: was this the appropriate tool, or should it have used another (or none)? - **Arguments**: were the parameters correct and complete? - **Necessity**: was the call needed, or redundant? - **Result handling**: did the agent correctly interpret and use the output? ## Reading traces from any framework Potato converts 15 trace formats into a common step view, so you can evaluate tool use regardless of how the agent was built: OpenAI and Anthropic tool/function calls, [ReAct](https://arxiv.org/abs/2210.03629) thought-action-observation traces, LangChain, LangFuse, and more. See [Agentic Annotation](/docs/features/agentic-annotation). ## Per-step rating setup Attach a rating to each step (each tool call) with a conditional follow-up for the failures: ```yaml annotation_schemes: - annotation_type: trajectory_eval name: tool_call_correctness description: "For each tool call, judge whether it was the right call." steps_key: agentic_steps correctness_options: ["Correct", "Wrong tool", "Wrong arguments", "Unnecessary"] - annotation_type: text name: notes description: "If not correct, what should it have done?" label_requirement: required: false ``` ## Quality considerations - Show the tool's *output*, not just the call, annotators can't judge result handling otherwise. - Pretty-print JSON arguments and responses so they're readable (Potato does this in the agent trace display). - Distinguish "wrong tool" from "right tool, wrong arguments", they point to different model fixes. ## Further reading - [How to Get Reliable Labels on Agent Trajectories](/blog/annotating-agent-trajectories-reliably), on measuring agreement across multi-step traces. - [How to Evaluate AI Agents](/docs/guides/evaluating-ai-agents) - [Annotating Agent Trajectories](/docs/guides/agent-trajectory-annotation) - [Agentic Annotation feature reference](/docs/features/agentic-annotation) --- # Web-Agent Evaluation Section: Guides Source: https://www.potatoannotator.com/docs/guides/web-agent-evaluation > How to evaluate web-browsing agents with screenshots and action overlays, per-step web action correctness, using Potato's web agent display. **A web agent completes tasks by browsing, clicking, typing, scrolling across pages. Evaluating one means looking at what it saw (the screenshot) and what it did (the action) at each step, and judging whether that action was right.** Potato renders the screenshots with visual overlays of each action so annotators can review a browsing session like a filmstrip. This is the human-evaluation counterpart to benchmarks like [WebArena](https://webarena.dev/) and Mind2Web. See [Web Agent Annotation](/docs/features/web-agent-annotation). ## What the annotator sees Potato's web agent display shows, for each step: - the **screenshot** of the page at that moment, - an **overlay** marking the action, a circle where it clicked, a box on the field it typed into, an arrow for a scroll, - the **action description** and any target element, - a **filmstrip** to move between steps. ## What to judge per step - **Right target?** Did it click/​type on the correct element? - **Right action type?** Click vs. type vs. scroll vs. navigate. - **Progress?** Did the step move the task forward or waste a turn? ```yaml annotation_schemes: - annotation_type: trajectory_eval name: web_action_correctness description: "Judge each browsing action against the task." steps_key: agentic_steps correctness_options: ["Correct", "Wrong target", "Wrong action", "No progress"] ``` ## Setting up the display Point Potato at a web-agent trace (screenshots plus actions) and enable the web agent display. Traces can come from WebArena/VisualWebArena exports or your own runs in HAR-plus-screenshot form. See [Web Agent Annotation](/docs/features/web-agent-annotation) for the trace schema. ## Quality considerations - Screenshots must be legible, set a sensible max width and keep overlays from hiding the target. - Long sessions fatigue annotators; the filmstrip and step numbers help them keep place. - For overall task success, add a trajectory-level label on top of the per-step ratings. See [Annotating Agent Trajectories](/docs/guides/agent-trajectory-annotation). ## Further reading - [Web Agent Annotation feature reference](/docs/features/web-agent-annotation) - [Live Agent Evaluation](/docs/guides/live-agent-evaluation) - [How to Evaluate AI Agents](/docs/guides/evaluating-ai-agents) --- # Coding-Agent Evaluation Section: Guides Source: https://www.potatoannotator.com/docs/guides/coding-agent-evaluation > How to evaluate coding agents, reviewing diffs, terminal output, and SWE-bench/Aider/Claude Code traces, with Potato's coding trace display. **A coding agent edits files, runs commands, and reads output to solve a programming task. Evaluating one is like reviewing a pull request that also includes the terminal session: you judge the code changes and the steps that produced them.** Annotate SWE-agent, Aider, and Claude Code trajectories with Potato, a free self-hosted tool that renders diffs and terminal blocks for review. This pairs with automated benchmarks such as [SWE-bench](https://www.swebench.com/); human review catches the plausible-but-wrong patch that passes a weak test. ## What does the annotator review in a coding-agent run? - **Diffs**: color-coded unified diffs of each file change, with line numbers and a file-tree sidebar. - **Commands and output**: terminal blocks showing what the agent ran and what came back. - **Reasoning**: the agent's thoughts between actions. Potato reads coding-agent traces including SWE-bench, [Aider](https://aider.chat/), and Claude Code formats. See [Coding Agent Annotation](/docs/features/coding-agent-annotation) and [Code Review Annotation](/docs/features/code-review-annotation). ## What should I judge in a coding-agent run? - **Correctness**: does the change solve the task without breaking other things? - **Step quality**: was each edit/command sensible, or flailing? - **Efficiency**: did it take a reasonable path? ```yaml annotation_schemes: - annotation_type: trajectory_eval name: step_correctness description: "Judge each edit or command." steps_key: agentic_steps correctness_options: ["Correct", "Partially correct", "Incorrect", "Unnecessary"] - annotation_type: radio name: overall description: "Does the final change solve the task?" labels: [Solved, Partially solved, Not solved] ``` ## How do I keep coding-agent evaluation reliable? - Give annotators the task description and the repository context; a diff is meaningless without the goal. - For first-error reasoning chains, see [Process Reward Models](/docs/guides/process-reward-models). - To watch an agent code in real time rather than review a recording, see [Live Agent Evaluation](/docs/guides/live-agent-evaluation). ## Further reading - [Coding Agent Annotation feature reference](/docs/features/coding-agent-annotation) - [Code Review Annotation](/docs/features/code-review-annotation) - [How to Evaluate AI Agents](/docs/guides/evaluating-ai-agents) --- # How to Evaluate Multi-Agent Systems Section: Guides Source: https://www.potatoannotator.com/docs/guides/evaluating-multi-agent-systems > A practical guide to evaluating multi-agent LLM systems, attributing failures to the responsible agent and handoff, reviewing the interaction graph, and scoring each agent and the team. **A multi-agent system is several LLM agents (a planner, a coder, a reviewer, and so on) cooperating on one task. Evaluating it means more than scoring the final answer, because the failures that matter happen between agents: a dropped constraint at a handoff, the wrong agent taking over, a team that never verifies its work. The unit of judgment is which agent, which step, and which handoff.** Potato is an open-source tool for human evaluation of multi-agent runs, with a [purpose-built set of annotation surfaces](/docs/agent-evaluation/multi-agent-evaluation) for team structure. A [multi-agent system](https://en.wikipedia.org/wiki/Multi-agent_system) here means an [LLM](https://en.wikipedia.org/wiki/Large_language_model)-driven workflow where distinct agents, each with a role, exchange messages and hand off control. Research on why these systems fail (the [MAST taxonomy](https://arxiv.org/abs/2503.13657), *Why Do Multi-Agent LLM Systems Fail?*) finds that a large share of failures are inter-agent: specification problems, misalignment between agents, and missing verification. A flat transcript hides those. ## Why isn't single-agent evaluation enough? When you evaluate one agent, you judge a single sequence of thoughts, tool calls, and observations. A team adds failure modes that only exist between agents: - **Handoff loss**: agent A knows a constraint that agent B never receives. - **Misattribution**: the run fails, but the responsible agent is upstream of where the error surfaced. - **Coordination failure**: each agent is individually competent, yet the team loops, stalls, or never verifies. - **Resource contention**: two agents touch the same tool or file at once and deadlock. Scoring only the final output tells you *that* the team failed, not *where*. Attribution is what makes the data useful for debugging or training. ## How do I attribute a multi-agent failure? The failure-attribution literature (Zhang et al., *Which Agent Causes Task Failures and When?*, ICML 2025) frames the label as a triple: the **responsible agent**, the **decisive step**, and a **reason**. In Potato the `failure_attribution` schema populates the agent and step pickers from the trace itself, so the annotator chooses from agents and steps that actually occurred: ```yaml annotation_schemes: - annotation_type: radio name: outcome description: "Did the system succeed?" labels: [success, failure] - annotation_type: failure_attribution name: attribution description: "If it failed: which agent, which step, and why?" steps_key: steps agent_key: agent ``` Pairing the outcome scheme with attribution means the triple is only collected on runs that actually failed. ## How do I review the team structure, not just the transcript? Two surfaces make the structure visible. The [interaction graph](/docs/agent-evaluation/multi-agent-evaluation#interaction-graph-agent_interaction_graph) renders agents as nodes and handoffs as edges, and the annotator marks the critical path and flags problematic edges. [Handoff review](/docs/agent-evaluation/multi-agent-evaluation#handoff-review-handoff_review) turns every control transfer into a card to flag misalignment and rate quality: ```yaml annotation_schemes: - annotation_type: handoff_review name: handoffs description: "For each handoff: flag any misalignment and rate the quality." steps_key: steps agent_key: agent flags: [info_loss, dropped_constraint, garbling, goal_drift] quality_scale: 5 ``` For scoring, the [`agent_scorecard`](/docs/agent-evaluation/multi-agent-evaluation#per-agent-and-per-team-scorecard-agent_scorecard) rates each agent on role fidelity, contribution, and coordination, and scores the team on its own dimensions, so a strong individual agent inside a poorly coordinated team is visible in the numbers. ## Which method should I use? - **Debugging a pipeline**: start with the interaction graph and failure attribution to localize where runs break. - **Comparing orchestration patterns**: add the scorecard to score sequential vs. hierarchical vs. group-chat designs on the same tasks. - **Building training or reward data**: tag failures at step granularity with the MAST modes (via [`trajectory_eval`](/docs/guides/agent-trajectory-annotation)) so the labels attach to the acting agent and step. - **Concurrency bugs**: use the tool-contention timeline to catch deadlocks and races a transcript cannot show. Measure agreement on attribution the same way you would for any subjective label; see [Inter-Annotator Agreement](/docs/guides/inter-annotator-agreement). ## Further reading - [Multi-Agent Team Evaluation](/docs/agent-evaluation/multi-agent-evaluation) — the full schema reference - [How to Evaluate AI Agents](/docs/guides/evaluating-ai-agents) — the levels of agent evaluation - [Annotating Agent Trajectories](/docs/guides/agent-trajectory-annotation) — per-step error taxonomies - [Evaluating Computer-Use and Multimodal Agents](/docs/guides/evaluating-computer-use-agents) --- # Rubric-Based LLM Evaluation Section: Guides Source: https://www.potatoannotator.com/docs/guides/rubric-based-llm-evaluation > How to score LLM outputs on several defined criteria at once with Potato's rubric_eval grid, MT-Bench style, and how to write criteria that raise agreement. **A rubric breaks a vague judgment ("is this a good answer?") into specific scored criteria (helpfulness, accuracy, completeness, tone, safety), each rated on a scale. It makes LLM evaluation repeatable and reveals *why* one answer beats another.** Benchmarks such as [MT-Bench](https://arxiv.org/abs/2306.05685) score model answers this way. A [rubric](https://en.wikipedia.org/wiki/Rubric_(academic)) turns subjective quality into a grid of defined criteria and scale points, which raises agreement and makes results interpretable. ## When to use a rubric - The output is rich enough that a single score loses information. - You need to know which dimension is weak (accuracy vs. tone), rather than an overall verdict alone. - You want criteria stakeholders agree on up front. If you only need "which is better", a [pairwise comparison](/docs/guides/pairwise-model-comparison) is cheaper. Rubrics fit better when you need an absolute, multi-dimensional profile. ## Setting it up in Potato `rubric_eval` carries the criteria and the scale in one scheme, so an annotator fills the whole grid in a single pass instead of answering a separate question per criterion. Naming the scale points as well as the criteria is what stops two annotators from reading the same number differently: ```yaml annotation_schemes: - annotation_type: rubric_eval name: answer_quality description: "Rate the answer on each criterion." scale_points: 5 scale_labels: ["Poor", "Below average", "Average", "Good", "Excellent"] criteria: - {name: Helpfulness, description: "Does it address the user's actual need?"} - {name: Accuracy, description: "Is it factually correct?"} - {name: Completeness, description: "Does it cover the important points?"} - {name: Tone, description: "Is the style appropriate?"} ``` Potato renders the scheme as a table with one row per criterion and one column per scale point. Annotators pick one cell in every row, and `show_overall: true` adds a summary row at the bottom when you want a single number alongside the breakdown. ## Writing good criteria - **Make them independent.** Overlapping criteria ("helpful" and "useful") get scored together and add noise. - **Anchor the scale.** Describe what a 1 and a 5 look like for each criterion, rather than only the ends of the scale as a whole. - **Keep it short.** Four to six criteria is a common range, and long rubrics fatigue annotators and lower agreement. ## Rubrics and LLM-as-judge The same rubric you give humans can prompt an "LLM judge" for cheap pre-scoring, then have humans verify, as in [LLM pre-annotation](/docs/guides/llm-pre-annotation). Keep a human-scored sample to check the judge against, and watch for the judge's own biases (length, formatting, self-preference). ## Further reading - [Pairwise Model Comparison](/docs/guides/pairwise-model-comparison) - [Collecting RLHF and Preference Data](/docs/guides/rlhf-preference-data) - [Rating Scales](/docs/guides/rating-scales) - [FLASK Skill-based Rubric Evaluation](/showcase/flask-skill-rubric-evaluation), a runnable `rubric_eval` design that scores model responses on alignment skills --- # How to Measure Agreement Between an LLM Judge and Human Annotators Section: Guides Source: https://www.potatoannotator.com/docs/guides/llm-judge-human-agreement > Check an LLM-as-a-judge against human labels: compare judge-human kappa with human-human kappa, measure calibration with ECE and Brier score, and test for position bias. **To decide whether to trust an LLM judge, have at least two people label a sample of items without seeing the judge's answers, then compare the judge's agreement with those people to the people's agreement with each other. A judge whose Cohen's kappa with the humans is close to the humans' kappa with each other is doing about as well as another annotator. One well below it is not.** An [LLM-as-a-judge](https://huggingface.co/learn/cookbook/en/llm_judge) is a language model prompted to grade outputs, usually against a rubric. [Cohen's kappa](https://en.wikipedia.org/wiki/Cohen%27s_kappa) measures agreement between two raters, corrected for the agreement expected by chance. A judge is [calibrated](https://en.wikipedia.org/wiki/Calibration_(statistics)) when its confidence matches how often it is right: of the verdicts it gives with 80% confidence, about 80% should be correct. ## Why accuracy against one person is not enough The usual first check is accuracy. You label a few hundred items yourself and count how often the judge matches. That number has two problems. It treats one person's labels as the truth, mistakes included, and it is not corrected for chance, so a judge that calls every response "helpful" scores well on a dataset where most responses are helpful. Kappa fixes the second problem. A second human annotator fixes the first, because the disagreement between two careful people shows how much the task itself allows, and a judge cannot be held to more than that. ## Compare the judge with the humans' agreement with each other Report three kinds of pair: human with human, judge with human, and judge with judge when you are comparing several judges. If two people agree at κ = 0.62, a judge at 0.60 with each of them is behaving like a third annotator, and asking it for 0.9 asks more than the task allows. A judge at 0.35 is not agreeing with people, however good its raw accuracy looks. [Zheng et al. (2023)](https://arxiv.org/abs/2306.05685) made the same comparison on MT-Bench and found GPT-4's agreement with human preferences comparable to the agreement between humans. With more than two humans, or with items missing some labels, report Fleiss' kappa or Krippendorff's alpha across all raters as well as the pairwise kappas. [Inter-Annotator Agreement Explained](/docs/guides/inter-annotator-agreement) covers the choice. ## Keep the humans blind If annotators can see the judge's answer, they anchor on it, and part of the agreement you measure is the judge agreeing with itself. Collect the human labels without the judge's answers on screen. The same effect raises agreement when annotators review model pre-labels. See [Detecting Rubber-Stamped Pre-Labels](/docs/guides/detecting-rubber-stamped-prelabels). ## Check confidence as well as verdicts A judge can agree with people and still be overconfident. If its confidence decides which items go to a human, overconfidence lets the wrong items skip review. A confidence the model states about itself is generated text like the rest of its answer. Sampling gives an empirical one: query the judge k times per item at a temperature above zero, take the most common answer as its verdict, and use the share of samples that gave that answer as its confidence. At temperature 0 the k samples are identical and every confidence is 1.0. Expected calibration error (ECE) groups verdicts by confidence and compares each group's confidence with its accuracy. The [Brier score](https://en.wikipedia.org/wiki/Brier_score) is the mean squared difference between confidence and outcome. A reliability diagram plots the groups against the diagonal a perfectly calibrated judge would follow. ## Test pairwise judges for position bias A judge comparing two responses may favour whichever one comes first, or second, regardless of content. Zheng et al. documented this position bias in LLM judges. Run each pair twice with the order swapped. A verdict that flips when the order flips was decided by position, and the share of pairs that stay consistent belongs in the report next to agreement. See [Pairwise Model Comparison](/docs/guides/pairwise-model-comparison). ## Ordered scales On a 1–5 rating, a judge that says 4 where a human says 5 has nearly agreed. Plain kappa counts that the same as 1 against 5. Use weighted kappa or Krippendorff's alpha with an ordinal or interval metric, and report the mean absolute error alongside. See [Rating Scales](/docs/guides/rating-scales). ## How many items to label A kappa computed on 50 items has a wide confidence interval, wide enough that two judges can look different when the difference is noise. Report the interval with the estimate, and size the human sample before you start; [statistical power for annotation studies](/docs/guides/statistical-power-annotation) shows how. Stratifying the sample by the judge's label keeps rare classes in it. ## Doing it in Potato Potato's judge calibration runs each judge k times per item and keeps its labels in a separate store, so annotators never see them. It then draws a random or stratified sample for blind human labelling and writes a report. ```yaml annotation_schemes: - annotation_type: radio name: helpfulness description: "Is this response helpful?" labels: [helpful, unhelpful] judge_calibration: enabled: true prompt: | You are an impartial expert annotator. Classify the response as exactly one of: helpful, unhelpful. models: - endpoint_type: ollama model: llama3.1:8b base_url: http://localhost:11434 temperature: 0.7 k_samples: 5 sampling: strategy: stratified sample_size: 200 seed: 42 human: num_raters: 2 gold: majority schemas: [helpfulness] ``` The report gives accuracy, precision, recall and F1 for each model against the human gold label, and Cohen's κ split into human–model, model–model and human–human pairs. It adds Fleiss' κ and Krippendorff's α across all raters, ECE with reliability bins and the Brier score, and a confusion matrix per model. For Likert schemes it adds mean absolute error and ordinal α. Since version 2.9 the judge eval card also shows position-swap consistency, and the generated labels can be read at `/judge_calibration/results` before the human phase finishes. [LLM-as-Judge Calibration](/docs/agent-evaluation/judge-calibration) lists every option. [Judge Alignment](/docs/agent-evaluation/judge-alignment) covers tuning one judge's rubric against an existing gold set. ## Other tools LangSmith, Langfuse and Galileo each let you align a judge with human corrections. Langfuse computes Cohen's κ between a human score and a judge score, two series at a time. Label Studio's paid plans show where annotators and LLMs agree. See [AI Agent Evaluation Tools Compared](/docs/guides/agent-evaluation-tools-compared). ## Frequently asked questions ### What is a good kappa between an LLM judge and humans? There is no fixed threshold. Compare the judge-human kappa with the kappa between two humans on the same items. A judge close to the human-human figure agrees with people about as well as another annotator would. The usual rough guide for κ and α, 0.8 and above to rely on and 0.67 to 0.8 for tentative conclusions, describes the task as a whole, and a hard task caps what any judge can reach. ### Should annotators see the LLM judge's answer? Not while you are measuring agreement. Annotators who can see the judge's verdict tend to anchor on it, which inflates the judge's apparent agreement. Show the verdict only after the calibration sample is labelled, if at all. ### How do I check an LLM judge for position bias? Present every pair twice, once in each order, and count how often the verdict changes with the order. Report the share of consistent pairs next to agreement, and discard or re-run the pairs where the verdict flips. ### What is expected calibration error for an LLM judge? ECE measures the gap between a judge's confidence and its accuracy. Verdicts are grouped by confidence, and ECE is the average difference between each group's confidence and the share of its verdicts that matched the human label, weighted by group size. A well-calibrated judge has an ECE near zero. ## Further reading - [LLM-as-Judge Calibration](/docs/agent-evaluation/judge-calibration) - [Can You Trust Your LLM Judge?](/blog/trust-your-llm-judge-calibration) - [Rubric-Based LLM Evaluation](/docs/guides/rubric-based-llm-evaluation) - [Inter-Annotator Agreement Explained](/docs/guides/inter-annotator-agreement) - [AI Agent Evaluation Tools Compared](/docs/guides/agent-evaluation-tools-compared) --- # Evaluating Computer-Use and Multimodal Agents Section: Guides Source: https://www.potatoannotator.com/docs/guides/evaluating-computer-use-agents > How to human-evaluate computer-use and GUI agents, plus voice, video, and document agents, judging each action and click, scoring turn-taking, and grounding events in time. **A computer-use agent controls a graphical interface the way a person would: it reads a screenshot, decides on an action (click, type, scroll), and acts. Evaluating one means checking, step by step, whether each action was right and whether the click actually landed on the intended element, not just whether the task eventually succeeded.** Potato is an open-source tool for human evaluation of computer-use, GUI, voice, video, and document agents, with [annotation surfaces purpose-built for each modality](/docs/agent-evaluation/multimodal-agent-evaluation). A [computer-use agent](https://en.wikipedia.org/wiki/Software_agent) (also called a GUI or OS agent) perceives the screen as pixels or a DOM and acts through the same controls a user has. Benchmarks like [OSWorld](https://arxiv.org/abs/2404.07972), ScreenSpot, and AndroidWorld score task success automatically; human review adds what automation misses, the action that produced the right outcome by luck, or the click that hit the wrong button but still advanced the task. ## What do you judge in a computer-use trajectory? Each step pairs a **screenshot** (what the agent saw) with an **action** (what it did). The annotator judges the action and, when the step carries click coordinates, checks the grounding marker on the screenshot: - **Action correctness**: correct, wrong element, wrong action, or hallucinated. - **Click grounding**: did the coordinates land on the element the action named? - **Outcome**: did the run complete the task, and at which step did it first go wrong? ```yaml annotation_schemes: - annotation_type: gui_trajectory name: gui_review description: "For each step: was the action correct and did the click land right?" steps_key: steps screenshot_key: screenshot action_key: action coord_space: normalized verdict_options: [correct, wrong_element, wrong_action, hallucinated] ``` Catching the first wrong step matters more than a single pass/fail, because that step is what you would fix or train against; see [Process Reward Models](/docs/guides/process-reward-models). ## How do I evaluate a voice agent's turn-taking? Spoken agents fail between turns: cutting the user off, talking over them, or pausing too long. The [`voice_interaction`](/docs/agent-evaluation/multimodal-agent-evaluation#voice--full-duplex-interaction-voice_interaction) schema lays the conversation out as a dual-track timeline and highlights overlap regions where both speakers talk at once, which the annotator classifies (the agent should respond, should resume, was a backchannel, or it is unclear) and then rates the overall turn-taking. This is the [full-duplex](https://en.wikipedia.org/wiki/Duplex_(telecommunications)) view that a flat transcript cannot express. ## How do I score video and document agents? - **Video temporal grounding**: for each event prompt, mark the gold `[start, end]` interval; when the data includes a model's predicted interval, a live [IoU](https://en.wikipedia.org/wiki/Jaccard_index) updates as you adjust, so you score localization directly. - **Speech transcripts**: tag ASR/TTS errors segment by segment and correct the text inline. - **Document tables**: mark the cell structure (column headers, row headers, data, empty) that bounding boxes cannot capture. - **Interleaved reasoning**: rate each step of a text-image-tool trace for coherence and flag visual hallucinations. Each is a separate schema in the [multimodal-agent reference](/docs/agent-evaluation/multimodal-agent-evaluation), and several can run on the same task. ## Which schema should I use? | Agent type | Schema | What you label | |---|---|---| | Computer-use / GUI | `gui_trajectory` | Action correctness + click grounding | | Voice / spoken | `voice_interaction` | Barge-in handling and turn-taking | | Video | `temporal_grounding` | Gold event intervals vs. prediction (IoU) | | Speech transcript | `speech_transcript` | ASR/TTS errors per segment | | Document / table | `table_grid` | Cell-structure roles | | Multimodal reasoning | `multimodal_reasoning` | Step coherence and visual hallucination | ## Further reading - [Multimodal-Agent Evaluation](/docs/agent-evaluation/multimodal-agent-evaluation) — the full schema reference - [Web-Agent Evaluation](/docs/guides/web-agent-evaluation) — screenshot-and-action web agents - [How to Evaluate AI Agents](/docs/guides/evaluating-ai-agents) — the levels of agent evaluation - [How to Evaluate Multi-Agent Systems](/docs/guides/evaluating-multi-agent-systems) --- # Pairwise Model Comparison Section: Guides Source: https://www.potatoannotator.com/docs/guides/pairwise-model-comparison > How to compare two models or two responses head-to-head with human annotators, including multi-dimensional comparison and bias controls, using Potato. **To decide which of two models is better, show annotators a prompt and both responses and ask which one wins. Aggregated over many prompts, these head-to-head judgments rank models more reliably than absolute scores do.** It is the method behind public model leaderboards built on human votes. This is [pairwise comparison](https://en.wikipedia.org/wiki/Pairwise_comparison) applied to model outputs; many comparisons can be turned into a single ranking with an [Elo](https://en.wikipedia.org/wiki/Elo_rating_system) or [Bradley–Terry](https://en.wikipedia.org/wiki/Bradley%E2%80%93Terry_model) model. ## Basic head-to-head ```yaml annotation_schemes: - annotation_type: pairwise name: which_better description: "Which response is better overall?" mode: binary allow_tie: true ``` ## Multi-dimensional comparison A single "better" hides trade-offs, model A is more accurate but model B is clearer. Compare on several dimensions at once: ```yaml annotation_schemes: - annotation_type: pairwise name: comparison description: "Compare the two responses on each dimension." mode: multi_dimension ``` A required justification makes the data auditable and surfaces cases where annotators rewarded the wrong thing. ## Controlling for bias Head-to-head data is only as good as its bias controls: - **Position bias**: randomize which model is shown as "A"; annotators favor one side otherwise. - **Length/style bias**: annotators often prefer longer or more confident text regardless of quality. Name it in the guidelines. - **Verbosity ≠ quality**: consider capturing length so you can check whether it's driving wins. - **Agreement**: collect overlap and track [inter-annotator agreement](/docs/guides/inter-annotator-agreement). ## Comparison vs. rubric Use pairwise when you need a *ranking* and want high agreement. Use a [rubric](/docs/guides/rubric-based-llm-evaluation) when you need an absolute, per-dimension profile of each model. Many evaluations run both. ## Further reading - [Pairwise and Best–Worst Scaling](/docs/guides/pairwise-and-best-worst) - [Rubric-Based LLM Evaluation](/docs/guides/rubric-based-llm-evaluation) - [Collecting RLHF and Preference Data](/docs/guides/rlhf-preference-data) --- # RAG Evaluation Section: Guides Source: https://www.potatoannotator.com/docs/guides/rag-evaluation > How to evaluate retrieval-augmented generation with human annotation, retrieval relevance, answer faithfulness, and citation spans, using Potato. **Retrieval-augmented generation (RAG) answers a question by first retrieving documents and then generating an answer from them. Evaluating RAG means judging two things separately: did it retrieve the *right* documents, and is the answer actually *supported* by them?** Conflating the two hides where the system fails. See [retrieval-augmented generation](https://en.wikipedia.org/wiki/Retrieval-augmented_generation) for background. ## The three things to annotate 1. **Retrieval relevance**: for each retrieved document, is it relevant to the query? 2. **Answer faithfulness**: is the generated answer grounded in the retrieved documents, with no unsupported claims? 3. **Citation accuracy**: do the answer's claims actually trace to the cited sources? ## Setting it up in Potato Combine three schemes on one screen, rate each document, rate faithfulness, and highlight problem spans in the answer: ```yaml annotation_schemes: - annotation_type: multirate name: retrieval_relevance description: "Rate the relevance of each retrieved document to the query." options_from_data: retrieved_docs labels: ["Irrelevant", "Somewhat", "Relevant", "Highly relevant"] - annotation_type: likert name: faithfulness description: "Is the answer faithful to the retrieved documents?" size: 5 min_label: "Many unsupported claims" max_label: "Fully grounded" - annotation_type: span name: problems description: "Highlight any unsupported or incorrect claim in the answer." labels: [unsupported_claim, contradicted, hallucination] ``` `multirate` rates many documents on the same scale at once, and `options_from_data` builds one row per retrieved document from the item itself, which is what a varying number of documents per query needs; the span scheme marks exactly *where* the answer departs from its sources. See [Detecting Hallucinations](/docs/guides/detecting-hallucinations). ## Why separate retrieval from generation A RAG system can fail two ways: it retrieved bad context (a retrieval problem) or it ignored good context (a generation problem). Scoring them separately tells you which half to fix. A faithfulness score alone can't. ## Quality considerations - Show annotators the query, the documents, and the answer together, faithfulness can't be judged without the sources. - "Relevant" needs a definition: relevant to the query, or actually used in the answer? Decide up front. - Track [agreement](/docs/guides/inter-annotator-agreement) on faithfulness; it's the most subjective of the three. ## Further reading - [Detecting Hallucinations](/docs/guides/detecting-hallucinations) - [Rubric-Based LLM Evaluation](/docs/guides/rubric-based-llm-evaluation) - [How to Evaluate AI Agents](/docs/guides/evaluating-ai-agents) --- # Live Agent Evaluation Section: Guides Source: https://www.potatoannotator.com/docs/guides/live-agent-evaluation > How to evaluate an AI agent in real time, pause, send instructions, take over, rollback, and branch, using Potato's live agent display. **Most agent evaluation reviews a recorded trace. Live evaluation watches an agent run in real time and lets a human intervene, pausing it, sending instructions, taking control, or rolling back to try a different path.** For human-in-the-loop agent evaluation on your own server, Potato is an open-source tool that captures things a recording can't: where a person *would* have stepped in, and what better guidance looks like. For the feature reference, see [Live Agent Evaluation](/docs/features/live-agent-evaluation) and [Live Coding Agent](/docs/features/live-coding-agent). ## What does live agent evaluation add? - **Pause and resume**: stop the agent mid-task to inspect its state. - **Send instructions**: nudge it with guidance and observe how it adapts. - **Take over**: drive manually, then hand control back. The handoff points are valuable labels. - **Rollback and branch**: return to an earlier step and try an alternative, comparing paths from the same state. This produces *interventional* data, counterfactuals about what helps, not just observational labels. ## How do I set up live agent evaluation in Potato? Live mode connects Potato to a running agent through an endpoint (an OpenAI-compatible proxy, a custom HTTP endpoint, or a coding-agent backend). The annotator interacts through the live agent display. ```yaml live_agent: endpoint_type: anthropic_vision # or coding_agent, openai_proxy, ... ai_config: model: claude-sonnet-4-20250514 api_key: ${ANTHROPIC_API_KEY} max_steps: 30 allow_takeover: true allow_instructions: true ``` ## When should I use live agent evaluation? - **Building guidelines**: watching live reveals the failure modes worth encoding into a taxonomy for later batch labeling. - **Interactive tasks**: chat assistants and tool-using agents where the *interaction*, not just the transcript, is what you're judging. - **Stress testing**: probing how an agent recovers from a nudge or a forced detour. Live evaluation is higher-touch and lower-throughput than reviewing recorded traces, so it's best for a focused sample or for designing the batch task. For volume, switch to [trajectory annotation](/docs/guides/agent-trajectory-annotation) over recorded runs. ## Further reading - [Live Agent Evaluation feature reference](/docs/features/live-agent-evaluation) - [Web-Agent Evaluation](/docs/guides/web-agent-evaluation) - [How to Evaluate AI Agents](/docs/guides/evaluating-ai-agents) --- # Detecting Hallucinations with Span Annotation Section: Guides Source: https://www.potatoannotator.com/docs/guides/detecting-hallucinations > How to find and label hallucinations and factual errors in model output using span annotation and MQM-style error marking in Potato. **A hallucination is a confident statement a model makes that isn't supported by its input or by fact. The most useful way to capture one is to highlight the exact words and label what's wrong with them, a [span annotation](/docs/guides/span-annotation) task over model output.** Span-level labels are far more actionable than a single "this answer is wrong" flag. See [hallucination (artificial intelligence)](https://en.wikipedia.org/wiki/Hallucination_(artificial_intelligence)) for background. ## Spans versus whole-answer flags A whole-answer "unfaithful" label tells you *that* something is wrong; a span tells you *what* and *where*. Span data lets you measure error rates per type, find patterns, and build targeted training data. It mirrors [MQM](https://themqm.org/) (Multidimensional Quality Metrics), the standard error-span framework from machine-translation evaluation. ## Setting up error-span annotation ```yaml annotation_schemes: - annotation_type: span name: errors description: "Highlight each problematic span and label the error type." labels: [unsupported_claim, factual_error, contradiction, fabricated_citation] - annotation_type: radio name: severity description: "How serious is the worst error?" labels: [Minor, Major, Critical] ``` Add a severity judgment so you can weight a trivial slip differently from a dangerous fabrication, the way MQM does. ## Defining the error types - **Unsupported claim**: not backed by the source (the [RAG](/docs/guides/rag-evaluation) case). - **Factual error**: contradicts established fact. - **Contradiction**: conflicts with something earlier in the same output. - **Fabricated citation**: a reference that doesn't exist or doesn't say what's claimed. Keep the set small and give each a one-line definition with an example, per [Writing Annotation Guidelines](/docs/guides/writing-annotation-guidelines). Two public datasets do this: [FAVA](/showcase/fava-hallucination-spans) marks fine-grained hallucination spans in generated text, and [SHROOM](/showcase/shroom-hallucination-detection) is a shared task on overgeneration mistakes. ## Quality considerations - Give annotators the source material; "unsupported" is undefinable without it. - Boundary rules matter, does the span cover the whole sentence or just the false clause? Decide once. - Faithfulness is subjective at the edges; collect overlap and track [agreement](/docs/guides/inter-annotator-agreement). ## Further reading - [RAG Evaluation](/docs/guides/rag-evaluation) - [Span Annotation](/docs/guides/span-annotation) - [How to Evaluate AI Agents](/docs/guides/evaluating-ai-agents) --- # Human Evaluation of Generated Text Section: Guides Source: https://www.potatoannotator.com/docs/guides/human-evaluation-generated-text > How to run a defensible human evaluation of LLM and NLG output: defining criteria precisely, choosing absolute vs pairwise ratings, powering the study, and reporting enough to reproduce it. **Automatic metrics like BLEU and ROUGE correlate weakly with how good generated text actually is, so human evaluation is still the standard, and it is done badly more often than not. Three things make a human eval trustworthy: define each criterion precisely, prefer relative judgments over absolute scores, and report enough detail that someone else could rerun it.** This guide covers the protocol rather than the rubric wording. ## What automatic metrics miss For open-ended generation, summaries, dialogue, translations, LLM responses, [automatic metrics](https://en.wikipedia.org/wiki/Evaluation_of_machine_translation) compare against reference texts and miss most of what matters: a fluent, faithful answer phrased differently from the reference scores badly, and a fluent lie scores well. So human judgment remains the ground truth. The catch is that human evaluation is itself a measurement instrument, and a poorly designed one produces numbers as noisy as the metrics it replaced. [Howcroft et al. (2020)](https://aclanthology.org/2020.inlg-1.23/) surveyed twenty years of NLG evaluations and found the field could not even agree on what its own criteria meant: terms like "fluency," "adequacy," and "naturalness" were defined differently (or not at all) across papers, making results impossible to compare. Their recommendation is to pin down what each criterion means before collecting a single judgment. ## Define the criteria precisely Vague criteria are where most human evals go wrong. "Rate the quality from 1 to 5" invites every annotator to invent their own definition of quality. Split it into named, separately-defined dimensions, and write a one-sentence operational definition for each: - **Fluency**: is the text grammatical and well-formed, ignoring whether it's correct? - **Coherence**: do the sentences follow each other sensibly as a whole? - **Faithfulness / factual accuracy**: is every claim supported by the source (for summarization/RAG) or true (for open generation)? This is where [hallucinations](/docs/guides/detecting-hallucinations) get caught. - **Relevance**: does it actually address the prompt? - **Helpfulness**: for assistant-style tasks, does it accomplish what the user wanted? Measuring these separately tells you *why* one system beats another, not just that it did. ## Absolute scores or relative comparisons The main design choice is whether annotators rate one output at a time or compare several. - **Absolute ([Likert](/docs/guides/rating-scales)) ratings** are simple but suffer from scale bias: annotators anchor differently, avoid the extremes, and drift over a session, so a "4" from one rater isn't a "4" from another. - **Pairwise preference** (is A or B better?) sidesteps scale bias entirely and is generally more reliable, which is why it underpins [RLHF preference data](/docs/guides/rlhf-preference-data) and [model comparison](/docs/guides/pairwise-model-comparison). The cost is that you get a ranking, not an absolute level. - **[Best-worst scaling](/docs/guides/pairwise-and-best-worst)** shows a small set and asks only for the best and worst, which is a cheap way to get reliable rankings from few judgments. [van der Lee et al. (2021)](https://doi.org/10.1016/j.csl.2020.101151) lay out best-practice guidelines covering exactly these choices: how many items and evaluators, which scale, which statistical analysis. Worth reading before you commit to a design. ## Power and reporting Two failure modes remain even after the design is right. First, **underpowered comparisons.** Detecting a small quality difference between two good systems takes more items than people expect; run the [power analysis](/docs/guides/statistical-power-annotation) first, use a proper significance test, and report effect sizes, not just which mean was higher. Second, **unreported detail.** [Belz et al. (2021)](https://aclanthology.org/2021.eacl-main.29/) reviewed reproducibility in NLP and found human evaluations especially hard to reproduce, usually because the paper omits the exact criteria, instructions, annotator pool, and analysis. Record all of it as part of the study, not as an afterthought. A few mechanics that prevent avoidable bias: **randomize output order** so position doesn't leak (people favor the first option), **blind the system identity** so annotators can't tell which model produced what, and **pilot on a small batch** to measure [agreement](/docs/guides/inter-annotator-agreement) and fix confusing criteria before scaling up. ## Doing it in Potato Potato has a scheme for each evaluation style, so the design choice above maps straight to config. For per-criterion absolute ratings: ```yaml annotation_schemes: - name: faithfulness annotation_type: likert description: "Is every claim in the response supported by the source?" size: 5 min_label: "Many unsupported claims" max_label: "Fully supported" - name: fluency annotation_type: likert description: "Is the response grammatical and well-formed?" size: 5 min_label: "Not fluent" max_label: "Perfectly fluent" ``` For a blind A/B comparison, use a `pairwise` scheme and randomize which system is shown as A: ```yaml annotation_schemes: - name: preference annotation_type: pairwise description: "Which response is more helpful overall?" mode: binary items_key: responses labels: ["Response A", "Response B"] allow_tie: true tie_label: "Tie" ``` For structured, multi-criterion scoring in one pass, the [`rubric_eval`](/docs/guides/rubric-based-llm-evaluation) scheme collects a score per rubric dimension. Whichever you pick, keep overlap on a shared subset so you can report agreement, and keep per-annotator labels in the [export](/docs/features/export-formats) so the significance test has the variance it needs. ## Further reading - [Rubric-Based LLM Evaluation](/docs/guides/rubric-based-llm-evaluation), for turning fuzzy quality into scored dimensions. - [Pairwise Model Comparison](/docs/guides/pairwise-model-comparison), for A/B evaluation at scale. - [Statistical Power and Sample Size](/docs/guides/statistical-power-annotation), so the comparison can actually support its claim. - [RAG Evaluation with Human Annotation](/docs/guides/rag-evaluation), for the faithfulness/relevance case specifically. --- # How to Evaluate Generated Video and World Models Section: Guides Source: https://www.potatoannotator.com/docs/guides/world-model-evaluation > Ask annotators to mark the frame where the world stops making sense rather than to rate plausibility. Break-point annotation is checkable, localised, and produces a real agreement statistic. **Rating a generated video 3 out of 5 for "physical plausibility" produces a number that cannot be checked, cannot be localised, and cannot be used to fix anything.** Asking instead for *the frame at which the world stops making sense*, plus why, produces an annotation that can be all three. ## Why a break-point A point in time plus a category is: - **Checkable.** A researcher can open frame 47 and look. - **Localised.** The failure has a place in the tensor rather than only a score. - **Comparable.** Two annotators' answers are two points on a line, so a real chance-corrected agreement statistic applies. Video-generation benchmarks report FVD and win rates. None of them reports whether the humans producing those win rates agreed with each other, which means none of them can separate a model difference from annotator noise. ## The taxonomy Give annotators named categories rather than asking them to describe the failure. A workable physics and causality set: | Category | What it means | |---|---| | Object permanence | An object vanishes, or appears, with nothing causing it | | Rigid-body violation | A solid object bends, stretches or changes size | | Interpenetration | Two solid objects pass through each other | | Gravity violation | Something floats, falls upward, or stands unsupported | | Causality violation | An effect happens before, or without, its cause | | Identity flicker | An object swaps identity or category between frames | | Appearance drift | Texture, colour or shape drifts with no event to explain it | Add a severity scale. "Visible on a second look" and "nothing after this point is worth judging" are different findings. ## Blinding must be stable Shuffle panel order per annotator so model identity is not visible, and **seed the shuffle on the annotator and the item** so it is reproducible. Randomising afresh on each view breaks re-annotation: the annotator's second look at an item disagrees with their first for reasons that have nothing to do with the videos. ## Make "no breaks" explicit Require a positive "nothing wrong here" answer. Without it, a rollout with no marks is indistinguishable from one that was never reviewed, and your denominator is wrong in a direction that flatters the model. ## Report the sweep Break-point agreement decomposes three ways: - **Detection** — did they find a break at all? - **Localization** — did they put it in the same place? - **Category** — did they call it the same thing? And localization depends on a matching tolerance. Report the **sweep** across tolerances rather than one number, because agreement at 0.25 s and agreement at 2 s are different claims, and picking one lets the reader assume whichever supports their conclusion. ## Config ```yaml annotation_schemes: - annotation_type: rollout_evaluation name: rollout_review description: "Where does each rollout stop making sense?" streams: - {field: real, name: "Recording", role: real} - {field: gen_a, name: "Model A"} - {field: gen_b, name: "Model B"} fps: 25 layers: [violations, preference, counterfactual] blind: true shuffle: true require_clean: true ``` Declare `fps`. Without it, frame numbers are omitted rather than guessed. ## Where this fits [VBench](https://arxiv.org/abs/2311.17982), WorldModelBench and Physics-IQ provide the metrics and the reference protocols. Crowd panels provide scale. What none of them provides is an instrument for running the human side repeatedly with reliability measured — which is the gap this fills, and importing a benchmark's rollouts is a reasonable way to collect the human data those benchmarks validate against. ## Further reading - [World-model evaluation reference](/docs/vision-spatial/world-model-evaluation) - [Temporal agreement](/docs/measurement/temporal-agreement) - [Robot episode annotation](/docs/guides/robot-episode-annotation) - [World Model and Robot Episode Evaluation Tools Compared](/docs/guides/world-model-evaluation-tools-compared) --- # How to Evaluate VLM Grounding and Pointing Section: Guides Source: https://www.potatoannotator.com/docs/guides/vlm-grounding-evaluation > Score grounding by IoU at several thresholds, pointing by point-in-region hit rate, and ungroundedness as its own answer. Why a point scored like a box always fails. **Grounding evaluation asks whether a model put a region in the right place; pointing evaluation asks whether it put a point on the right thing.** They need different measures, and using the grounding measure for pointing reports total failure for a model that is pointing perfectly. ## Grounding: report several thresholds Score grounding by [IoU](https://en.wikipedia.org/wiki/Jaccard_index) against the human region, at **0.25, 0.5, 0.75 and 0.9**. One threshold cannot distinguish "nearly right" from "nowhere near", and a model tuned to clear 0.5 exactly looks identical to one that is genuinely tight. The shape of the curve across thresholds is the finding. ## Pointing is not grounding with a small box Models in the [Molmo](https://arxiv.org/abs/2409.17146) family emit points. **A point has no area, so every IoU against it is 0.** Score a pointing model the way you score a grounding model and it reports complete failure regardless of how well it points. The question a point answers is "is it *in* the thing", so the measure is a hit rate: `point_in_region`. Report `mean_miss_distance` over the **misses only**. Averaged over hits as well, it mostly measures how large the objects are rather than how badly the model missed. ### Annotator agreement on points is a distance The same structural problem appears between annotators. An overlap-derived measure applied to points saturates: two annotators pointing at **opposite corners of the same image** still score about 0.86, which any coefficient scale bands as strong agreement. So report the **mean and median distance** between annotators' points in normalized image units, where 0 is perfect. A distance has no ceiling to saturate against, and calling it a distance stops anyone banding it as a coefficient. ## Ungroundedness is three different errors Track these apart: | Outcome | Meaning | |---|---| | Correctly declined | The expression refers to nothing present, and the model said so | | Hallucinated a location | A region was given for something that is not there | | Missed a present referent | Something was there and was not found | Folding them into one "wrong" number hides the difference between a model that over-claims and one that under-claims, which have opposite fixes. **An expression the annotator never answered is excluded, not counted as a miss.** Counting it makes a model look worse the more phrases were skipped, which is a statement about the annotator rather than the model. ## Region captions need a semantic distance Two annotators describing the same object rarely share a content word. "A red mug with a chipped handle" and "the damaged cup on the left" describe the same region and have no overlapping content terms. Compute caption agreement through a **semantic distance** rather than string overlap. Krippendorff's α accepts any distance function, so the same coefficient machinery applies once the distance is right. ## Config ```yaml annotation_schemes: # grounding_eval owns the binding but does not draw, so an image schema # supplies the canvas. - annotation_type: image_annotation name: region description: "Draw the region for the selected phrase." source_field: image tools: [bbox, polygon, landmark] labels: [{name: referent, color: "#6e56cf"}] - annotation_type: grounding_eval name: grounding description: "What does each phrase refer to?" region_type: box # use `point` for pointing evaluation expressions_field: expressions ``` Items carry their phrases with stable ids: ```json {"id": "img1", "image": "/media/img1.jpg", "expressions": [{"id": "e1", "text": "the man in the red shirt"}]} ``` A bare list of strings works, but ids then fall back to position, so reordering the list re-points existing annotations. Working examples: the [RefCOCO](/showcase/refcoco-expression) and [PixMo-Points](/showcase/pixmo-points-pointing) showcase designs. ## Reviewing model predictions If you are auditing a model rather than collecting ground truth, keep predictions labelled as predictions the whole way through. A prediction the annotator has not accepted is evidence about the model, and merging it into the human regions destroys the only thing it is for. ## Further reading - [VLM grounding reference](/docs/vision-spatial/vlm-grounding) - [Measuring agreement on bounding boxes](/docs/guides/measuring-agreement-on-bounding-boxes) - [Open-vocabulary object detection](/docs/guides/open-vocabulary-object-detection) --- # Running a Study on Prolific and MTurk Section: Guides Source: https://www.potatoannotator.com/docs/guides/crowdsourcing-prolific-mturk > How to run a crowdsourced annotation study with Potato on Prolific or Amazon Mechanical Turk, linking participants, quality control, and fair pay. **Crowdsourcing platforms like Prolific and Amazon Mechanical Turk give you access to many annotators on demand. Potato integrates with both: participants click through from the platform, do the task, and return with a completion code.** The wiring is the easy part. Quality control and fair treatment of workers are the hard ones. See [crowdsourcing](https://en.wikipedia.org/wiki/Crowdsourcing) and [Amazon Mechanical Turk](https://en.wikipedia.org/wiki/Amazon_Mechanical_Turk) for background. For the integration reference, see [MTurk Integration](/docs/deployment/mturk-integration) and Prolific integration in the deployment docs. ## How the flow works 1. You deploy a Potato task on a public URL. 2. The platform sends each participant to that URL with their ID in the link. 3. Potato records annotations against that ID. 4. On finish, Potato shows a completion code the worker submits back to the platform to get paid. Potato captures the participant ID from the URL so annotations tie back to the right worker automatically. ## Quality control Open crowdsourcing attracts some low-effort and automated responses. Build defenses in from the start: - **[Attention checks](/docs/guides/gold-standards-and-attention-checks)** to catch workers who aren't reading. - **[Gold standards](/docs/guides/gold-standards-and-attention-checks)** to score accuracy against known answers. - **A [training/qualification phase](/docs/features/training-phase)** so only workers who pass reach the real task. - **[Overlap](/docs/guides/how-many-annotators)** so you can measure agreement and aggregate. ## Treat workers fairly This is both ethical and practical: well-paid, well-briefed workers produce better data. - **Pay at least a fair hourly rate.** Time your pilot to estimate how long the task really takes, then price the reward accordingly. - **Write clear instructions** and a short consent/intro. See [Writing Annotation Guidelines](/docs/guides/writing-annotation-guidelines). - **Respond to questions** and pay promptly; reputation affects who takes your future tasks. ## Further reading - [MTurk Integration reference](/docs/deployment/mturk-integration) - [Deploying an Annotation Server](/docs/guides/deploying-annotation-server) - [Gold Standards and Attention Checks](/docs/guides/gold-standards-and-attention-checks) --- # Deploying an Annotation Server Section: Guides Source: https://www.potatoannotator.com/docs/guides/deploying-annotation-server > How to deploy a Potato annotation server for a real study, authentication options, HTTPS, and production setup with Docker or a reverse proxy. **Running an annotation study for real means moving Potato off your laptop and onto a server annotators can reach, with the right login method and HTTPS so credentials and data are protected.** This guide covers the deployment choices; for step-by-step setup see [Local Development](/docs/deployment/local-development) and the deployment section. ## Choose an authentication method Match the login to your annotator population: - **Passwordless**: annotators enter just a username. Fine for trusted internal teams or crowd studies where the platform already identified the worker. See [Passwordless Login](/docs/features/passwordless-login). - **Passwords**: a managed user list, for a fixed set of named annotators. - **OAuth / SSO**: sign in with Google or GitHub, or your institution's [single sign-on](https://en.wikipedia.org/wiki/Single_sign-on). Best for larger or institutional deployments. ## Put it behind HTTPS Any real deployment should serve over [HTTPS](https://en.wikipedia.org/wiki/HTTPS) so usernames, answers, and any personal data are encrypted in transit. The usual pattern is a [reverse proxy](https://en.wikipedia.org/wiki/Reverse_proxy) such as nginx terminating TLS in front of Potato, or a hosting platform that handles TLS for you. ## A typical production stack - **Potato** running under a production server process. - **nginx** (or similar) as a reverse proxy handling TLS and static assets. - **Docker** to package it reproducibly, if you containerize. - **A public DNS name** with a certificate (Let's Encrypt is free). Pilot the exact config locally first, then deploy the same config, the only things that change are the URL, auth, and TLS. ## Operating the study - Watch progress, throughput, and [agreement](/docs/guides/inter-annotator-agreement) in the admin dashboard. - Back up the output directory regularly; annotations are written there. - For crowd studies, see [Running a Study on Prolific and MTurk](/docs/guides/crowdsourcing-prolific-mturk). ## Further reading - [Local Development](/docs/deployment/local-development) - [Exporting Annotations for ML](/docs/guides/exporting-annotations-for-ml) - [Remote Data Sources](/docs/features/remote-data-sources) --- # Exporting Annotations for Machine Learning Section: Guides Source: https://www.potatoannotator.com/docs/guides/exporting-annotations-for-ml > How to export Potato annotations into ML-ready formats, JSON/JSONL, CoNLL, Hugging Face Datasets, COCO, and YOLO, and what each is for. **The point of annotation is usually to train or evaluate a model, so the export format matters. Potato writes plain JSON/JSONL/CSV and also ML-native formats that training pipelines read directly, no glue code.** Choosing the target format before you label tells you how to structure your data and IDs. For the reference, see [Export Formats](/docs/features/export-formats). ## Pick the format for the job | Format | Use it for | |---|---| | **JSON / JSONL** | General-purpose; one record per item. The safe default. | | **CSV** | Spreadsheets and quick analysis of classification labels. | | **[CoNLL](https://en.wikipedia.org/wiki/CoNLL)** | Token-level sequence labeling (NER, chunking) with [BIO tags](https://en.wikipedia.org/wiki/Inside%E2%80%93outside%E2%80%93beginning_(tagging)). | | **Hugging Face Datasets** | Loading straight into `transformers` training. | | **spaCy** (from CoNLL) | Training spaCy NER models. Export CoNLL, then run `spacy convert`. | | **COCO / YOLO** | Object detection and segmentation from [image annotation](/docs/guides/image-annotation). | | **Parquet** | Large-scale columnar storage and analytics. See [Parquet Export](/docs/features/parquet-export). | ## Setting the output format ```yaml output_annotation_dir: "annotation_output/" export_annotation_format: "jsonl" # jsonl, csv, conll, ... ``` ## What ends up in the export A typical record carries the item ID, the original content, every annotator's labels, and metadata (who, when). Keeping all annotators' labels, rather than only an aggregate, lets you compute [agreement](/docs/guides/inter-annotator-agreement) and re-aggregate later with a different method. ## Plan the export before you label The export format constrains your input design. Sequence-labeling exports need consistent tokenization; COCO/YOLO need image dimensions; Hugging Face needs a stable label set. Decide the destination first so you don't have to re-run the study. See [Designing Data Formats for Annotation](/docs/guides/data-formats-for-annotation). ## Further reading - [Export Formats feature reference](/docs/features/export-formats) - [Designing Data Formats for Annotation](/docs/guides/data-formats-for-annotation) - [Span Annotation](/docs/guides/span-annotation) --- # Multilingual and Low-Resource Annotation Section: Guides Source: https://www.potatoannotator.com/docs/guides/multilingual-low-resource-annotation > Annotating in languages beyond English: the diversity gap, participatory methods with native speakers, and how to localize the Potato interface with right-to-left support, fonts, and translated labels. **Annotating in a language other than English is two problems at once. The scientific one: most NLP resources cover a handful of languages, categories don't transfer cleanly across cultures, and you need actual speakers, not just bilinguals, to produce good labels. The practical one: getting the tool to display the language correctly, including right-to-left scripts and non-Latin fonts. Potato handles the second through config; the first is on you.** ## The diversity gap There are roughly [7,000 languages](https://en.wikipedia.org/wiki/Language) in the world and NLP meaningfully serves a few dozen. [Joshi et al. (2020)](https://aclanthology.org/2020.acl-main.560/) quantified this: a small number of languages have abundant labeled data while the vast majority, spoken by billions of people, have almost none, and the gap is self-reinforcing because resources flow to languages that already have them. Annotation is usually the bottleneck. If you want a model to work in a [low-resource language](https://en.wikipedia.org/wiki/Low-resource_language), someone has to label data in it, and that labeling is where the quality is won or lost. ## Working with the language community The instinct is to hire cheap bilingual crowdworkers and translate an English guideline. Both parts of that are risky. [Bird (2020)](https://aclanthology.org/2020.coling-main.313/) argues against extractive approaches that treat a language community as a data source, and the [Masakhane project's participatory model (Nekoto et al., 2020)](https://aclanthology.org/2020.findings-emnlp.195/) shows the alternative working at scale for African languages: native speakers help design the task, author the guidelines, and own the labels, rather than validating decisions made elsewhere. Two practical consequences: - **Recruit fluent speakers, ideally of the right variety.** A language is not monolithic: dialect, region, and register matter, and a speaker of one variety may mislabel another. Bilingual is not the same as native. - **Don't assume categories transfer.** Sentiment, offensiveness, politeness, and even named-entity types are culturally specific. A guideline that works for English politeness can break in a language where politeness is grammatically encoded. Have speakers adapt the *scheme*, not just translate the words. This is a [scheme-design](/docs/guides/choosing-an-annotation-scheme) decision, not a translation task. Several showcase tasks reproduce this kind of multilingual data and make good starting points: cross-lingual natural language inference in 15 languages ([XNLI](/showcase/xnli-cross-lingual-nli)), translation quality across 200 languages ([FLORES](/showcase/flores-mt-quality)), and sentiment for Indonesian regional languages ([NusaX](/showcase/nusax-indonesian-sentiment)). ## Localizing the Potato interface The practical half. Potato localizes the annotator-facing interface through configuration, no code changes, though it's worth knowing the edges up front. **Interface text.** The `ui_language` block is a string table for the interface chrome. Set the document language and translate the buttons and headings the annotator sees: ```yaml ui_language: html_lang: ar html_dir: rtl # right-to-left for Arabic, Hebrew, etc. submit_button: "إرسال" instructions_heading: "التعليمات" ``` Setting `html_dir: rtl` flips the whole document for right-to-left scripts using the browser's native bidirectional handling. One limitation: `ui_language` covers the core annotation and login screens, but several admin-side pages (the dashboard, adjudication, training, and logout screens) are still English-only, so plan for that if your annotators will see them. **Fonts for non-Latin scripts.** Browsers don't always ship a good default font for CJK, Arabic, or Indic scripts. Load one through a project stylesheet with [`base_css`](/docs/features/layout-customization): ```yaml base_css: "css/noto_font.css" ``` **Non-English data.** Potato reads data files as UTF-8 by default and renders arbitrary Unicode in the text field, so non-English content displays as-is. If a file uses a different encoding, override it per file (`encoding:`) or globally (`data_directory_encoding:`). **Translated labels.** By default Potato "humanizes" label names, reformatting and title-casing them, which can mangle non-Latin scripts. Keep machine-readable English `name`s for your data, and show the annotator a translated label with `displayed_label`, with humanizing turned off: ```yaml annotation_schemes: - name: sentiment annotation_type: radio description: "ما هو شعور هذا النص؟" labels: - name: positive displayed_label: "إيجابي" - name: negative displayed_label: "سلبي" - name: neutral displayed_label: "محايد" ``` The stored annotation is still `positive`, so your data stays clean while the annotator works entirely in Arabic. **Instructions and consent.** Author your [`surveyflow`](/docs/features/surveyflow) consent and instruction pages directly in the target language, they are HTML files you write, with nothing forcing English. For running the same task across several languages, Potato ships a `setup_multilingual_config.py` helper that generates per-language configs from a template. ## Further reading - [Choosing an Annotation Scheme](/docs/guides/choosing-an-annotation-scheme), because categories rarely transfer across languages unchanged. - [Writing Annotation Guidelines](/docs/guides/writing-annotation-guidelines), which native speakers should help author, not just translate. - [Running a Study on Prolific and MTurk](/docs/guides/crowdsourcing-prolific-mturk), for recruiting speakers of the right language and variety. - [Layout Customization](/docs/features/layout-customization), for fonts and custom templates. --- # How to Run Annotation on an Air-Gapped Network Section: Guides Source: https://www.potatoannotator.com/docs/guides/air-gapped-annotation > What "offline" has to mean for an annotation tool, why an @import in a stylesheet defeats most air-gap audits, and how to get models onto a machine with no route out. **An annotation tool that "works offline" usually means the annotation page renders. That is not the same as the deployment making no outbound requests, and the gap is where the privacy problems live.** If you are deploying into a secure research environment, a clinical setting or a government network, this is what to check. ## Audit the rendered page, not the source The only reliable test is a live one. Open the annotation interface with DevTools' network panel filtered to third-party requests and count. Reading the templates is not sufficient, and this is not a hypothetical distinction. Potato's own `styles.css` opened with an `@import` of Google Fonts, which survived **three previous air-gap audits** because every guard checked `