What's New

This page covers new features and improvements across Potato v2.x releases.

Potato 2.11.1

Released October 2026

Potato 2.11.1 lets potato start config.yaml --debug start on the default address again, adds image classifiers to model training, and fixes prelabels that were lost on every restart.

pip install --upgrade potato-annotation

Debug mode

From 2.8.2 through 2.11.0, a debug server on the default 0.0.0.0 address refused to start unless --host 127.0.0.1 was given. In 2.11.1 it starts on any address, and other machines on the network can reach it. The debugging guide describes what debug mode now does:

• The login page stays unless you add --debug-phase. With --debug-phase, every visitor is signed in as the shared debug_user, so do not use it for a real study.
• Admin pages open without a key only for requests from the server's own machine, meaning a loopback address with no X-Forwarded-For, X-Real-IP or Forwarded header. This holds on a 127.0.0.1 bind too, where 2.11.0 let every request through, because a proxy or tunnel on the same machine forwards other machines' requests to 127.0.0.1.
• The Werkzeug interactive debugger runs only on a 127.0.0.1 bind, because it executes code typed into the browser.

POTATO_ALLOW_REMOTE_DEBUG is no longer read. A debug command with --host 127.0.0.1 still works and still gives you the interactive debugger.

Image classification and prelabels

• Model training fits image classifiers. A radio or multiselect scheme whose items name image files is trained on CLIP embeddings, where it used to fit TF-IDF on the file paths. Images that are remote or missing are left out with a warning.
• Image active learning works as documented. vectorizer: {name: image} builds the CLIP vectorizer, which was not a name the manager could build, so such a study was ranked by TF-IDF over file paths.
• The labeling engine distills image classifiers on an image study (labeling_engine.distill.modality).
• Prelabels for unlabeled items. model_training.writeback.include_unlabeled: true predicts on the items nobody has labeled yet. Before, a run predicted only on held-out items that were already labeled, so write-back never reached the items a prelabel is for.

The model training documentation covers training runs, write-back and the review queue.

Fixes that affect earlier studies

• Prelabels did not survive a restart. The startup replay of stored predictions ran before any item was loaded, so the model review queue emptied on every restart. It now runs after the items load and applies the same checks as write-back.
• A text model trained on URLs. Items with a url, path or file_name field gave the trainer that value instead of the text.
• Password resets failed under SSO. With method: oauth and allow_local_login, every reset raised. Local accounts can now be reset.
• ai_support refused OpenRouter. endpoint_type: openrouter is now accepted.

The release notes have the full list.

Potato 2.11.0

Released October 2026

Potato 2.11.0 adds the labeling engine, which labels a dataset with models and uses a blind random sample of human labels to measure how often the models are wrong. You state what the finished labels must satisfy, such as "at most 5% of accepted machine labels are wrong, with 90% confidence". The engine accepts the machine labels that a statistical test supports, sends the rest to people, and writes a certificate that records what was tested and on how many human labels. The labeling engine covers setup.

Idle sessions are now signed out, after 8 hours by default. Before 2.11.0, session_timeout_minutes had no effect on a served app, because the check was registered on an app object that potato start discards. A session that sits idle longer than the setting (480 minutes by default) is now cleared, and its next request goes to the login page. Kiosks and shared lab machines that stay signed in overnight will start seeing sign-outs, so raise session_timeout_minutes where that matters.

pip install --upgrade potato-annotation

The labeling engine

A labeling_engine block and assignment_strategy: engine turn the engine on. Its sources are models, tried in cascade order, and an item goes to people after the last source. Each contract names one annotation scheme and is tested against human labels that the engine requests at random and serves blind, with no pre-annotation, judge verdict or AI help on the page.

• Sources. LLMs, vision-language models, decision models (Laya and any /v1/systemone server), YOLO detectors, LLM judges with a label-order swap, juries, models trained on the study's own labels, and labels you already have.
• Contracts. selective_risk bounds the share of accepted machine labels that are wrong, at a stated confidence. ppi gives prevalence or mean intervals that stay valid whatever the machine's quality. no_worse_than_human runs the alternative annotator test, budget_first projects what a human-label budget buys, and distributional compares the machine's labels with how people split on subjective items.
• Modalities. One-of-N, ordinal, numeric and multi-label schemes, image regions, audio, video, temporal-grounding and tiered segments, 3D objects, and agent-trace step labels. Regions, segments, 3D objects and traces are scored with the measures the agreement report uses.
• Audits. A share of accepted machine labels goes back to people, blind. When the audits show an error rate above the target, the engine stops accepting machine labels on that scheme (with features.drift, on by default), and certifying again restores acceptance.
• Interfaces. potato label run|status|certify|certificate|export|calls|distill|optimize|migrate-config, a REST API under /api/engine/v1, five engine_* MCP tools, a Python SDK, and an admin page at /admin/labeling-engine.

A met contract is a statistical statement about agreement with the people who labeled the sample. The engine does not check that those people are right, so a contract tested against unreliable annotators certifies agreement with unreliable annotators. Labeling engine contracts says what each contract tests and when it fails.

icl_labeling, judge_calibration and pre_annotation keep working, and each prints a note at startup that names its engine equivalent. potato label migrate-config writes the equivalent labeling_engine block. Solo Mode and judge alignment are not deprecated. Migrating to the labeling engine has the translation table.

Agreement

• /admin/iaa reports more schemes. spatial_annotation, temporal_grounding, tiered_annotation and the agent-trace schemes (process_reward, trajectory_eval, tool_call_review, failure_attribution, gui_trajectory) were skipped as unsupported and are now scored.
• /admin/api/agreement no longer reports structured schemes. Reduced to label names, agent traces, segments, regions and 3D objects showed perfect agreement between every pair of annotators. The route now points to /admin/iaa, which compares the structures.
• Two identical zero-length segments now agree. They used to score as complete disagreement.
• Machine raters stay out of published figures. Publishing and Paper Mode leave machine raters out of gold labels and agreement figures.

Solo Mode

• Blind suggestions. solo_mode.suggestions withholds the LLM's label on all items or on a sample, and the handoff reads agreement on the blind items, because a suggestion on screen anchors the person checking it.
• A spend cap and a response cache. ai_budget.cap_usd is checked before labeling starts and before each batch, and solo_mode.cache reuses the reply to a repeated call.
• Decision models. The systemone and laya endpoints label fixed-choice schemes in one forward pass, as a fast first tier. Solo Mode has a config.
• Gold set, jury and curriculum. solo_mode.gold scores a prompt on known labels before the full run, solo_mode.jury has several models vote on each item, and instance_selection.curriculum starts you on the items the LLM found easiest. A prompt-sensitivity check is new as well.

Fixes that affect 2.10 studies

• An image page saved a prefilled answer on its own. On an image_annotation scheme with pre_annotation, the canvas saved the prefilled shapes as the annotator's answer when it re-laid itself out after loading, so an untouched item counted as answered. Only an edit saves now. In a 2.10 image study with pre-annotations, some items counted as answered may never have been touched.
• Images on hosts without CORS headers did not load. A host that sends no Access-Control-Allow-Origin header, such as cocodataset.org, refused the canvas's request, and the page showed "Failed to load image". The page now loads such an image without CORS. Drawing works, and region fill, SAM and text-prompt segmentation, which read the image's pixels, say why they cannot run.
• A pre-annotated shape without a color was dropped. The item showed "Annotations: 0" with no warning, and the shape now takes its label's color.
• Solo Mode's "Generate prompt" never called a model. It now asks the revision model to write the first prompt. Without a revision model it falls back to the task description and says so.
• Refusals over potato mcp connect looked like successes. A refused tool call came back with isError: false, and it now comes back as a tool error that carries the server's message.
• The full-study skeleton example had a dead key. Potato reads no training.feedback.enabled, so the example now sets feedback.show_explanations.
• Idle sessions were never signed out, as described at the top of this entry.
• Adjudication counted machine raters. Machine annotators could send an item to adjudication, min_human_annotations was ignored, and a missing agreement score crashed the formatting.
• Prolific auto-scaling counted machine annotations and ignored per-item caps. The number of places to open now counts only human annotations against each item's cap.
• Edge-case discovery did nothing on vLLM. Guided decoding on vLLM never returned the edge-case fields, and they are now part of the response schema.

The release notes have the full list.

Potato 2.10.3

Released October 2026

Potato 2.10.3 fixes three bugs found in 2.10.2 after it was published.

pip install --upgrade potato-annotation

• Agreement on unanimous data. When every annotator gave the same answer on every item, /admin/api/agreement returned the schema as {"error": "float division by zero"}. Chance agreement is 1 in that case, so alpha and both kappas are undefined. The route now reports them as null, with percent_agreement at 1.0 and the same note that /admin/iaa gives, and the overall average leaves the schema out.
• potato deploy button --dry-run stays offline. Without --hf-backup-repo, the dry run asked HuggingFace who owned the token, and a bad token ended in a traceback. The dry run now uses the placeholder <hf-account>/<name>-annotations, as potato deploy up --dry-run does, and a real run reports a rejected token as an error message.
• Importing Potato no longer hashes a password. Potato computed a PBKDF2 hash at import, so every process spent 100,000 iterations at startup, CLI tools included. The hash is now made the first time an unknown username tries to log in. The import also works under Pyodide again, whose hashlib has no pbkdf2_hmac.

The release notes have the full list.

Potato 2.10.2

Released October 2026

Potato 2.10.2 fixes features that did not work from a pip install or the Docker image, closes ways to sign in as another user, makes the MySQL backend work, and corrects agreement statistics. If you published agreement numbers from an earlier version, recompute them.

pip install --upgrade potato-annotation

• SSO accounts accepted any password. On a task with OAuth, the password form signed anyone into an account a provider had registered. SSO identities no longer authenticate by password, and on a MySQL study a stranger can no longer register an existing annotator's username and inherit their work.
• ai_support and survey instruments failed from pip. The wheel left out the AI prompts and the survey instruments, so a study that used either stopped at startup. Containers and potato deploy targets also skipped ten subsystems, chat support among them.
• The MySQL backend never turned on. A config that named it stored annotations on disk. Install the driver with pip install 'potato-annotation[mysql]'; export, backups and potato deploy pull now read the database.
• 59 agreement statistics fixes, each checked against a reference implementation such as scipy, sklearn or the krippendorff package. An undefined coefficient is reported as undefined rather than 1.0, ordinal alpha uses Krippendorff's ordinal metric, MACE fits pure spammers at about 0.01 competence instead of 0.35, results no longer change after a restart, and plackett_luce fits the best-worst likelihood instead of repeating Bradley-Terry.
• Point clouds in map coordinates keep their precision. A LAS file in UTM coordinates was drawn on a grid as coarse as 0.5 m. Potato now serves it relative to a local origin and saves annotations in the file's own coordinates.

The release notes have the full list.

Potato 2.10.1

Released October 2026

Potato 2.10.1 fixes five problems in potato deploy and the backup that were found after 2.10.0. Upgrade before you rely on a dry run.

pip install --upgrade potato-annotation

• A dry run passed for deploys that up then refused. Heroku or ECS Express without --backup, Render or Railway with no backup storage to upload the project to, and Fly with a project too large to send inline all printed a clean plan. The plan and up now run the same check, and both print REFUSED: <reason> and exit with status 2.
• --demo was offered where it cannot work. Render, Railway and ECS Express upload your project to the backup's storage, so they need --backup hf or --backup s3 even for a disposable study. The preflight warning now offers --demo only for Heroku and HuggingFace.
• --cloud jetstream2 planned the wrong flavor. The dry run showed m1.small, while up created an m3.small.
• Two prices were wrong. potato deploy providers said about $16 a month for aws-ec2 and about $7 for fly. The plans compute $18.31 and $5.85, and the summaries now say about $18 and about $6.
• backup: {enabled: false} did not turn the backup off when the config still had a huggingface_backup block. The new block now wins.

The release notes have the full list.

Potato 2.10.0

Released October 2026

Potato 2.10.0 adds eight hosting platforms to potato deploy, backs up annotations on hosts whose disk does not survive a restart, and adds text_as_image, which shows each item's text as a picture so annotators cannot paste it into a chatbot. Running Potato Locally or in the Cloud shows which host fits which study.

Python 3.9 or later is now declared. A core dependency already required it, so earlier releases could not be installed on 3.7 or 3.8 either, and pip now says so before it installs anything.

pip install --upgrade potato-annotation

Deploying

• AWS, three ways. --provider aws creates a Lightsail instance at a flat $12 a month that needs only lightsail:* permissions. aws-ec2 is for accounts where Lightsail is turned off, and aws-ecs runs the container on ECS Express Mode behind an AWS-managed HTTPS address, at a higher cost and with a required backup.
• Heroku, on one Basic dyno. Heroku wipes the disk daily, so Potato refuses to deploy there without --backup or --demo.
• Fly and Railway, each with a persistent volume.
• Hetzner, Vultr and Linode, as VMs set up the same way as DigitalOcean, with a Let's Encrypt certificate for the IP address.
• OpenStack, with a Jetstream2 preset. Jetstream2 gives each instance a DNS name, so the task gets an ordinary certificate. Campus OpenStack clouds and EGI work too.
• Deploy buttons. potato deploy button config.yaml --target heroku|render|aws|railway writes the files for a "Deploy to Heroku", "Deploy to Render" or AWS "Launch Stack" button into your repository, so other people can run the task in their own accounts.

The new deploy-aws and deploy-openstack extras install the AWS and OpenStack clients.

Backup and restore

A new backup block copies the annotation output and snapshots of the SQLite databases to a HuggingFace dataset, an S3-compatible bucket, or both, every five minutes. When a server starts with an empty task directory and the backup holds data, Potato restores it before loading anything, so annotators continue where they stopped after a restart. Pass --backup hf, --backup s3 or --backup hf,s3 to potato deploy up. The older huggingface_backup block is read as a backup block with one sink and restore turned off, so existing deployments behave as before.

Text as image

text_as_image: true renders each item's text to a PNG on the server and removes the words from the page, the embedded item record and the JSON endpoints, so an annotator has nothing to select, copy or paste. The feature raises the cost of pasting an item into a chatbot, and an annotator can still retype the text or run OCR on a screenshot. A screen reader gets nothing from the picture, so the feature does not suit a study with annotators who use one. Potato refuses to start when text_as_image is combined with a span scheme, a scheme that reads the item text field, Pocket Mode or Rooms. Contributed by @Eyecatch3r in PR #169.

Fixes that affect collected data

• A restart on a host without a persistent disk lost earlier work. The server started empty, and a returning annotator's new state overwrote the backed-up copy. Restoring on start fixes this.
• DigitalOcean with --volume-gb kept annotations on the root disk rather than on the volume. The output and the SQLite databases now live on /data.
• The HuggingFace backup did nothing in the published image, which did not include huggingface_hub, and on Render and DigitalOcean --hf-token set a variable that nothing read. The image now installs the new hosting extra.
• Render deploys started without the project, and the container exited with "config file not found". The container now downloads the project at start and checks its sha256.

The release notes have the full list.

Potato 2.9.2 to 2.9.4

Released September 2026

Upgrade to 2.9.4 or later if people you do not trust can reach your server. Before 2.9.4, the /forgot-password form created a reset token for any username submitted and showed the reset link on the page, so anyone who could reach the server could take over any password account, admin accounts included (GHSA-43v4-43pw-4rp2). The form now creates no token. Reset links come only from POST /admin/create_reset_token and the potato reset-password command. Reported, with a patch, by HDEF.

• Phones (2.9.4). On touch screens, controls are at least 44 px tall, the header takes two rows instead of three, multirate becomes a list of labeled rows, and span highlighting works by long-press.
• Machine annotators (2.9.4). A machine_annotators block declares raters that are programs, such as detectors or LLMs. /admin/iaa then reports human-human, machine-machine and pooled agreement, with human-only headline figures, and csv, tsv and parquet exports gain an annotator_origin column when any record came from a machine.
• Session cookies behind a URL prefix (2.9.4). Two studies served under /app1 and /app2 on one host shared one session cookie, so signing in to one signed you out of the other. The cookie is now scoped to the prefix. Fixed by @Eyecatch3r in PR #172.
• Faster navigation (2.9.2). Scripts and stylesheets stay cached between items, and Next takes fewer round trips. Over a link with 80 ms round trips, Next on a radio-button task went from about 2.6 s to about 0.8 s.
• Text edits right before Next (2.9.2). Clearing a text box and pressing Next within a second kept the old text. Every save now applies pending edits first.

Potato 2.9.0

Released September 2026

Potato imports projects from brat, doccano, Prodigy, CoNLL and REFI-QDA, and transcribes and diarizes audio locally. Exports record whether an image shape was drawn by a person, proposed by a model or imported, and which room a vote was cast in. Most of the release is bug fixes.

Run potato validate --strict on each config before upgrading. Several changes can make a working 2.8.2 study refuse to start or behave differently: duplicate scheme names in one phase are refused, min_annotators_per_instance became an alias for the per-item cap, require_fully_annotated: true is enforced, a phase that fails to load aborts the boot, and span offsets are code points rather than UTF-16 units. Agreement values for set-valued, order-valued, matrix and labelled ordinal schemes change and should be recomputed. The release notes have the full list.

pip install --upgrade potato-annotation

Import and export

• Importers for brat, doccano, Prodigy, CoNLL-2003/CoNLL-U and REFI-QDA .qdpx, and a .qdpx exporter. See Importing a Project.
• Provenance in exports. Each image shape records whether a person drew it or a model proposed it (source, ai_model, confidence, edited, carried_over, import_format). Votes cast in a multiplayer room carry the room, role, initial vote, whether it changed after the reveal, and the voter count.
• Span exports in csv, jsonl and parquet include the covered text.
• python -m potato.export --format adjudication is new, and every schema type now has an "adopt an answer" adjudication control (only radio and likert had one).

Audio, video and transcripts

• Local transcription and speaker diarization with faster-whisper and sherpa-onnx, no PyTorch and no Hugging Face token: install potato-annotation[transcribe], then run potato transcripts <dir> --transcribe --diarize --num-speakers N -o out.json. See Transcript Formats.
• segment_schemes renders any annotation type inside an audio or video segment. It used to be accepted and ignored.

Agreement, quality and AI

• Agreement over time, joined against codebook revisions, with a re-calibration trigger on /admin/iaa.
• A model-output review mode that includes the empty-prediction slice, so recall can be computed.
• ai_budget.cap_usd refuses to start a run it cannot afford, and ai_budget.prices sets per-model rates in the config.
• ai_support records final_annotation and pre_annotation_seeds, so acceptance rates can be measured from your own data.

Deployment and interface

• Reverse-proxy and path-prefix support on every page (#168, Aldo Costa). See Reverse Proxy.
• card_sort works without a mouse, and moves are announced to screen readers.
• resizable, max_height and min_height work on every display type.

Fixes that affect results

• /admin/api/agreement never produced a number, and compared stored values rather than label names, so every categorical schema showed perfect agreement.
• The second annotator on a two-per-item study was sent to the done page, and every answer they then gave was lost.
• AI suggestions were about the wrong item under every assignment strategy except fixed_order.
• Sign-ups were never saved under the default in-memory config, so after a restart anyone could claim an existing username.

Thanks to Aldo Costa (@Eyecatch3r) for #168 and #171, and to @ruthenian8 for reporting and diagnosing #170.

Potato 2.8.2

Released August 2026

2.8.1 did not contain two of the five security fixes its release notes described: the SSRF fix on the media endpoints (GHSA-9394-85jj-2mhj) and the debug-mode hardening (GHSA-r4ww-vc5h-35qf). 2.8.2 ships both. If you upgraded to 2.8.1 for either issue, upgrade again. See the 2.8.2 release notes.

Potato 2.8.1

Released August 2026

A security release. It fixed remote code execution through the live coding agent (GHSA-cm4p-5rfv-x252), an admin config readable without the admin key, and a trace-ingestion webhook that accepted every request. Upgrade if you run Potato anywhere other than localhost, and go straight to 2.8.2, which carries the two fixes 2.8.1 was missing.

Potato 2.8.0

Released August 2026

Potato was a text annotation platform with an image mode. It now covers images, video, gigapixel scans, 3D point clouds, depth maps, robot episodes, generative-video rollouts and vision-language grounding. For each of them it reports chance-corrected agreement.

Upgrade if you installed from PyPI. Wheels up to 2.7.1 declared package data with a single-level glob, so 27 templates in subdirectories were missing from every wheel. Solo mode's setup routes, the admin pages, judge calibration and the corpus map were all broken for anyone who ran pip install; anyone running from a git checkout never saw it. Fixed in PR #164 by @0x6362.

pip install --upgrade potato-annotation

Agreement over geometry and time

• Chance-corrected agreement for spatial labels, decomposed into detection (did annotators find the same objects), classification (did they call them the same thing), and localization (did they put them in the same place, reported as σ against an empirical chance baseline plus a KS test).
• STAPLE for mask consensus. It estimates a latent boundary and a sensitivity and specificity for each rater. On a corpus where two careful annotators are outnumbered three-to-two by noisy ones, majority vote scores Dice 0.846 against truth and STAPLE scores 1.000.
• Exact rotated 3D IoU, so cuboid agreement is correct for drone, handheld and indoor data rather than only level boxes.
• Temporal agreement reported as a sweep across matching tolerances, because agreement at 0.25 s is a different claim from agreement at 2 s.
• Undefined values explain themselves. α is genuinely undefined on a unanimous corpus, and a bare NaN reads as a broken computation.

This also fixes a bug: adjudication compared the keys of an annotation, and image schemas store everything under one key named _data. Every image pair therefore scored 1.0 agreement, so two annotators who agreed on nothing looked unanimous and no image was ever routed for review.

Vision

• Geometry primitives: polylines, ellipses, 2D cuboids, keypoint sets with skeleton topology and COCO visibility flags, and instance-keyed brush masks.
• Click-to-segment in the browser: MobileSAM under ONNX Runtime Web, about 132 ms per click, no GPU and no per-click network call.
• Open-vocabulary text prompting: type a phrase, get every match boxed. Grounding DINO, Apache-2.0, also in the browser.
• Video mask propagation with SAM 2's memory modules, measured at 0.974–0.979 IoU per frame with no decay across the sequence. This one runs server-side by design; the browser has a separate, lighter carry-forward path.
• Deep zoom for gigapixel images, served as both DZI and IIIF Image API 3.0, with brush masks at the source's full resolution.
• Media ingest: multi-page 16-bit TIFF, HEIC, camera RAW, and ProRes/MKV/MOV through a cached server-side proxy.
• 15 import and 29 export formats, 11 of them round-tripping. Darwin works in both directions.

Image annotation keyboard shortcuts now follow V7 conventions by default: b brush, r rectangle, f fill, k keypoint, v select. Set keybinding_profile: legacy on the schema to restore the old bindings for a study already in the field.

3D, depth and robotics

• spatial_annotation for PCD, PLY, LAS, KITTI .bin and .xyz clouds, with 3D cuboids, points, polylines and per-point segments, octree level-of-detail, and orthographic slab panels. Rotation is stored as a quaternion, so the mounting tilt a yaw-only field would silently discard survives a KITTI round-trip.
• Depth maps in 16-bit PNG/TIFF, NPY, PFM and EXR, with windowing, colormaps, a metres readout under the cursor, and unprojection into the same 3D viewer.
• Calibration-aware projection, so a cuboid drawn in 3D appears in every camera image and can be checked in 2D.
• episode_annotation puts N synchronized video streams and M robot time-series lanes on one timeline, with phase segmentation, per-phase outcomes and dense reward curves. Imports LeRobot v2, RLDS/TFDS and HDF5. Lane downsampling preserves minima and maxima, so a one-frame force spike survives the trip to a 300-pixel lane.

Evaluation

• rollout_evaluation shows 2–N generated videos on one clock and asks the annotator to mark the frame at which the world stops making sense, then tag which physical or causal property broke. Panels can be blinded and stably shuffled per annotator.
• grounding_eval and region_caption score grounding by IoU at four thresholds and pointing by point-in-region hit rate, because a point has no area and IoU against one is always zero. Ungroundedness is counted separately.

Everything else

• Threaded conversations: the dialogue display renders reply structure from each turn's reply_to. potato convokit imports any ConvoKit corpus and --format convokit exports back, with no convokit dependency.
• Drawing telemetry: time per shape, stroke dynamics, revision counts and AI-suggestion accept latency. Pre-labels accepted wholesale make agreement rise while accuracy does not, because annotators who accept the same suggestion are unanimous by construction. Timing is a process signal that separates review from rubber-stamping where the labels cannot.
• Live database ingestion: rows created after startup become annotatable within poll_interval_seconds with no restart (#166).
• Machine-readable specs: a JSON Schema covering 159 config keys, 61 annotation types and 24 display types, plus an OpenAPI document with 419 paths. Both are generated from the code and CI fails when either drifts, so an editor or a coding agent can check a config without inventing options.
• No outbound requests: every stylesheet, script, font and icon now serves from the install across all 14 templates. styles.css had been opening with an @import of Google Fonts, which sent every annotator's IP address to a third party on page load, and the login page carried a third-party favicon. Both are gone.
• Admin Instances tab went from quadratic to linear: 2,000 items dropped from 15.1 s to 23 ms.

Potato 2.7.2

Released August 2026

Keystroke logging for free-text fields. Potato records the timing behind an answer and never the characters, which turns out to be enough to tell typing apart from pasting.

• Keystroke logging: content-blind capture of pauses, bursts, revisions and pastes on every free-text field, with about forty summary features computed per response. Off by default.
• Writing-process detection: six named rules with explicit thresholds that report the feature values behind every flag. Thresholds can be refitted to your own project, and there is a supervised path if you have labels.
• Ethics guidance: sample consent language, retention and deletion, the base-rate problem, and why a flag is a prompt to look rather than a finding.
• Opt-in exports: export_include_typing_dynamics writes a summary sidecar, and --format keystrokes writes the raw streams to Parquet. Neither is on by default, so behavioral data never lands in a dataset release by accident.

Disclosure to annotators defaults to on, and turning it off logs a warning at startup.

Potato 2.7.1

Released July 2026

Transcript ingestion, on the premise that the transcript already exists. Someone ran Whisper, or downloaded captions, or was handed a corpus, and the next step should not be writing a conversion script.

• 21 transcript and subtitle formats, up from 6. Whisper, WhisperX, whisper.cpp, AWS Transcribe, Deepgram, AssemblyAI, Rev.ai, SPoRC, SubRip, WebVTT, SubStation Alpha, TTML/DFXP, YouTube json3 and srv1/srv2/srv3, NIST CTM, Praat TextGrid, and ELAN EAF. Detection is by file contents, not extension.
• Sidecar transcript files: a data file can point at media/int_001.srt on disk instead of inlining the transcript. Paths resolve under task_dir through the standard path-security check, so your transcripts stay files you can diff and re-export.
• potato transcripts: a converter that turns a folder of ASR output into an annotation-ready data file and pairs each transcript with its media by basename. --dry-run reports what it detected before writing anything, and --emit-config prints a matching config.
• One format vocabulary across four schemas: speech_transcript, voice_interaction, and tiered_annotation now accept everything the audio_dialogue display accepts. Previously only the display did.
• Annotating Whisper Transcripts and Annotating YouTube Subtitles walk through both starting points end to end.

In 2.7.1 the ASR and diarization steps run upstream and Potato reads their output. Potato 2.9 added local transcription and diarization through potato transcripts --transcribe --diarize, and Think-Aloud Mode has transcribed annotators' own speech locally since 2.7.0.

Potato 2.7.0

Released July 2026

Potato 2.7 measures how judgments are made as well as recording what they were, both for the humans who annotate and for the AI models being evaluated.

Measuring how humans label

Seven opt-in features, none of which requires an LLM.

• Psychometrics engine: a live item-response-theory layer that gives every label a posterior probability and a sensitivity interval (p = 0.94 [0.88–0.97]) instead of a bare majority vote. It estimates annotator ability and item difficulty from agreement patterns alone, routes items to the annotator whose judgment carries the most information, and flags likely codebook bugs.
• Multiplayer Rooms: the calibration meeting your team already runs over screen-share, built into the tool and instrumented: blind vote, host reveal, discussion, revote, with a live Krippendorff's α meter showing what the session was worth.
• Boundary Lab: counterfactual probes after each label ("would that still hold?"), which turn ordinary annotation into contrast sets and catch inconsistent annotators without planting gold items.
• Truth Serum: surprisingly-popular peer-prediction scoring, which beats majority vote where annotation is hard, with no gold labels.
• Think-Aloud Mode: annotators talk while they work; speech-to-text runs fully locally and the verbatim transcript becomes the rationale.
• Paper Mode: one command turns a project into a compilable LaTeX dataset report, agreement statistics and citations included.
• Pocket Mode: first-class mobile annotation with a swipeable card stack, offline sync, and PWA install.

Evaluating AI agents

The agent-evaluation suite grew to cover team structure and multimodal agents: a clickable agent interaction graph, cross-agent failure attribution, handoff review, per-agent and team scorecards, tool-contention timelines, and emergent-behavior tagging, plus GUI/computer-use trajectories, full-duplex voice timelines, video temporal grounding, and document-table structure.

Everywhere else

Cross-document event annotation, turn-level annotation, a living-document codebook with a full-page editor, PDF cross-page linking with optional OCR, RBAC roles with per-cohort schemas, admin and annotator dashboards in 10 languages, and a much lighter install, since AI SDKs now load lazily and boot time drops from roughly 2.0s to 0.7s.

Potato 2.6.0

Released June 2026

Potato 2.6 moves into qualitative data analysis and deepens the agent-evaluation toolkit. It adds QDA Mode, an LLM-as-judge calibration and alignment workflow with a signal-based triage queue, and trajectory-editing schemas that produce SFT and DPO training data. Potato is also relicensed to GPL-3.0-or-later (from PolyForm Shield).

QDA Mode

An opt-in qda_mode turns Potato into a collaborative qualitative-coding workspace. Enabling it composes a living codebook, in-vivo coding, analyst memos, cases, and full-text search, with defaults tuned for one analyst coding a whole corpus.

qda_mode:
  enabled: true            # codebook + memos + cases + search
codebook_invivo_key: i     # mint a code from a text selection
search:
  enabled: true
  annotator_claim: true

Learn more about QDA Mode →

LLM-as-Judge Calibration and Alignment

Auto-label data with one or more LLM judges, then run a blind human calibration pass to measure accuracy, agreement, and calibration error. A separate single-judge alignment workflow tracks Cohen's kappa against your human gold labels as you refine the rubric, with an optional inline verdict during annotation.

Learn more about Judge Calibration → · Judge Alignment →

Signal-Based Triage Queue

Prioritize the annotation queue by a per-item quality signal (an agent error, a production thumbs-down, a low score, or any custom field), so reviewers see the most-suspect items first instead of in arrival order.

triage:
  enabled: true
  signal_field: quality_score
  invert_signal: true
assignment_strategy: priority

Learn more about the Triage Queue →

Trajectory Editing for SFT/DPO

The new trajectory_edit and trajectory_correction schemas let annotators rewrite agent trace steps. The exporter turns each original/corrected pair into supervised fine-tuning targets (trajectory_sft.jsonl) and DPO preference pairs (trajectory_dpo.jsonl).

Learn more about Trajectory Editing →

eval_trace Display

A three-pane agent-trace display (Reasoning, Function Calls, and Final Answer) built for continuous evaluation, where traces arrive over a webhook, a Langfuse poller, or a watched directory and are judged as they land.

Learn more about eval_trace →

Workflow and Assignment

• Heterogeneous coverage: per-item annotator caps, inter-annotator agreement reporting, and adjudication routing for tasks where items need different numbers of annotators.
• Reclaim abandoned assignments: recover assignments left by Prolific or QC-blocked workers, with configurable retention and idempotent reclaim.
• Custom Batch assignment strategy: assign predefined batches of items to specific annotators.
• Reverse-proxy URL prefixes: serve Potato under a sub-path behind a reverse proxy.

Licensing

Potato is now released under GPL-3.0-or-later, relicensed from PolyForm Shield. You can use, modify, and redistribute it, including commercially, as long as derivative works stay under the GPL. See the About page for details.

Performance and Stability

• Roughly 3× faster boot. The ML stack is no longer eager-loaded at startup: import time dropped from about 6.5s to 2s, a 50k-item boot from about 10s to 5.7s, and resident memory from about 750MB to 365MB.
• Schema rename. annotation_type: highlight is now span, with a migration in place. Update old configs by renaming the type; existing span configs are unaffected.
• A broad QA-hardening wave touched route registration, training phases, Prolific, persistence, export and survey handling, webhooks, solo mode, and active learning.

Potato 2.5.0

Released 2026

A qualitative-coding wave that makes Potato credible for qualitative-research workflows alongside its NLP and ML annotation surface.

• Inter-annotator agreement: Cohen's kappa (pairwise) and Fleiss' kappa (N raters) in addition to Krippendorff's alpha, surfaced via the admin agreement API. See the inter-annotator agreement guide.
• New exporters: codebook (per-schema CSV with code hierarchy, color, description, and use count) and quotation_report (per-span CSV with text, offsets, source document, and coder).
• Code analytics: admin endpoints for pairwise code co-occurrence and a codes-by-attribute crosstab over existing instance metadata.

Potato 2.4.5

Released 2026

A stability and tooling release.

• Validated refinement: a pluggable framework for iteratively improving solo-mode annotation guidelines, with guardrails against guideline contradictions and metadata leakage.
• Config validator: a new python -m potato.validate_cli CLI that checks config keys against the known-config schema.
• Security: a fix for a path-traversal sibling-prefix bypass in path validation (GHSA-q9m2-fhv9-3jcf).
• Fixes across save/navigation state-sync, Prolific integration, multi-phase navigation, and quality-control feedback.

Potato 2.3.0

Released March 9, 2026

Potato 2.3 is the largest release in Potato's history, introducing agentic annotation, Solo Mode, Best-Worst Scaling, SSO/OAuth authentication, Parquet export, 15 new demo projects, and security hardening.

Agentic Annotation

A complete system for evaluating AI agents through human annotation. Includes 12 trace format converters, 3 specialized display types, and 9 pre-built annotation schemas.

12 Trace Format Converters: Import agent traces from OpenAI, Anthropic, SWE-bench, OpenTelemetry, MCP, CrewAI/AutoGen/LangGraph, LangChain, LangFuse, ReAct, WebArena/VisualWebArena, ATIF, and raw browser recordings. Auto-detection available.

Conversion runs as a CLI step before the server starts:

python -m potato.trace_converter \
  --input traces.json \
  --input-format react \
  --output data/traces.jsonl

The converted file is then an ordinary data file, rendered by one of the agent display types:

data_files:
  - "data/traces.jsonl"
 
instance_display:
  fields:
    - key: steps
      type: agent_trace
      label: "Agent trajectory"

3 Display Types:

• Agent Trace Display: Color-coded step cards with collapsible observations, JSON pretty-printing, and timeline sidebar for tool-using agents
• Web Agent Trace Display: Full screenshots with SVG overlays showing click targets, text inputs, and scroll actions; filmstrip navigation for browsing agents
• Interactive Chat Display: Live chat mode (annotator interacts with agent via proxy) and trace review mode for conversational agents

Per-Turn Ratings: Rate individual steps alongside the overall trace for fine-grained evaluation.

9 Pre-Built Schemas: agent_task_success, agent_step_correctness, agent_error_taxonomy, agent_safety, agent_efficiency, agent_instruction_following, agent_explanation_quality, agent_web_action_correctness, agent_conversation_quality.

Agent Proxy System: OpenAI, HTTP, and echo proxies for live agent evaluation.

Learn more about Agentic Annotation →

Solo Mode

A workflow where a single human annotator collaborates with an LLM to label an entire dataset, achieving 95%+ agreement with multi-annotator pipelines while requiring only 10-15% of total human labels.

Solo Mode runs as a phase state machine rather than a fixed sequence. It moves through setup, prompt review, an optional edge-case loop, parallel annotation, disagreement resolution, active annotation with periodic and rule review, autonomous labeling and final validation. Five of those phases can return to prompt review when the prompt turns out to be the problem, and final validation falls back to active annotation when the sample does not clear the threshold.

solo_mode:
  enabled: true
  labeling_models:
    - endpoint_type: openai
      model: "gpt-4o"
      api_key: ${OPENAI_API_KEY}
  thresholds:
    end_human_annotation_agreement: 0.92
    minimum_validation_sample: 50
  state_dir: "solo_state"

Multi-Signal Instance Prioritization: instance_selection weights seven signals (low confidence, diversity, LLM-predicted, disagreement, random, cartography, edge-case rule) to choose what the human sees next.

Learn more about Solo Mode →

Best-Worst Scaling

Efficient comparative annotation where annotators select the best and worst items from tuples. Automatic tuple generation with balanced incomplete block designs and three scoring methods (Counting, Bradley-Terry, Plackett-Luce).

annotation_schemes:
  - annotation_type: bws
    name: fluency
    description: "Which sentence reads most fluently?"
    tuple_size: 4
    best_description: "Most Fluent"
    worst_description: "Least Fluent"

Learn more about Best-Worst Scaling →

SSO & OAuth Authentication

Production-ready authentication with Google OAuth (domain restriction), GitHub OAuth (organization restriction), and generic OIDC (Okta, Azure AD, Auth0, Keycloak). Supports auto-registration, mixed mode, and session management.

authentication:
  method: "oauth"
  providers:
    google:
      client_id: ${GOOGLE_CLIENT_ID}
      client_secret: ${GOOGLE_CLIENT_SECRET}
      allowed_domain: "umich.edu"
  user_identity_field: "email"
  auto_register: true
 
secret_key: ${POTATO_SECRET_KEY}

Learn more about SSO & OAuth →

Parquet Export

Export annotations to Apache Parquet format, producing three structured files: annotations.parquet, spans.parquet, and items.parquet. Supports snappy, gzip, zstd, lz4, and brotli compression, incremental export, and date/annotator partitioning. Compatible with pandas, DuckDB, PyArrow, Polars, and Hugging Face Datasets.

export_annotation_format:
  - parquet

Learn more about Parquet Export →

15 New Demo Projects

New demos in project-hub/ covering agentic annotation (5 demos), Solo Mode (3 demos), Best-Worst Scaling (3 demos), authentication (2 demos), and export workflows (2 demos). Start any demo with potato start config.yaml.

Security Hardening

• Cryptographically secure session tokens with configurable expiration
• CSRF protection enabled by default
• Rate limiting on authentication endpoints
• Input sanitization for user-provided content
• Dependency audit with all packages updated
• Content Security Policy headers

Other Improvements

• Custom trace converters for unsupported agent frameworks
• Hybrid Solo Mode with multi-annotator verification sampling
• BWS admin dashboard tab with score convergence charts
• Incremental Parquet export with date partitioning

v2.2 vs v2.3 Comparison

Potato 2.2.0

Released February 20, 2026

Potato 2.2 is a major feature release with 9 new annotation schemas, a pluggable export system, MACE competence estimation, 55 validated survey instruments, and remote data sources.

New Annotation Schemas (9)

Event Annotation: N-ary event structures with trigger spans and typed argument roles. Annotate events like ATTACK, HIRE, and TRAVEL with constrained entity arguments and hub-spoke arc visualization.

annotation_schemes:
  - annotation_type: event_annotation
    name: events
    description: "Mark the event triggers and their arguments"
    span_schema: entities
    event_types:
      - type: "ATTACK"
        trigger_labels: ["EVENT_TRIGGER"]
        arguments:
          - role: "attacker"
            entity_types: ["PERSON", "ORGANIZATION"]
            required: true

Learn more about Event Annotation →

Entity Linking: Link span annotations to external knowledge bases (Wikidata, UMLS, custom REST APIs). Add an entity_linking: block to any span schema to enable KB search and linking.

Learn more about Entity Linking →

Triage: Prodigy-style accept/reject/skip interface for rapid data screening. Customizable labels, keyboard shortcuts, and auto-advance for high-throughput annotation.

Learn more about Triage →

Pairwise Comparison: Compare two items with binary (click preferred tile) or scale (slider) modes. Supports items_key, allow_tie, scale: block with configurable range.

Learn more about Pairwise Comparison →

Conversation Trees: Annotate hierarchical conversation structures with per-node ratings, path selection, and branch comparison.

Learn more about Conversation Trees →

Coreference Chains: Group coreferring text mentions into chains with visual indicators. Supports entity types, singleton control, and multiple highlight modes.

Learn more about Coreference Chains →

Segmentation Masks: New fill, eraser, and brush tools for pixel-level image segmentation.

Bounding Box for PDF/Documents: Draw boxes on PDF pages for document annotation tasks.

Discontinuous Spans: allow_discontinuous: true enables selecting non-contiguous text segments as a single span.

Intelligent Annotation

MACE Competence Estimation: Variational Bayes EM algorithm that jointly estimates true labels and annotator competence scores (0.0-1.0). Works with radio, likert, select, and multiselect schemas.

mace:
  enabled: true
  trigger_every_n: 10
  min_annotations_per_item: 3

Learn more about MACE →

Option Highlighting: LLM-based highlighting of likely correct options for discrete annotation tasks. Highlights top-k options with a star indicator while dimming less-likely options.

ai_support:
  option_highlighting:
    enabled: true
    top_k: 3
    dim_opacity: 0.4

Learn more about Option Highlighting →

Diversity Ordering: Embedding-based clustering and round-robin sampling to ensure annotators see diverse content rather than similar items in sequence.

assignment_strategy: diversity_clustering
diversity_ordering:
  enabled: true
  prefill_count: 100

Learn more about Diversity Ordering →

Export System

A new pluggable export CLI (python -m potato.export) converts annotations to 6 industry-standard formats: COCO, YOLO, Pascal VOC, CoNLL-2003, CoNLL-U, and Segmentation Masks.

python -m potato.export --config config.yaml --format coco --output ./export/

Learn more about Export Formats →

Remote Data Sources

Load annotation data from URLs, S3, Google Drive, Dropbox, Hugging Face, Google Sheets, and SQL databases via the new data_sources: config block. Includes partial loading, caching, and credential management.

Learn more about Remote Data Sources →

Survey Instruments

55 validated questionnaires across 8 categories (Personality, Mental Health, Affect, Self-Concept, Social Attitudes, Response Style, Short-Form, Demographics). Use in prestudy/poststudy phases with instrument: "tipi".

Learn more about Survey Instruments →

Other Improvements

• Video object tracking with keyframe interpolation
• External AI config file support
• Form layout grid improvements
• Format handlers for PDF, Word, code, and spreadsheets

Potato 2.1.0

Released February 5, 2026

Potato 2.1 introduces the instance display system, visual AI support, span linking, multi-field span annotation, and layout customization.

Instance Display System

A new instance_display config block that separates content display from annotation. Display any combination of images, videos, audio, text, and dialogues alongside any annotation schemes.

instance_display:
  fields:
    - key: image_url
      type: image
      display_options:
        max_width: 600
        zoomable: true
    - key: description
      type: text
 
annotation_schemes:
  - annotation_type: radio
    name: category
    description: "What category best describes this image?"
    labels: [nature, urban, people]

Supports 24 display types including text, html, image, video, audio, depth_map, dialogue, conversation_tree, pairwise, code, spreadsheet, document, pdf, gallery, and five agent-trace displays.

Learn more about Instance Display →

Multi-Field Span Annotation

Span annotation schemes now support a target_field option to annotate across multiple text fields in the same instance.

annotation_schemes:
  - annotation_type: span
    name: source_entities
    description: "Label the entities in the source document"
    labels: [PERSON, ORGANIZATION]
 
  - annotation_type: span
    name: summary_entities
    description: "Label the entities in the summary"
    labels: [PERSON, ORGANIZATION]

Learn more about Span Annotation →

Span Linking

A new span_link annotation type for creating typed relationships between annotated spans. Supports directed and undirected links, n-ary relationships, visual arc display, and label constraints.

annotation_schemes:
  - annotation_type: span
    name: entities
    description: "Highlight the named entities in the text"
    labels:
      - name: "PERSON"
        color: "#3b82f6"
      - name: "ORGANIZATION"
        color: "#22c55e"
 
  - annotation_type: span_link
    name: relations
    description: "Link the entities that stand in a relation"
    span_schema: entities
    link_types:
      - name: "WORKS_FOR"
        directed: true
        allowed_source_labels: ["PERSON"]
        allowed_target_labels: ["ORGANIZATION"]
        color: "#dc2626"

Learn more about Span Linking →

Visual AI Support

Four new vision endpoints for AI-powered image and video annotation assistance:

• YOLO: Fast local object detection
• Ollama Vision: Local vision-language models (LLaVA, Qwen-VL)
• OpenAI Vision: GPT-4o cloud vision
• Anthropic Vision: Claude with vision

Features include object detection, pre-annotation, classification, hints, scene detection, keyframe detection, and object tracking.

Learn more about Visual AI Support →

Layout Customization

Create sophisticated custom visual layouts using HTML templates and CSS. Potato generates an editable layout file, or you can provide a fully custom template with grid layouts, color-coded options, and section styling.

task_layout: layouts/custom_task_layout.html

Three example layouts included: content moderation, dialogue QA, and medical review.

Learn more about Layout Customization →

Label Rationales

A fourth AI capability that generates balanced explanations for why each label might apply, helping annotators understand different classification perspectives.

ai_support:
  features:
    rationales:
      enabled: true

Learn more about AI Support →

Other Improvements

• 50+ new tests for improved reliability
• Responsive design improvements
• Enhanced project-hub organization with layout examples
• Bug fixes across annotation types

v2.0 vs v2.1 Comparison

Potato 2.0

Potato 2.0 is a major release that adds AI assistance, active learning, training phases, multi-phase workflows, and a MySQL backend.

AI Support

Integrate Large Language Models to assist annotators with intelligent hints, keyword highlighting, and label suggestions.

Supported providers:

• OpenAI (GPT-4, GPT-3.5)
• Anthropic (Claude 3, Claude 3.5)
• Google (Gemini)
• Ollama (local models)
• vLLM (self-hosted)

ai_support:
  enabled: true
  endpoint_type: openai
  ai_config:
    model: gpt-4
    api_key: ${OPENAI_API_KEY}
  features:
    hints:
      enabled: true
    label_suggestions:
      enabled: true

Learn more about AI Support →

Audio Annotation

Full-featured audio annotation with waveform visualization powered by Peaks.js. Create segments, label time regions, and annotate speech with keyboard shortcuts.

Key features:

• Waveform visualization
• Segment creation and labeling
• Per-segment annotation questions
• 15+ keyboard shortcuts
• Server-side waveform caching

annotation_schemes:
  - annotation_type: audio_annotation
    name: speakers
    description: "Mark each speaker turn in the audio"
    mode: label
    labels:
      - Speaker A
      - Speaker B

Learn more about Audio Annotation →

Active Learning

Automatically prioritize annotation instances based on model uncertainty. Train classifiers on existing annotations and focus annotators on the most informative examples.

Capabilities:

• Multiple classifier options (LogisticRegression, RandomForest, SVC, MultinomialNB)
• Various vectorizers (TF-IDF, Count, Hashing)
• Model persistence across restarts
• LLM-enhanced selection
• Multi-schema support

active_learning:
  enabled: true
  schema_names:
    - sentiment
  min_instances_for_training: 30
  update_frequency: 50
  classifier:
    type: LogisticRegression

Learn more about Active Learning →

Training Phase

Qualify annotators with practice questions before the main task. Provide immediate feedback and ensure quality through configurable passing criteria.

Features:

• Practice questions with known answers
• Immediate feedback and explanations
• Configurable passing criteria
• Retry options
• Progress tracking in admin dashboard

phases:
  training:
    enabled: true
    data_file: "data/training.json"
    passing_criteria:
      min_correct: 8
      total_questions: 10

Learn more about Training Phase →

Enhanced Admin Dashboard

Comprehensive monitoring and management interface for annotation tasks.

Dashboard tabs:

• Overview: High-level metrics and completion rates
• Annotators: Performance tracking, timing analysis
• Instances: Browse data with disagreement scores
• Configuration: Real-time settings adjustment

admin_api_key: ${ADMIN_API_KEY}

Learn more about Admin Dashboard →

Database Backend

MySQL support for large-scale deployments with connection pooling and transaction support.

database:
  type: mysql
  host: localhost
  database: potato_db
  username: ${DB_USER}
  password: ${DB_PASSWORD}

Potato automatically creates required tables on first startup.

Annotation History

Complete tracking of all annotation changes with timestamps, user IDs, and action types. Enables auditing and behavioral analysis.

{
  "history": [
    {
      "timestamp": "2024-01-15T10:30:00Z",
      "user": "annotator_1",
      "action": "create",
      "schema": "sentiment",
      "value": "Positive"
    }
  ]
}

Multi-Phase Workflows

Build complex annotation workflows with multiple sequential phases:

1. Consent - Informed consent collection
2. Pre-study - Demographics and screening
3. Instructions - Task guidelines
4. Training - Practice questions
5. Annotation - Main task
6. Post-study - Feedback surveys

phases:
  consent:
    enabled: true
    data_file: "data/consent.json"
  prestudy:
    enabled: true
    data_file: "data/demographics.json"
  training:
    enabled: true
    data_file: "data/training.json"
  poststudy:
    enabled: true
    data_file: "data/feedback.json"

Learn more about Multi-Phase Workflows →

v2.0 Configuration Changes

New Configuration Structure

Potato 2.0 uses a cleaner configuration format:

v1 (old):

data_files:
  - data.json
id_key: id
text_key: text
output_file: annotations.json

v2 (new):

data_files:
  - "data/data.json"
 
item_properties:
  id_key: id
  text_key: text
 
output_annotation_dir: "output/"
export_annotation_format: "jsonl"

Security Requirement

Configuration files must now be located within the task_dir:

With the config beside the data, the task directory is the current one:

task_dir: "."

With the config in a subdirectory, name the project directory it belongs to:

task_dir: "my_project/"

Quick Comparison

Migration Guide

Updating Your Configuration (v1 to v2)

1. Data configuration

   yamlCopy

   # Old
   id_key: id
   text_key: text
    
   # New
   item_properties:
     id_key: id
     text_key: text

2. Output configuration

   yamlCopy

   # Old
   output_file: annotations.json
    
   # New
   output_annotation_dir: "output/"
   export_annotation_format: "jsonl"

3. Config file location Ensure your config file is inside the project directory.

Starting the Server

# v2 command
python -m potato start config.yaml -p 8000
 
# Or shorthand
potato start config.yaml

Getting Started

Ready to try Potato? Start with the Quick Start Guide or explore specific features:

v2.3 Features:

• Agentic Annotation - Evaluate AI agents with trace converters and step-level displays
• Solo Mode - Human-LLM collaborative labeling
• Best-Worst Scaling - Comparative annotation with scoring
• SSO & OAuth - Google, GitHub, and OIDC authentication
• Parquet Export - Columnar data export

v2.2 Features:

• Event Annotation - N-ary event structures
• Entity Linking - Knowledge base linking
• Triage - Rapid data screening
• Coreference Chains - Entity coreference
• Conversation Trees - Hierarchical dialogue annotation
• MACE - Annotator competence estimation
• Option Highlighting - AI-assisted option guidance
• Diversity Ordering - Embedding-based item ordering
• Export Formats - Export CLI with 6 formats
• Remote Data Sources - Cloud data loading
• Survey Instruments - 55 validated questionnaires

v2.1 Features:

• Instance Display - Multi-modal content display
• Visual AI Support - AI for image and video annotation
• Span Linking - Entity relationship annotation

Core Features:

• AI Support - Intelligent annotation assistance
• Active Learning - Smart instance prioritization
• Audio Annotation - Waveform-based annotation
• Training Phase - Annotator qualification
• Admin Dashboard - Monitoring and management