Skip to content
Announcements12 min read

potato-skill: Build an Annotation Study with Claude Code, Codex, or Cursor

potato-skill gives a coding agent 28 reference files for Potato, so it designs the annotation study, builds the interface, and checks that it renders.

Potato Team

potato-skill turns a plain description of an annotation study into a running Potato task. It ships 28 reference files and nine helper scripts covering the design decisions that come before the config, the interface the annotator sees, and the checks that catch a task that validates but does not work. It installs into Claude Code, Codex, and Cursor, and it is released under the GPL-3.0-or-later.

Installing the skill

Claude Code installs it from a plugin marketplace, in two commands:

text
/plugin marketplace add davidjurgens/potato-skill
/plugin install potato-skill@potato

Codex and Cursor install it with the skills CLI, which copies the skill into the project you are working in:

bash
npx skills add davidjurgens/potato-skill --agent codex cursor

Potato itself has to be installed wherever the agent runs commands, because the nine helpers import its registries and drive the potato command:

bash
pip install potato-annotation

What a Codex or Cursor install includes

The skills CLI writes .agents/skills/potato-skill/ with SKILL.md, all 28 references, and all nine scripts, so a Codex or Cursor project gets the same material a Claude Code install gets. It leaves AGENTS.md and .cursor/rules untouched, and how each tool discovers a skill in that directory is up to the tool.

The repository also carries an AGENTS.md at its root, which Codex and Cursor read by filename with no install step. That file is a short summary against roughly 79,000 words across the references, and it points at reference files and scripts that only a full install puts on disk. Use it when you want an agent to have the check-before-handoff loop and the rules that bite, and use the skills CLI when you want the reference material behind them.

Defaults the skill sets without asking

A decision is made locally when being wrong is cheap to fix, and handed back when being wrong is expensive, irreversible, or not the agent's to make. Changing a keyboard shortcut costs nothing, while re-running 5,000 items because the unit of annotation was wrong costs the study's budget. These are the defaults it takes, each of which it lists as an assumption when it hands the task over:

DecisionDefault
Scale points5, with every point labeled, unless you named a number
Keyboard shortcutson any question with nine labels or fewer
Question orderthe gating question first, follow-ups behind display_logic
Item orderrandom, with a fixed random_seed
Required fieldsall of them, plus require_fully_annotated: true
Instructions pagea draft written from your description, marked as a draft

Decisions it hands back to you

Five decisions go back to the researcher in one batched message, each with a proposed answer so it can be accepted in a word:

  • The unit of annotation, when the data does not settle it. A file of paragraphs where the question is really about sentences is a fork in the study, and either choice can mean re-running it.
  • Annotators per item, when agreement is going to be reported. Three annotators cost three times one, so the skill proposes a number with its reason rather than spending the budget for you.
  • Consent wording. It drafts from what you described and says plainly that the draft needs replacing with the text your ethics board approved.
  • A label set with nowhere to put the hard cases. Three labels with no "unclear" and no "none" is usually an oversight, and annotators then put genuinely ambiguous items somewhere arbitrary, where nothing downstream can see it happened. The skill names the items it thinks have nowhere to go.
  • Anything showing usernames, faces, or locations to annotators.

None of this blocks the build. The skill states its assumptions, builds the task, verifies it, and asks the questions alongside something you can already click through.

Checks before handoff

A config that validates tells you the server will start, and tells you nothing about whether a person can do the work. So the skill instructs the agent to render the task and look at it:

bash
potato validate config.yaml --strict
potato preview config.yaml --screenshot shot-01.png

--strict turns an unrecognized config key from a warning into a failure. Without it a typo is accepted, ignored, and the feature you thought you enabled stays off, which is the usual cause of a config that reads correctly while nothing happens.

--screenshot boots a real Potato server and drives it in headless Playwright, recording console errors, uncaught page exceptions, and HTTP responses of 400 or above. It exits 0 only when all three lists are empty. That step catches what validation cannot, because most of the annotation interface is built by JavaScript after the HTML arrives. An invisible radio button is the standard case, since the label text still renders and the question looks right at a glance.

The cycle around those two commands is guidance rather than automation. The reference material tells the agent to make one change per round, write each render to a new filename so the pair shows what the change did, and expect three or four rounds on a custom layout. It also tells the agent that anything behind display_logic is absent from the first render and that its absence proves nothing, so the condition is commented out, rendered, and put back.

Three of the nine helpers automate the parts that can be measured rather than judged. check_ui.py measures the live layout at 1280x900 and reports schemes or a Next button below the fold, empty media widgets, and duplicate shortcuts. boot_and_check.py starts the server and reports every feature that is configured but loaded nothing, such as a training round that loaded zero training items. walk_task.py walks a running task the way an annotator would and reports where it stops.

Safe and unsafe changes after annotators start

Most of what goes wrong in an annotation study goes wrong after annotators have started, and the failure modes that matter produce no error message. The skill carries two of them.

Renaming a question mid-study is the sharpest. Nothing errors, and three surfaces then report three different things. The admin overview says the study is 100% complete, the agreement report says that scheme has zero items, and the CSV export carries the old column name, because exports are driven by what was stored rather than by the config. A single WARNING in the boot log is the entire safety net. Adding a question has a quieter version of the same problem, because Potato tracks completion per item and never per question, so a question added later never reaches anyone who already finished the corpus.

Authentication is the second. The default backend keeps accounts in memory, so stopping the server takes every annotator login with it while the annotations stay safely on disk. An annotator comes back, re-registers under the same username, and is reattached to their work, which hides the problem until someone types a different username and starts the corpus again as a second person. Setting authentication.user_config_path writes accounts out with a salted hash. It matters most where it is easiest to forget, since Render and Hugging Face Spaces restart containers on their own.

For a study already in flight, study_status.py reads the admin routes for progress, per-annotator rates, agreement on each question, who is failing the attention checks, and who is sitting on items they abandoned. The skill then says which fixes are still safe and which would corrupt answers you have already paid for.

Potato's MCP servers

Potato ships two MCP servers, which give an agent the same answers without shelling out. potato mcp serve --root . exposes 12 authoring tools that read the same registries as the CLI, including render_task_screenshot, which returns the rendered page as an image. potato mcp connect bridges a running study and adds 12 live tools covering status, progress, annotators, agreement, assignment, and export. The bridge needs an mcp block in the config listing the tools to expose, and a token from potato mcp issue-token.

Hand-written labeling tools and what they cost

A common way to get a few hundred items labeled is to ask a coding agent for a quick labeling tool, such as a Streamlit or Flask page that shows one item at a time and appends each answer to a CSV. That is a reasonable thing to build when one person labels an afternoon's worth of data once. The cost arrives when the study needs a login per annotator, three annotators per item with each item routed to the right people, a training round and attention checks, agreement statistics, a completion code for Prolific, an export in the format the next script expects, or a second study next month that works the same way. Each of those is a feature someone writes, tests, and debugs inside a tool built for one study, and Potato has all of them already.

The two also differ in what survives the study. A hand-written labeling tool is usually tied to one dataset in one repository, while a Potato task is a folder holding a config and data that anyone with Potato can start. The showcase holds more than 400 of them, most taken from published papers, and find_design.py searches it for a design close to the one you described.

Studies the skill sets up

Five cases show the range, each starting from a short description and ending in a runnable task:

Tests against Potato's registries

A plausible wrong identifier is worse than no documentation, because an agent will use it. Three of the 28 references are therefore generated from Potato's own registries and cannot drift from what the server enforces: all 61 annotation types, each with a worked example lifted from a real project, the documented top-level config keys, and the documented sub-keys.

CI covers the skill's core files, checking that every annotation type, display type, config key, operator, strategy name, and command named in them exists in Potato. Every YAML sample is spliced into a working config and run through Potato's real validator, and the worked example boots a real server where each feature it switches on has to report a non-zero count in the log.

The reference material is published at davidjurgens.github.io/potato-skill, so you can read what the skill tells the agent before installing it. Anthropic recommends reviewing any skill first.

Questions

The questions below cover choosing between potato-skill and a hand-built labeling tool, what the skill checks before handoff, and how to change the interface it builds.

Should a coding agent build a custom annotation tool or use an existing one?

Build a custom tool only when one person labels a few hundred items once. A Streamlit or Flask page that shows one item at a time and appends each answer to a CSV covers that case. Once the study needs several annotators per item, a training round, attention checks, agreement statistics, or a Prolific completion code, each of those has to be written and tested inside a tool built for one study, and Potato already has all of them. potato-skill builds on Potato for that reason, and the task it produces is a folder that anyone with Potato can rerun.

What does potato-skill verify before handing a task over?

potato-skill has the agent run two Potato commands before it hands a task back, and three of its helper scripts check what those commands cannot. potato validate config.yaml --strict fails on any unrecognized config key. potato preview config.yaml --screenshot boots a real server in headless Playwright and exits 0 only when the page produced no console errors, no uncaught exceptions, and no HTTP responses of 400 or above. check_ui.py reports annotation questions or a Next button below the fold, boot_and_check.py reports features that are configured but loaded nothing, and walk_task.py walks the task the way an annotator would and reports where it stops.

Can I customize the interface potato-skill builds?

Yes. The task is an ordinary Potato config, so you can edit the layout by hand or ask the agent to change it. For a custom layout, potato-skill tells the agent to make one change per round, write each render to a new screenshot file so consecutive renders show what the change did, and expect three or four rounds. Anything behind display_logic is missing from the first render, so the agent comments the condition out, renders it, and puts it back.

Further reading

References

David Jurgens, Michael Chen, and Lina Iyer (2026). Potato 2.0: A Comprehensive Annotation Platform with AI-in-the-Loop Support. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations). https://aclanthology.org/2026.acl-demo.37/