Skip to content
Announcements7 min read

potato-skill: Build a Human Annotation Study with Claude Code

potato-skill lets Claude Code design a human annotation study from a plain description, then build and run it as a Potato task with its own interface and checks.

Potato Team
Эта страница пока недоступна на вашем языке. Показана английская версия.

potato-skill is an agent skill for getting data labelled by people. You describe the study to Claude Code, and it designs the task, builds it in Potato with its own annotation interface, starts it, and exports the labels in the format you need. It is free and open source, and it installs with two commands:

text
/plugin marketplace add davidjurgens/potato-skill
/plugin install potato-skill@potato

Setup details, including other agents, are on the Quick Start with a Coding Agent page.

Coding agents and one-off labelling tools

A common way to get a few hundred items labelled is to ask Claude Code, Codex or Cursor to write a quick annotation tool: a Streamlit or Flask page that shows one item at a time and saves each answer to a CSV. That is fine for one person labelling an afternoon's worth of data. It gets harder once the study needs any of these:

  • a login for each annotator, so you know who labelled what
  • three annotators per item, with each item shown to the right people
  • instructions, a training round and attention checks
  • agreement between annotators
  • a completion code for Prolific, or payment through MTurk
  • export in the format the next script expects
  • a second study next month that works the same way

Each of those is a feature someone has to write, test and debug, in a tool built for one study. Potato already has all of them. With potato-skill, the agent spends its effort on the design of the study and builds the task on Potato.

The result is also something other people can use. A hand-written labelling app is usually tied to one dataset in one repository. A Potato task is a folder with a config and data that anyone with Potato can start, and the showcase holds more than 400 of them, most taken from published papers.

Design decisions the skill makes

Given a description of what needs labelling, potato-skill works through the design before any YAML is written: what to ask annotators and on what scale, how many annotators each item needs, and how to train and check them. It then builds the task in Potato, starts it, goes through it the way an annotator would, and exports the results in the format the next step needs.

It also handles studies that are already running: checking progress and agreement, and changing a task after annotators have started.

The data can be text, images, audio, video, dialogue or AI agent traces. The annotators can be crowdworkers, students or domain experts.

Six studies it can set up

Stance labels from crowdworkers on Prolific

A research team wants 2,000 social media posts labelled for stance toward a policy by crowdworkers on Prolific. potato-skill drafts the labels and instructions, adds a short training round and attention checks, has three people label each post, sets up the Prolific completion code, and reports agreement once the labels are in. See the crowdsourcing guide and measuring inter-annotator agreement.

Comparing two chatbot responses

An ML team wants people to compare two chatbot responses to the same prompt, pick the more helpful one and say why. It builds a side-by-side comparison with a preference scale and a free-text reason, and exports the judgements for analysis. See pairwise model comparison and RLHF preference data.

Marking the failing step in agent traces

A lab wants experts to read multi-step AI agent traces and mark the step where the agent went wrong. Each trace is laid out one step at a time, and the task records which step was marked and the annotator's explanation. See agent trajectory annotation.

Continuing a CVAT bounding-box project

A computer vision group has a half-finished CVAT project and wants more bounding boxes drawn on new images. It imports the existing boxes, sets up the drawing task with the same labels, and exports everything back out as COCO. See measuring agreement on bounding boxes.

Two coders and a codebook

A social scientist has a codebook for open-ended survey responses and wants two coders to apply it independently, then settle the cases where they disagree. The codebook becomes the coding interface, both coders are assigned every response, and an adjudication step handles the disagreements. See adjudication and disagreement.

A study where agreement looks low

A study has been running for a week and the researcher thinks agreement looks low. The skill reports progress for each annotator and agreement for each question, and says which fixes are safe while annotators are still working and which would corrupt the data already collected.

Tests against Potato's registries

potato-skill's CI checks that every annotation type, config key and sample config its instructions name exists in Potato. Sample YAML goes through Potato's own validator, and the worked example boots a real Potato server. Three of the references are generated from Potato's registries, so the annotation types and keys the agent reads are the ones the server enforces.

The reference material is published at davidjurgens.github.io/potato-skill, so you can read what the skill tells the agent before you install it.

Other coding agents

The skills CLI installs potato-skill for Codex, Cursor and other agents:

bash
npx skills add davidjurgens/potato-skill --agent codex cursor

That copies the skill into .agents/skills/potato-skill/. Codex and Cursor can also read the AGENTS.md file at the root of the repository. Whichever agent you use, Potato has to be installed where it runs commands (pip install potato-annotation), because the skill's helper scripts import Potato and call the potato command.

Questions

Do I need to know Potato first?

No. You describe the study in plain language. Potato needs to be installed on the machine where the agent runs. Before annotators see the task, open it in a browser and label a few items yourself.

Should I have my coding agent build a custom annotation tool instead?

For a few hundred items that one person labels once, a small script can be enough. Once the study needs several annotators per item, training, attention checks, agreement or a crowdsourcing platform, each of those needs building and testing. potato-skill builds on Potato, which already has them, and the task it produces can be rerun and shared.

Can I edit what it builds?

Yes. The output is an ordinary Potato task. You can change it by hand, using the Quick Start and Configuration Basics, or ask the agent to change it.

Does it work in Codex or Cursor?

The skills CLI installs it into both, and both can read the repository's AGENTS.md. The skill was written for Claude Code.

What does it cost?

Nothing. Potato is free and open source, and potato-skill is released under the GPL-3.0-or-later. You pay for the coding agent you already use, and for annotators if you recruit them.

What is Potato?

Potato is an open-source annotation tool from the University of Michigan, described in an ACL 2026 system demonstration. It supports 61 annotation types across text, images, audio, video, dialogue and agent traces.

Get started