Potato is an open-source text annotation tool for NLP research, with span labels for named entities, document classification, relations, and coreference, several annotators per item, and agreement measured as the study runs. Potato installs with pip, runs on your own computer or server, and exports CoNLL, JSONL, or a Hugging Face dataset.
Potato comes from NLP research at the University of Michigan, and text is the kind of data Potato has supported longest. The design choices follow from how research groups annotate. A task is a config file that can sit in version control next to the paper, several annotators can label the same items so agreement can be measured, and the data stays on a machine you control. Potato 2.0 was presented at ACL 2026, and the original version at EMNLP 2022.
NLP tasks in Potato
Each row is an annotation type that ships with Potato, and the linked page has its options.
| Task | Potato annotation type | Docs |
|---|---|---|
| Named entity recognition and other spans | span, with overlapping spans allowed | Span annotation |
| Text classification, single or multi-label | radio, multiselect, and hierarchical_multiselect for taxonomies | Radio and multiselect |
| Relation extraction | Links between spans with typed relations | Span linking |
| Coreference | Chains of mentions that refer to the same entity | Coreference chains |
| Entity linking | Spans linked to Wikidata or UMLS identifiers | Entity linking |
| Event extraction | Triggers and arguments in the style of ACE | Event annotation |
| Extractive question answering | Answer spans for a question, as in SQuAD | Extractive QA |
| Ratings and free text | Likert scales, sliders, and text boxes | Text annotation guide |
Entities and a document label in one task
A task can ask several questions about the same text. The config below asks annotators to mark entities and give the whole document a sentiment label, with two annotators per item so their agreement can be measured.
annotation_task_name: "News entities and sentiment"
task_dir: "."
data_files:
- data/articles.jsonl
item_properties:
id_key: id
text_key: text
output_annotation_dir: annotation_output/
user_config:
allow_all_users: true
num_annotators_per_item: 2
agreement_metrics:
enabled: true
min_overlap: 2
annotation_schemes:
- annotation_type: span
name: entities
description: "Highlight every person, organization and location"
labels: [PERSON, ORGANIZATION, LOCATION]
- annotation_type: radio
name: sentiment
description: "What is the overall sentiment of the article?"
labels: [positive, neutral, negative]
label_requirement:
required: trueagreement_metrics reports Krippendorff's alpha for each scheme on the admin dashboard once items have two annotations. A second annotator on every item doubles the cost, so a common compromise is two annotators on a sample of a few hundred items and one on the rest. How many annotators do you need? works through the trade-off.
Bringing data in and getting labels out
Data files are JSON, JSONL, CSV, or TSV with one item per row. An existing project can come across with potato import, which reads Label Studio, Argilla, brat, doccano, Prodigy, CoNLL-2003, CoNLL-U, and TEI exports and writes a runnable project with the scheme already declared.
Labels go out in the formats NLP training code reads. The conll_2003 and conll_u exporters write token-level tags, jsonl keeps the full answer structure, parquet suits large datasets, and huggingface pushes a dataset to the Hub. Exporting annotations for ML shows how to load each one for training.
Working alongside Label Studio, Prodigy, INCEpTION, and doccano
Each of these tools is built around a different setting. Prodigy is a paid, scriptable tool built around one annotator working quickly with a model in the loop. INCEpTION does multi-layer text annotation with knowledge-base linking. Label Studio covers many data types in one interface, and doccano is a small open-source tool for classification and sequence labeling. Potato is built around the research study, with several annotators per item, agreement measured as you go, quality checks such as attention items and gold questions, and a config that reproduces the task. Text annotation tools compared goes through the differences feature by feature.
Frequently asked questions
Is Potato an open-source, self-hosted text annotation tool?
Yes. Potato is released under the GPL-3.0 license, installs with pip install potato-annotation, and runs on your own computer or server. Annotations are written to your own disk, under the output directory in the config, and nothing is sent to a third party unless you configure an AI endpoint that does.
How do I measure inter-annotator agreement on NER annotations?
Assign each item to at least two annotators with num_annotators_per_item, and turn on agreement_metrics, which reports Krippendorff's alpha per scheme on the admin dashboard. Span annotations have their own complications, such as partial overlaps and unmatched spans, which Agreement for spans explains, and the span and box agreement calculator computes span-level agreement from an export in the browser.
Can a model pre-annotate the text so annotators only correct it?
Yes. With ai_support, a model proposes labels or highlights likely options, and the annotator confirms or corrects each suggestion, so only the verified label becomes data. Potato connects to OpenAI, Anthropic, Gemini, and Hugging Face endpoints, and to local models through Ollama or vLLM when the text must not leave your network. LLM pre-annotation covers the setup.
What format are span labels exported in?
Span labels export as CoNLL-2003 or CoNLL-U token tags, as JSONL with character offsets, as Parquet, or as a Hugging Face dataset. Export formats lists all 60 formats Potato writes.
Further reading
- Named entity recognition, for choosing a label set and running an NER task
- Writing annotation guidelines, for the document annotators work from
- Inter-annotator agreement, for which statistic to report and how to read it
- LLM evaluation and qualitative research, for related fields on this site