# Text Annotation Tool for NLP, Open Source and Self-Hosted

Source: https://www.potatoannotator.com/use-cases/text-annotation

**Potato is an open-source text annotation tool for NLP research, with span labels for named entities, document classification, relations, and coreference, several annotators per item, and agreement measured as the study runs. Potato installs with pip, runs on your own computer or server, and exports CoNLL, JSONL, or a Hugging Face dataset.**

Potato comes from NLP research at the University of Michigan, and text is the kind of data Potato has supported longest. The design choices follow from how research groups annotate. A task is a config file that can sit in version control next to the paper, several annotators can label the same items so agreement can be measured, and the data stays on a machine you control. Potato 2.0 was presented at [ACL 2026](https://aclanthology.org/2026.acl-demo.37/), and the original version at [EMNLP 2022](https://aclanthology.org/2022.emnlp-demos.33/).

## NLP tasks in Potato

Each row is an annotation type that ships with Potato, and the linked page has its options.

| Task | Potato annotation type | Docs |
|---|---|---|
| Named entity recognition and other spans | `span`, with overlapping spans allowed | [Span annotation](/docs/annotation-types/span-annotation) |
| Text classification, single or multi-label | `radio`, `multiselect`, and `hierarchical_multiselect` for taxonomies | [Radio and multiselect](/docs/annotation-types/radio-multiselect) |
| Relation extraction | Links between spans with typed relations | [Span linking](/docs/annotation-types/span-linking) |
| Coreference | Chains of mentions that refer to the same entity | [Coreference chains](/docs/annotation-types/coreference) |
| Entity linking | Spans linked to Wikidata or UMLS identifiers | [Entity linking](/docs/annotation-types/entity-linking) |
| Event extraction | Triggers and arguments in the style of ACE | [Event annotation](/docs/annotation-types/event-annotation) |
| Extractive question answering | Answer spans for a question, as in SQuAD | [Extractive QA](/docs/annotation-types/extractive-qa) |
| Ratings and free text | Likert scales, sliders, and text boxes | [Text annotation guide](/docs/guides/text-annotation) |

## Entities and a document label in one task

A task can ask several questions about the same text. The config below asks annotators to mark entities and give the whole document a sentiment label, with two annotators per item so their agreement can be measured.

```yaml
annotation_task_name: "News entities and sentiment"
task_dir: "."
data_files:
  - data/articles.jsonl
item_properties:
  id_key: id
  text_key: text
output_annotation_dir: annotation_output/
user_config:
  allow_all_users: true
num_annotators_per_item: 2
agreement_metrics:
  enabled: true
  min_overlap: 2
annotation_schemes:
  - annotation_type: span
    name: entities
    description: "Highlight every person, organization and location"
    labels: [PERSON, ORGANIZATION, LOCATION]
  - annotation_type: radio
    name: sentiment
    description: "What is the overall sentiment of the article?"
    labels: [positive, neutral, negative]
    label_requirement:
      required: true
```

`agreement_metrics` reports Krippendorff's alpha for each scheme on the admin dashboard once items have two annotations. A second annotator on every item doubles the cost, so a common compromise is two annotators on a sample of a few hundred items and one on the rest. [How many annotators do you need?](/docs/guides/how-many-annotators) works through the trade-off.

## Bringing data in and getting labels out

Data files are JSON, JSONL, CSV, or TSV with one item per row. An existing project can come across with `potato import`, which reads Label Studio, Argilla, brat, doccano, Prodigy, CoNLL-2003, CoNLL-U, and TEI exports and writes a runnable project with the scheme already declared.

Labels go out in the formats NLP training code reads. The `conll_2003` and `conll_u` exporters write token-level tags, `jsonl` keeps the full answer structure, `parquet` suits large datasets, and `huggingface` pushes a dataset to the Hub. [Exporting annotations for ML](/docs/guides/exporting-annotations-for-ml) shows how to load each one for training.

## Working alongside Label Studio, Prodigy, INCEpTION, and doccano

Each of these tools is built around a different setting. [Prodigy](https://prodi.gy/) is a paid, scriptable tool built around one annotator working quickly with a model in the loop. [INCEpTION](https://inception-project.github.io/) does multi-layer text annotation with knowledge-base linking. [Label Studio](https://labelstud.io/) covers many data types in one interface, and [doccano](https://github.com/doccano/doccano) is a small open-source tool for classification and sequence labeling. Potato is built around the research study, with several annotators per item, agreement measured as you go, quality checks such as attention items and gold questions, and a config that reproduces the task. [Text annotation tools compared](/docs/guides/text-annotation-tools-compared) goes through the differences feature by feature.

## Frequently asked questions

### Is Potato an open-source, self-hosted text annotation tool?

Yes. Potato is released under the GPL-3.0 license, installs with `pip install potato-annotation`, and runs on your own computer or server. Annotations are written to your own disk, under the output directory in the config, and nothing is sent to a third party unless you configure an AI endpoint that does.

### How do I measure inter-annotator agreement on NER annotations?

Assign each item to at least two annotators with `num_annotators_per_item`, and turn on `agreement_metrics`, which reports Krippendorff's alpha per scheme on the admin dashboard. Span annotations have their own complications, such as partial overlaps and unmatched spans, which [Agreement for spans](/docs/guides/agreement-for-spans) explains, and the [span and box agreement calculator](/tools/span-and-box-agreement) computes span-level agreement from an export in the browser.

### Can a model pre-annotate the text so annotators only correct it?

Yes. With `ai_support`, a model proposes labels or highlights likely options, and the annotator confirms or corrects each suggestion, so only the verified label becomes data. Potato connects to OpenAI, Anthropic, Gemini, and Hugging Face endpoints, and to local models through Ollama or vLLM when the text must not leave your network. [LLM pre-annotation](/docs/guides/llm-pre-annotation) covers the setup.

### What format are span labels exported in?

Span labels export as CoNLL-2003 or CoNLL-U token tags, as JSONL with character offsets, as Parquet, or as a Hugging Face dataset. [Export formats](/docs/features/export-formats) lists all 60 formats Potato writes.

## Further reading

- [Named entity recognition](/docs/guides/named-entity-recognition), for choosing a label set and running an NER task
- [Writing annotation guidelines](/docs/guides/writing-annotation-guidelines), for the document annotators work from
- [Inter-annotator agreement](/docs/guides/inter-annotator-agreement), for which statistic to report and how to read it
- [LLM evaluation](/use-cases/llm-evaluation) and [qualitative research](/use-cases/qualitative-research), for related fields on this site