Text Annotation Tools Compared: Open-Source and Paid NLP Tools

A text annotation tool shows documents to annotators and records what they mark: a label for the whole text, highlighted spans such as named entities, and links between spans. Potato, INCEpTION, doccano and Label Studio's Community Edition are free to self-host, and all of them handle named-entity recognition and relations. The tools differ most once a project has many annotators. Potato recruits them through Prolific and MTurk, trains and screens them before the real task, and reports chance-corrected span agreement across all of them, with no paid plan. INCEpTION is ahead on knowledge-base linking, and Prodigy on training a spaCy model while you annotate. brat and Argilla are no longer adding features.

Most text tasks take one of three forms. Text classification gives the whole document a label. Named-entity recognition and other span tasks mark stretches of text. Relation extraction and coreference resolution link spans to each other. Every tool on this page handles spans, and linking is where they differ most.

This page covers text only. Annotation Tools Compared covers every data type on one page.

Which text annotation tools are worth comparing?

Capabilities side by side

On the annotation itself, the five maintained tools are close:

They are not close on running a study with many annotators:

A dash means we found nothing for it in the tool's documentation, pricing page or repository in September 2026. If a cell is wrong, tell us.

Where Potato is ahead for text

The study around the labels. A lot of text annotation is research with crowdworkers: sentiment, stance, offensive language, the quality of model output. Participants arrive from Prolific or MTurk and leave with a completion code. Before the real task they give consent, answer screening or demographic questions, and work through practice items with feedback until they pass. Gold items and attention checks then run alongside the real ones. Potato sets all of this up from its YAML config, and its Prolific integration reassigns items that workers abandoned and can pause the study when the server is busy. The other tools here assume the annotators are already on your team. See Running a Study on Prolific and MTurk, training phases and gold standards and attention checks.

Span agreement you can report. Pairwise F1, which Prodigy computes, compares two annotators at a time and does not correct for chance. Potato's α over per-character labels and its γ score where each span starts and stops as well as its label, correct for chance, and take every annotator at once. INCEpTION offers Krippendorff's unitizing α itself, as a matrix of annotator pairs.

Who wrote the answer. When a free-text answer was pasted in rather than typed, the annotator may not have written it. Potato's keystroke logging records how each answer was produced, and writing-process flags mark answers that arrived mostly by paste, with thresholds you can read and change.

Agreement on text labels

Two annotators who mark the same entity rarely agree on its exact boundaries, and a metric that compares labels token by token counts every boundary difference as a disagreement about the label. How a tool scores span agreement matters as much as whether it reports agreement at all.

• Potato reports token-level κ over BIO tags, span F1 at exact and partial match, Krippendorff's α over per-character labels, and a simplified γ from Mathet et al. (2015). The last two score where a span starts and stops as well as what it was called. See measuring agreement on spans.
• INCEpTION computes Cohen's κ, Fleiss' κ and Krippendorff's α, including the unitizing form, shown as a matrix of annotator pairs, and has a curation interface for settling disagreements.
• Prodigy's metric.iaa.doc recipe reports percent agreement, Krippendorff's α and Gwet's AC2 for document labels, and metric.iaa.span reports pairwise F1 for spans. Its review recipe handles adjudication.
• Label Studio keeps agreement for its paid plans. The metrics listed for its Enterprise edition are exact match, numeric difference, IoU and span overlap.
• doccano has no agreement metrics.

For the general case, see Inter-Annotator Agreement Explained.

The tools in more detail

INCEpTION

INCEpTION, from TU Darmstadt, succeeded WebAnno and is the closest comparison for linguistic work. It has relation layers, chain layers for coreference, document-level annotation, and concept features that link an annotation to an entry in a knowledge base such as Wikidata or DBpedia. On knowledge-base linking it is ahead of Potato. Its recommenders suggest annotations, with active learning built in. It is a Java application and takes more work to deploy than the other tools here. The system is described in Eckart de Castilho et al. (EMNLP 2024).

Prodigy

Prodigy, from Explosion, is built around annotation with a model in the loop and pairs with spaCy more closely than anything else here. It covers classification, spans, relations through its rel.manual recipe and coreference through coref.manual, and it reaches LLMs through spacy-llm. Tasks are defined in Python recipes. It is the one paid tool on this page.

Label Studio

Label Studio's text support is one part of a tool that also handles images, audio, video and time series. Its Relations tag links labelled regions. The Community Edition is free and open source. Agreement, ground-truth review and the quality dashboards are on the paid plans, starting with Starter at $99 a month.

doccano

doccano is the quickest of these to set up for text. Its project types are document classification, sequence labelling, sequence-to-sequence, intent detection and slot filling, and speech-to-text. Sequence-labelling projects can allow overlapping spans and relations. It has no coreference or event annotation and no agreement metrics.

brat

Many NLP corpora were built in brat, and its standoff format covers entities, n-ary events, binary relations, equivalence sets, attributes and normalizations. The last release is v1.3 from November 2012, and the last commit to master is from October 2021. Its standalone server imports Python's cgi module, which was removed in Python 3.13, so it will not start on a current Python without a shim. Reading a brat corpus is still worth doing. Starting a new project in brat is harder to justify.

Argilla

Argilla focuses on feedback and preference data for language models and integrates with Hugging Face Datasets. It supports text classification, token classification and evaluation of generated text. Its repository now says the original authors have moved on, and that the project will get bug fixes and patches but no new features.

Research systems

Several annotation systems have been published as ACL system demonstrations. Thresh configures fine-grained text-evaluation tasks in YAML, for summarization, simplification and machine translation. GATE Teamware 2 handles document classification with annotator training and quality screening.

Checked against each project's documentation, pricing page and repository in August and September 2026.

Moving a corpus between tools

Since version 2.9, Potato imports brat standoff, doccano JSONL, Prodigy db-out files and CoNLL, so a project started in any of them can continue in Potato. Existing labels arrive as pre-annotations. brat relations and events are listed in the import's warnings and are not imported. See Importing a Project.

Going the other way, Potato exports CoNLL-2003, CoNLL-U, Hugging Face datasets, JSONL, CSV and Parquet. See Exporting Annotations for ML.

An entity-and-relation task in Potato

annotation_schemes:
  - annotation_type: span
    name: entities
    description: "Highlight each person, organization and location."
    labels: [PERSON, ORGANIZATION, LOCATION]
  - annotation_type: span_link
    name: relations
    description: "Link each person to the organization they work for."
    span_schema: entities
    link_types:
      - name: WORKS_FOR
        directed: true
        allowed_source_labels: [PERSON]
        allowed_target_labels: [ORGANIZATION]

span_link draws typed links between spans from the entities scheme, and allowed_source_labels stops an annotator from linking the wrong kinds of entity. Relation and Event Extraction covers events and n-ary relations, and Coreference Resolution covers chains.

What Potato does not do for text

• Its knowledge-base linking is simpler than INCEpTION's.
• It does not update a spaCy model while annotators work, as Prodigy's recipes can.
• It has no spaCy exporter. Convert from CoNLL or JSONL.
• Importing from brat skips relations and events.

Frequently asked questions

Which text annotation tool should I use?

For a single NER or classification project, doccano is the least work to set up. For linguistic annotation that links entities to a knowledge base, INCEpTION is the most complete. For a study with several annotators per item, crowdworkers and an agreement figure to report, Potato includes all of that without a paid plan.

Which text annotation tool suits an academic annotation study?

If the study recruits annotators, screens and trains them, and has to report agreement, Potato covers each of those steps in its free version. INCEpTION is the better fit when the annotation itself is complex linguistic structure, such as linking entities to a knowledge base. Prodigy suits a small team that wants to train a spaCy model as it annotates.

Which text annotation tools work with Prolific or Amazon Mechanical Turk?

Potato has both built in. It reads the participant ID from the URL, shows a completion code or redirects the participant back when the task is done, and can manage a Prolific study through Prolific's API. INCEpTION has experimental invite links for guest annotators, which can be shared with crowdworkers, but documents no completion-code handling. Prodigy, Label Studio and doccano document no crowdsourcing integration.

Is there an open-source alternative to Prodigy?

Potato, INCEpTION and doccano are free and open source. Potato is configured in YAML where Prodigy uses Python recipes, and it imports Prodigy's db-out files, so annotations already collected in Prodigy carry over.

Which text annotation tools report inter-annotator agreement?

Potato and INCEpTION do in their free versions, and Prodigy's licence includes its metric.iaa recipes. Label Studio keeps agreement for its paid plans, and doccano has none. For span tasks, check whether the metric accounts for boundaries and for chance. Krippendorff's unitizing α and γ do both, and so does α over per-character labels, which is what Potato reports beside a simplified γ. Pairwise F1 does not correct for chance.

Is brat still maintained?

No. Its last release was v1.3 in November 2012, and its standalone server does not start on Python 3.13 without a shim. From version 2.9, Potato imports brat corpora, with existing entities carried over as pre-annotations.

Can I annotate coreference in a free tool?

Yes. INCEpTION has chain layers for coreference, and Potato has a coreference scheme that groups mentions into chains. doccano does not support coreference.

Further reading

• Annotation Tools Compared, for every data type on one page
• Audio Annotation Tools Compared
• Image Annotation Tools Compared
• Video Annotation Tools Compared
• AI Agent Evaluation Tools Compared
• World Model and Robot Episode Evaluation Tools Compared
• Running a Study on Prolific and MTurk
• Span Annotation
• Named Entity Recognition