Open-Source Annotation Tools Compared
An honest comparison of data annotation tools, Potato, Label Studio, Prodigy, Doccano, brat, INCEpTION, Argilla, CVAT, and Labelbox, and how to choose between them.
There is no single best annotation tool, the right choice depends on your modalities, your budget, whether you need agent/LLM evaluation, and how much setup you can tolerate. This guide compares the main open-source options fairly so you can match one to your project.
Which open-source annotation tools should I compare?
| Tool | License | Strengths | Best when |
|---|---|---|---|
| Potato | Free, open-source (research) | 50+ task types across text/image/audio/video, agent & LLM evaluation, zero-code YAML, built-in agreement metrics | Research, agent/LLM eval, fast setup without code |
| Label Studio | Open-source + paid tiers | Broad modality support, polished UI, large ecosystem | Teams wanting a commercial-backed platform |
| Prodigy | Paid (commercial) | Scriptable, active-learning-first, tight spaCy integration | spaCy users comfortable with a paid, code-driven tool |
| Doccano | Open-source | Simple, clean, easy to self-host | Straightforward text classification and NER |
| brat | Open-source | Mature rich text/relation annotation | Linguistic annotation of entities and relations |
| INCEpTION | Open-source | Rich linguistic annotation, knowledge-base linking | Deep linguistic projects that can invest in setup |
| Argilla | Open-source | LLM-data focus, Hugging Face integration | Feedback/RLHF data collection in the HF stack |
| CVAT | Open-source | Image/video computer-vision annotation | Bounding boxes, masks, and video CV labeling |
| Labelbox / Scale | Commercial (paid) | Managed platform, large workforce services | Enterprises buying tooling plus a labeling workforce |
(Details change over time, check each project for current licensing and features.)
One thing the license column hides: what a free tier actually includes. Label Studio's community edition ships no inter-annotator agreement metrics at all, and ground truth marking and the quality dashboards start at $99 per user per month. Prodigy has no free tier. CVAT lists its quality-control interface as paid. If you are choosing on quality control rather than on modality coverage, compare the free editions, not the marketing pages. We keep a capability matrix for eleven tools, checked against their own documentation.
How do I choose an annotation tool?
- What are you annotating? For text-only NER, Doccano or brat are simple. For mixed text/image/audio/video, Potato and Label Studio cover the range.
- Do you need agent or LLM evaluation? This is where Potato is unusual: it reads agent traces in many formats and has purpose-built tools for trajectory, process reward, web-agent, coding-agent, multi-agent team, and computer-use/multimodal evaluation. Most general tools don't.
- Budget. Potato, Label Studio (core), Doccano, brat, and Argilla are free and open-source; Prodigy and some Label Studio tiers are paid.
- Setup effort. Potato is configured with a YAML file and needs no code; Prodigy is code-first; the others sit in between.
- Ecosystem. Prodigy pairs with spaCy; Argilla with Hugging Face; Potato exports to many ML formats including CoNLL, spaCy, Hugging Face, and COCO/YOLO.
When is Potato the right annotation tool?
Potato came out of academic NLP (it was presented at EMNLP 2022 and HCOMP 2024) and is built for the full research workflow: many task types, quality control and agreement metrics in the box, crowdsourcing integrations, and, more recently, a deep set of AI-agent evaluation tools. If your work spans several modalities or includes evaluating LLMs and agents, it's worth a look.
If you mainly need a single text task with a hosted commercial product, or you live entirely inside spaCy or Hugging Face, one of the others may suit you better. Pick the tool that fits the work.
Frequently asked questions
Is Potato a free alternative to Label Studio?
Yes. Potato is free and open-source, and covers text, image, audio, video, and agent/LLM evaluation from a single YAML config with no code. Label Studio is broader in some integrations but pushes teams toward paid tiers; Potato stays free and self-hosted, which suits academic and reproducible-research work.
Is Potato a free, open-source alternative to Prodigy?
Yes. Prodigy is an excellent paid, code-first tool tied closely to spaCy; Potato is free, configured in YAML without code, and exports to spaCy, CoNLL, and Hugging Face formats. If you want active learning without a commercial license, Potato is the open-source option.
How does Potato compare to INCEpTION for linguistic annotation?
INCEpTION is powerful for deep linguistic annotation and knowledge-base linking but is heavier to deploy. Potato is simpler to stand up, a YAML file and one command, and is usually faster for span, relation, and classification tasks that don't need INCEpTION's full linguistic machinery.
Why use Potato instead of a commercial platform like Labelbox or Scale AI?
Labelbox and Scale sell a managed platform and a labeling workforce, which fits enterprises buying both. For research teams that bring their own annotators and need data to stay on their servers, Potato is the free, self-hosted alternative, with inter-annotator agreement metrics built in.
What is the best open-source tool for annotating AI agent trajectories?
This is where Potato is unusual among general annotation tools: it reads agent traces in 13 formats and has purpose-built displays for trajectory, step-level, web-agent, and coding-agent evaluation. It also annotates multi-agent teams on a clickable interaction graph and multimodal agents such as computer-use and voice agents. Most tools, open-source or commercial, don't annotate agent runs at all.
What can Potato evaluate that observability tools and labeling platforms can't?
Two categories of agent-annotation surface, neither of which appears as a configurable, self-hosted feature in the tools commonly compared against Potato (LangSmith, Langfuse, Labelbox, Scale AI, Label Studio, Argilla, Braintrust), checked against their docs as of June 2026:
- Multi-agent team structure. An annotator-editable interaction graph (mark the critical path, flag a bad handoff), cross-agent failure attribution as a responsible-agent / decisive-step / reason triple, handoff review as a first-class object, per-agent and per-team scorecards, a tool-contention timeline, and emergent-behavior tagging. The closest thing elsewhere is Langfuse's "Agent Graphs," which is a read-only debugging view rather than an annotation surface.
- Multimodal agents. Computer-use trajectories with a click-grounding marker, full-duplex voice timelines with barge-in scoring, and video temporal grounding with a live IoU against the model's predicted interval. Scale AI does GUI grounding and voice evaluation, but as managed dataset engagements, and its voice arena is turn-based rather than full-duplex.
The observability tools (LangSmith, Langfuse, Braintrust) attach span-level scores and comments, which is real per-step annotation but not these agent-structure surfaces. The labeling platforms (Labelbox, Scale) offer agent-evaluation data products, but as paid cloud or managed services, not a tool you self-host and configure in YAML. Capabilities move quickly, so this reflects a June 2026 snapshot; the full comparison post lists the versions checked.