# AI Auto-Labeling Compared: Label Studio, CVAT, Labelbox, Prodigy, Argilla and Potato

Source: https://www.potatoannotator.com/blog/ai-auto-labeling-tools-compared

**All six tools compared here can import or generate model labels and send them to people for review, so this comparison focuses on what each tool measures about the labels it keeps. In the documentation we read in October 2026, the tools report accuracy, precision, recall or agreement against ground truth or between annotators as point values, and none of them describes an interval or an error bound on machine-label quality.**

*Auto-labeling* here means any feature that puts a model's label on an item before or instead of a person's, whether the model is a detector, a classifier or an LLM prompted with the task. Every tool in this comparison has some version of it, and the tools differ on three questions worth asking before choosing one:

1. How machine labels get into a project, whether imported as predictions, generated by a connected model, or produced by an LLM prompt inside the tool.
2. How a person reviews them, and whether the person sees the model's answer while labeling.
3. What the tool reports about how often the machine labels are wrong, against what reference, and with what uncertainty.

Each section below is drawn from the tool's own documentation as of October 2026, linked where it is quoted. Where we did not find something in the documentation, we say so, since a feature can exist without being documented.

## Label Studio

[Label Studio](https://labelstud.io/guide/ml) connects models through an ML backend, an SDK that wraps your model as a web server, and imports predictions under a `predictions` key with a `model_version` and a `score`. Imported predictions [cannot be modified](https://labelstud.io/guide/predictions), and the score applies to the whole prediction. The Enterprise and Starter Cloud editions add Prompts, which generates predictions from LLM prompts and scores them, and the ML page describes an Enterprise feature for "LLM-assisted auto-labeling and model evaluation".

Review is an Enterprise workflow. The [quality guide](https://docs.humansignal.com/guide/quality) states that "the annotation review workflow is only available in Label Studio Enterprise Edition", with accept, reject and fix-and-accept actions and an option to order tasks by model confidence. Review sampling settings are applied as probabilistic rates, and uncertainty sampling sends the least confident predictions to people first.

For quality, Label Studio Enterprise [compares annotations and model predictions with ground truth](https://docs.humansignal.com/guide/ground_truths) "to calculate an accuracy score between 0 and 1", and its agreement matrix can include model versions. Prompts reports overall accuracy and F1 against ground truth, the number of tasks evaluated, and inference cost, each as a point score. Its [Prompts overview](https://docs.humansignal.com/guide/prompts_overview) adds that "the more data you have for the prompt evaluation, the more confident you can be with the results", without a stated interval.

## CVAT

[CVAT's automatic annotation](https://docs.cvat.ai/docs/annotation/auto-annotation/automatic-annotation/) pre-annotates data with a detector or re-identification model, with an optional confidence threshold. Which models are available depends on the edition. The [AI models table](https://docs.cvat.ai/docs/annotation/auto-annotation/ai-models/) lists Nuclio functions for the Community edition, and third-party models from Hugging Face and Roboflow and native AI-agent functions for CVAT Online and Enterprise.

For human review, CVAT has a review mode for reporting problems and a validation stage, and [consensus](https://docs.cvat.ai/docs/qa-analytics/consensus/) merges several annotators' work into one result with a 0-to-1 agreement score. CVAT's [ground-truth and honeypot quality control](https://docs.cvat.ai/docs/qa-analytics/auto-qa/), marked for CVAT Online and Enterprise, compares annotation jobs with a ground-truth job and reports accuracy, precision and recall. The documentation describes the method as statistical, estimating quality on the whole dataset from a small portion, and says "5-15% of the data is typically good enough". The quality report compares every job in the task with the ground-truth job, so it scores automatic annotation output only as part of what those jobs contain, and we did not find a report that separates machine annotations from human ones.

## Labelbox

[Labelbox](https://docs.labelbox.com/horizon/guides/import-prelabels) imports pre-labels, which it describes as "suggested annotations that are programmatically imported into a Labelbox project" for people to review. Its [Foundry](https://docs.labelbox.com/horizon/guides/foundry-model-assisted-labeling) feature generates labels with a model, with a confidence threshold "that tells the model how certain it needs to be before it suggests a label". Foundry is available on all plans except Educational subscriptions.

The guide to [using Foundry predictions](https://docs.labelbox.com/horizon/guides/use-foundry-predictions) suggests two ways to route them to people. One filters on confidence, such as sending everything below 0.75 to review. The other sends "a random sample of predictions for review", which "helps you spot systemic issues and calculate overall accuracy metrics". Labelbox's [model metrics](https://docs.labelbox.com/horizon/guides/model-metrics) compute precision, recall, F1, IoU and a confusion matrix from predictions and ground truth at thresholds from 0 to 1. Its [deprecations page](https://docs.labelbox.com/horizon/guides/deprecations) says the Model experiments feature is no longer available as of July 15, with no year given, while the metrics guide still sends custom metrics to the Experiment type, so check that model-run metrics are still available in your workspace.

## Prodigy

[Prodigy](https://prodi.gy/docs/recipes) is built around a model in the loop. Recipes such as `ner.correct` show a spaCy model's predictions for a person to correct, `ner.teach` and `textcat.teach` prefer the examples the model is least sure about, and `ner.model-annotate` records a model's labels as if an annotator had made them. LLM recipes built on `spacy-llm` produce labels for the same correction workflow, and the documentation warns that "the LLM annotations can be wrong. You still want to be in the loop to curate the annotations."

The `review` recipe resolves conflicts between annotators into one final annotation, and the `metric.iaa.*` recipes report percent agreement, Krippendorff's alpha and Gwet's AC2 between annotators. The documentation does not say whether to include a model's `model-annotate` labels in those agreement figures. For comparing prompts or models, `ab.llm.tournament` has people judge pairs of outputs and ranks the combinations with the Glicko rating system. [Prodigy's pricing page](https://prodi.gy/buy) lists a Personal and a Company license with the same installer, web application and plugins, and only the Company license adds SSO with OpenID Connect.

## Argilla

[Argilla](https://docs.argilla.io/latest/reference/argilla/records/suggestions/) attaches model output to records as suggestions, each with a score, the agent that produced it, and a type of `model` or `human`. In the annotation view, suggestions appear as pre-filled responses marked with a sparkle icon, with confidence scores shown when the dataset includes them. Argilla 1.x had a metrics module that compared responses with suggestions and reported metrics such as accuracy, F1 and Krippendorff's alpha. We did not find agreement or model-metric reporting in the Argilla 2.x documentation. Synthetic data and AI feedback, including LLM judging, live in the separate distilabel library.

## Potato

Potato connects LLMs and vision models through `ai_support`, and from version 2.9 it records what the model answered first (`pre_annotation_seeds`) beside what the annotator saved (`final_annotation`), so the acceptance rate comes from your own data. `icl_labeling` gives an LLM the annotations your annotators already agree on as examples, has it label the rest, and routes a sample of its labels back to people for blind verification, with accuracy tracked per scheme.

For a model or an LLM judge, Potato's [judge calibration](/docs/agent-evaluation/judge-calibration) keeps the model's labels where annotators cannot see them, draws a random or stratified sample for blind human labeling, and reports accuracy against the human gold label, Cohen's kappa for human-model, model-model and human-human pairs, Krippendorff's alpha, and calibration as expected calibration error and Brier score. Comparing the model's kappa with the kappa among the human annotators tells you whether the model does about as well as another annotator, as [LLM judge agreement](/docs/guides/llm-judge-human-agreement) explains. [Solo Mode](/docs/features/solo-mode) has one expert label alongside an LLM and hands the rest of the corpus to it once their agreement crosses a threshold. Like the other tools, these features report point estimates, and an interval on the error rate has to be computed outside the tool, as the last section describes.

## The six tools on the three questions

The table compares the six on the three questions, from the documentation each project published as of October 2026:

| Tool | How machine labels get in | Human review of machine labels | Quality reported for machine labels |
|---|---|---|---|
| Label Studio | ML backend, imported predictions, Prompts (Enterprise, Starter Cloud) | review workflow in Enterprise | accuracy against ground truth, agreement matrix with models (Enterprise) |
| CVAT | automatic annotation with detectors, third-party and agent models by edition | review mode, validation stage, consensus | ground-truth quality reports for annotation jobs (Online, Enterprise) |
| Labelbox | pre-label import, Foundry | confidence filters, random samples routed to review | precision, recall, F1 and IoU from model runs |
| Prodigy | spaCy and LLM recipes, `model-annotate` | correction recipes, `review` | agreement between annotators |
| Argilla | suggestions with scores | pre-filled responses | not found in 2.x docs |
| Potato | `ai_support`, `icl_labeling`, Solo Mode, judge calibration | verification sample, blind judge calibration sample | kappa, alpha, ECE and Brier score against a blind sample |

## Checking machine labels in any tool

Whichever tool you use, the number you can defend comes from the same three steps. Draw the items for human checking at random, not from the low-confidence end, because a sample of the hardest items says nothing about the error rate on the rest. Have people label that sample without seeing the model's answer, since a visible suggestion pulls the answer toward it. Then report the error rate on the sample with a confidence interval, using a [binomial proportion interval](https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval) such as Wilson's, so a reader can see whether 200 checked labels support the claim you are making.

Labelbox's random-sample routing, CVAT's randomly chosen ground-truth frames (Online and Enterprise) and Potato's judge calibration sample, which can also be stratified, already draw at random. In the other tools, export a random subset of item IDs, assign them as a separate task with predictions hidden, and compute the interval outside the tool.

## Further reading

- [Annotation tools compared](/docs/guides/annotation-tools-compared), for the broader feature comparison
- [LLM pre-annotation](/docs/guides/llm-pre-annotation), on automation bias when people review suggestions
- [LLM judge agreement](/docs/guides/llm-judge-human-agreement), on comparing a judge with people's agreement with each other