AI Auto-Labeling Compared: Label Studio, CVAT, Labelbox, Prodigy, Argilla and Potato

All six tools compared here can import or generate model labels and send them to people for review, so this comparison focuses on what each tool measures about the labels it keeps. In the documentation we read in October 2026, the tools report accuracy, precision, recall or agreement against ground truth or between annotators as point values, and none of them describes an interval or an error bound on machine-label quality.

Auto-labeling here means any feature that puts a model's label on an item before or instead of a person's, whether the model is a detector, a classifier or an LLM prompted with the task. Every tool in this comparison has some version of it, and the tools differ on three questions worth asking before choosing one:

1. How machine labels get into a project, whether imported as predictions, generated by a connected model, or produced by an LLM prompt inside the tool.
2. How a person reviews them, and whether the person sees the model's answer while labeling.
3. What the tool reports about how often the machine labels are wrong, against what reference, and with what uncertainty.

Each section below is drawn from the tool's own documentation as of October 2026, linked where it is quoted. Where we did not find something in the documentation, we say so, since a feature can exist without being documented.

Label Studio

Label Studio connects models through an ML backend, an SDK that wraps your model as a web server, and imports predictions under a predictions key with a model_version and a score. Imported predictions cannot be modified, and the score applies to the whole prediction. The Enterprise and Starter Cloud editions add Prompts, which generates predictions from LLM prompts and scores them, and the ML page describes an Enterprise feature for "LLM-assisted auto-labeling and model evaluation".

Review is an Enterprise workflow. The quality guide states that "the annotation review workflow is only available in Label Studio Enterprise Edition", with accept, reject and fix-and-accept actions and an option to order tasks by model confidence. Review sampling settings are applied as probabilistic rates, and uncertainty sampling sends the least confident predictions to people first.

For quality, Label Studio Enterprise compares annotations and model predictions with ground truth "to calculate an accuracy score between 0 and 1", and its agreement matrix can include model versions. Prompts reports overall accuracy and F1 against ground truth, the number of tasks evaluated, and inference cost, each as a point score. Its Prompts overview adds that "the more data you have for the prompt evaluation, the more confident you can be with the results", without a stated interval.

CVAT

CVAT's automatic annotation pre-annotates data with a detector or re-identification model, with an optional confidence threshold. Which models are available depends on the edition. The AI models table lists Nuclio functions for the Community edition, and third-party models from Hugging Face and Roboflow and native AI-agent functions for CVAT Online and Enterprise.

For human review, CVAT has a review mode for reporting problems and a validation stage, and consensus merges several annotators' work into one result with a 0-to-1 agreement score. CVAT's ground-truth and honeypot quality control, marked for CVAT Online and Enterprise, compares annotation jobs with a ground-truth job and reports accuracy, precision and recall. The documentation describes the method as statistical, estimating quality on the whole dataset from a small portion, and says "5-15% of the data is typically good enough". The quality report compares every job in the task with the ground-truth job, so it scores automatic annotation output only as part of what those jobs contain, and we did not find a report that separates machine annotations from human ones.

Labelbox

Labelbox imports pre-labels, which it describes as "suggested annotations that are programmatically imported into a Labelbox project" for people to review. Its Foundry feature generates labels with a model, with a confidence threshold "that tells the model how certain it needs to be before it suggests a label". Foundry is available on all plans except Educational subscriptions.

The guide to using Foundry predictions suggests two ways to route them to people. One filters on confidence, such as sending everything below 0.75 to review. The other sends "a random sample of predictions for review", which "helps you spot systemic issues and calculate overall accuracy metrics". Labelbox's model metrics compute precision, recall, F1, IoU and a confusion matrix from predictions and ground truth at thresholds from 0 to 1. Its deprecations page says the Model experiments feature is no longer available as of July 15, with no year given, while the metrics guide still sends custom metrics to the Experiment type, so check that model-run metrics are still available in your workspace.

Prodigy

Prodigy is built around a model in the loop. Recipes such as ner.correct show a spaCy model's predictions for a person to correct, ner.teach and textcat.teach prefer the examples the model is least sure about, and ner.model-annotate records a model's labels as if an annotator had made them. LLM recipes built on spacy-llm produce labels for the same correction workflow, and the documentation warns that "the LLM annotations can be wrong. You still want to be in the loop to curate the annotations."

The review recipe resolves conflicts between annotators into one final annotation, and the metric.iaa.* recipes report percent agreement, Krippendorff's alpha and Gwet's AC2 between annotators. The documentation does not say whether to include a model's model-annotate labels in those agreement figures. For comparing prompts or models, ab.llm.tournament has people judge pairs of outputs and ranks the combinations with the Glicko rating system. Prodigy's pricing page lists a Personal and a Company license with the same installer, web application and plugins, and only the Company license adds SSO with OpenID Connect.

Argilla

Argilla attaches model output to records as suggestions, each with a score, the agent that produced it, and a type of model or human. In the annotation view, suggestions appear as pre-filled responses marked with a sparkle icon, with confidence scores shown when the dataset includes them. Argilla 1.x had a metrics module that compared responses with suggestions and reported metrics such as accuracy, F1 and Krippendorff's alpha. We did not find agreement or model-metric reporting in the Argilla 2.x documentation. Synthetic data and AI feedback, including LLM judging, live in the separate distilabel library.

Potato

Potato connects LLMs and vision models through ai_support, and from version 2.9 it records what the model answered first (pre_annotation_seeds) beside what the annotator saved (final_annotation), so the acceptance rate comes from your own data. icl_labeling gives an LLM the annotations your annotators already agree on as examples, has it label the rest, and routes a sample of its labels back to people for blind verification, with accuracy tracked per scheme.

For a model or an LLM judge, Potato's judge calibration keeps the model's labels where annotators cannot see them, draws a random or stratified sample for blind human labeling, and reports accuracy against the human gold label, Cohen's kappa for human-model, model-model and human-human pairs, Krippendorff's alpha, and calibration as expected calibration error and Brier score. Comparing the model's kappa with the kappa among the human annotators tells you whether the model does about as well as another annotator, as LLM judge agreement explains. Solo Mode has one expert label alongside an LLM and hands the rest of the corpus to it once their agreement crosses a threshold. Like the other tools, these features report point estimates, and an interval on the error rate has to be computed outside the tool, as the last section describes.

The six tools on the three questions

The table compares the six on the three questions, from the documentation each project published as of October 2026:

Checking machine labels in any tool

Whichever tool you use, the number you can defend comes from the same three steps. Draw the items for human checking at random, not from the low-confidence end, because a sample of the hardest items says nothing about the error rate on the rest. Have people label that sample without seeing the model's answer, since a visible suggestion pulls the answer toward it. Then report the error rate on the sample with a confidence interval, using a binomial proportion interval such as Wilson's, so a reader can see whether 200 checked labels support the claim you are making.

Labelbox's random-sample routing, CVAT's randomly chosen ground-truth frames (Online and Enterprise) and Potato's judge calibration sample, which can also be stratified, already draw at random. In the other tools, export a random subset of item IDs, assign them as a separate task with predictions hidden, and compute the interval outside the tool.

Further reading

• Annotation tools compared, for the broader feature comparison
• LLM pre-annotation, on automation bias when people review suggestions
• LLM judge agreement, on comparing a judge with people's agreement with each other