Skip to content

Model-Assisted Labeling with a Model Trained on Your Labels

When pre-labels from a model trained on your own annotations save time, how pre-labels anchor annotators, and how a holdout group measures the effect.

Model-assisted labeling speeds up annotation by training a model on the labels your annotators have already given and showing the model's predictions on new items as pre-labels to confirm or correct. Because a pre-label also pulls the annotator toward the model's answer, a study that uses pre-labels should hold some annotators out and measure how far the answers move.

New in v2.14.0

A pre-label is one label a model proposed for an item before any person labeled the item. Model-assisted labeling, also called pre-labeling, is a human-in-the-loop workflow in which annotators accept, edit, or reject those pre-labels, and the accepted and corrected labels become the dataset. This guide covers the case where the model is trained on the project's own annotations and retrained as more arrive. The LLM and vision pre-annotation guide covers pre-labels from a prompted general-purpose model, and the active learning guide covers choosing which items to label next.

How a model trained on your labels differs from LLM pre-annotation

An LLM or a pretrained vision model can pre-label from the first item, guided by a prompt and a few examples. A model trained on your labels starts with nothing and needs a seed set of labeled items first. Potato's sklearn-text trainer refuses to fit with fewer than 10 items in the training split, and the Hugging Face text, token, and image classifiers refuse below 20. In exchange, the trained model learns the label boundaries your annotators actually draw on your items. Each run is scored on items held out from training, and the score is on the Training page before any annotator sees a pre-label. A small classifier also runs on your own machine at no cost per item.

LLM suggestions can cover the start of a study while the seed set grows, and the trained model can take over once the trained model's held-out score passes the LLM's agreement with the same labels. Active learning combines with both, since active learning chooses the order of items and pre-labeling changes the work on each item.

When pre-labels save time

Pre-labeling saves time when reading and confirming a label is faster than producing one. In a part-of-speech tagging experiment with two annotators, Fort and Sagot (2010) measured one annotator at about 7.5 minutes per 10 sentences without pre-annotation and about 2.5 minutes when correcting a tagger's output. A tagger trained on only 50 sentences reached 81.6% accuracy, and the time that tagger's pre-annotations saved did not differ significantly from the time saved by a tagger trained on 37,990 sentences at 96.4%. Only the tagger trained on 10 sentences made annotation significantly slower than the other taggers, and that tagger was 66.5% accurate. Lingren et al. (2014) found time savings of 13.85% to 21.5% per entity for clinical named entity annotation, with no significant change in agreement or annotator performance. Mikulová et al. (2022) found that pre-annotation from an accurate parser made dependency annotation faster and more consistent.

For a model trained on a project's own labels, the Fort and Sagot result suggests that a model trained on the first few dozen labeled items may already save time. A model still close to chance costs time, so the first pre-labels should wait for a held-out score well above chance.

When pre-labels hurt

Pre-labels change the answers as well as the speed. Anchoring is the pull of a suggested answer on a later judgment, and rubber-stamping is accepting suggestions unread, a form of automation bias. In the Fort and Sagot experiment, the more accurate annotator's accuracy fell from 98.1% to 95.8% with the 81.6% tagger's pre-annotations, and with the best tagger the same annotator made systematic errors that matched the tagger's own (adjective for past participle). In a pre-registered study of LLM suggestions on subjective tasks, Schroeder et al. (2025) found that annotators did not get faster and took the suggestions often enough to shift the label distribution. The LLM's measured accuracy also rose when the assisted labels served as the evaluation reference.

Retraining on accepted pre-labels adds a feedback loop. A pre-label the annotator saves unchanged becomes that annotator's answer, and the next training run learns from the answer like any other label. An error the annotators accept therefore returns as training data, and the next model can repeat the error on new items. Agreement measures cannot catch the loop, because annotators who accept the same pre-label agree by construction. How to tell review from rubber-stamping describes timing signals for annotators who accept without reading, and measuring how LLM suggestions anchor annotators describes blind items for annotators who read and are still pulled.

Design choices

Four settings decide whether pre-labels help a given study: the confidence floor, the items that get pre-labels, the share of annotators held out, and how often the model retrains.

Confidence threshold

Below a confidence floor, a prediction is hidden and the annotator labels the item from scratch. A higher floor shows fewer pre-labels and a larger share of correct ones. A floor means something only when the model's confidences are calibrated. A calibrated model's predictions at 0.8 confidence are right about 80% of the time on held-out items. Potato's sklearn-text calibrates the classifier's confidences by default once a run has at least 50 training items, with Platt scaling below 1,000 items and isotonic regression from there. Set the floor from the held-out items, at the confidence above which the model is right often enough that checking a pre-label is faster than labeling the item.

Which items get pre-labels

Potato never pre-labels a model's own training items, because a model is confident about the training data for reasons that say nothing about new data. Beyond that rule, keep pre-labels off the items used to measure inter-annotator agreement and off gold items. Agreement on items with a shared pre-label measures the pre-label, and a gold item answered with the pre-label on screen measures the model as much as the annotator. Among the remaining items, a review queue sorted least-confident first puts annotators on the model's likely mistakes. A queue that serves every item least-confident first fits a study where every item still needs a human label.

Holding annotators out to measure the effect

Anchoring is measured by comparing answers given with a pre-label to answers given without one, by the same annotators on the same kind of item. Potato's ai_support.holdout withholds every form of assistance from a random share of annotator-item pairs, and training pre-labels are among the forms withheld. Potato draws per pair rather than per item, and most items then get some assisted and some held-out annotators. A salted hash keeps each pair in the same group after a restart. For radio, multiselect, likert, and select schemes, the admin dashboard's AI influence tab compares the two groups on how often answers match the model, how often answers match gold, and agreement within each group. The tab reports the difference in model agreement with a 95% bootstrap interval once each group has 10 answers on items with a model answer. Set the fraction and the salt before annotation starts, since changing either mid-study reshuffles the groups. Measuring AI influence lists every assistance surface the holdout withholds.

How often to retrain

Retraining often keeps pre-labels current with the latest guideline decisions and corrections, and every retrain also replaces the pre-labels annotators see. Potato's auto_retrain starts a run once update_frequency new annotations have arrived (25 by default) and min_interval_s seconds have passed since the last run finished (600 by default). A larger update_frequency saves compute and keeps the pre-labels stable across longer stretches of work. Keep split_seed fixed for the study, because changing the seed moves held-out items into training and makes one run's held-out score incomparable with the next. Potato also puts every item a reviewer corrected into the training split of every later run. A correction then teaches the next model and is never used to score the model.

Model-assisted labeling in Potato

Potato's model training fits a model on a study's own annotations from the admin Training page, scores the model on held-out items, and writes the model's predictions back to items as pre-labels. Write-back shows the model's answer pre-selected on the annotation page. A pre-label becomes the annotator's answer only when the annotator edits the pre-label or moves on. Until then the item does not count as done. These blocks add model-assisted labeling with automatic retraining and a 20% holdout to a config with a radio scheme named sentiment:

yaml
model_training:
  enabled: true
  auto_retrain:
    enabled: true
    trainer: sklearn-text
    schemas: [sentiment]
    update_frequency: 50
    min_interval_s: 600
  writeback:
    schemas: [sentiment]
    include_unlabeled: true
    min_confidence: 0.8
 
ai_support:
  holdout:
    fraction: 0.2
    salt: reviews-2026

writeback.schemas names the schemes whose predictions annotators see. The list is empty by default, and annotators see no predictions until the list names a scheme. include_unlabeled: true also predicts on items nobody has labeled yet, and those items are where a pre-label saves time. Without the setting, a run predicts only on the run's held-out items. Predictions below min_confidence are stored and not shown. The holdout block works with ai_support.enabled off.

assignment_strategy: model_review serves pre-labeled items least-confident first for review, and assignment_strategy: llm_confidence serves every item in that order. A reviewer records an accept, correct, or reject verdict on each pre-label. A rejected pre-label comes off the item and is not shown again. DELETE /admin/training/api/runs/<run_id>/predictions withdraws every pre-label a run wrote. Model in the loop walks through write-back, the review queue, acquisition order, and auto_retrain step by step, and the upstream write-back reference lists every check a prediction passes before an annotator sees the prediction.

Further reading

References

  1. Fort, K., and Sagot, B. (2010). Influence of Pre-annotation on POS-tagged Corpus Development. Proceedings of the Fourth Linguistic Annotation Workshop, 56-63. https://aclanthology.org/W10-1807/
  2. Lingren, T., Deleger, L., Molnar, K., Zhai, H., Meinzen-Derr, J., Kaiser, M., Stoutenborough, L., Li, Q., and Solti, I. (2014). Evaluating the impact of pre-annotation on annotation speed and potential bias: natural language processing gold standard development for clinical named entity recognition in clinical trial announcements. Journal of the American Medical Informatics Association, 21(3), 406-413. https://doi.org/10.1136/amiajnl-2013-001837
  3. Mikulová, M., Straka, M., Štěpánek, J., Štěpánková, B., and Hajič, J. (2022). Quality and Efficiency of Manual Annotation: Pre-annotation Bias. Proceedings of the Thirteenth Language Resources and Evaluation Conference, 2909-2918. https://aclanthology.org/2022.lrec-1.312/
  4. Schroeder, H., Roy, D., and Kabbara, J. (2025). Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Tasks. Findings of the Association for Computational Linguistics: ACL 2025, 25771-25795. https://doi.org/10.18653/v1/2025.findings-acl.1323