Training a Model on Annotation Data

To train a model on annotation data, resolve each item's labels to one target or to the annotators' vote split, assign items to train, validation, and test splits by a rule that stays fixed between rounds, and report the test score whenever the validation split chose the checkpoint. Potato's model training runs each step on a study's own annotations and records which items every run trained on.

New in v2.14.0

Annotation data rarely arrives in the shape a trainer reads. A study with overlap has several labels per item, and the labels often disagree. A study still in progress gains labels every day, and every retraining round sees more items than the round before. A trainer needs one target per item and a fixed set of items to score on, and the decisions below produce both. Each section below names the Potato setting that makes the decision. For the Training page itself, see the model training overview and the tutorial that trains a first model. To train in your own code instead, Exporting Annotations for Machine Learning covers the export formats, and the same decisions apply to the exported file.

Resolving several annotators to one label

Aggregation turns the labels several annotators gave one item into the single label a model trains on. The simplest method is majority vote. Majority vote counts every annotator's answer once and keeps the most common answer. Dawid and Skene (1979) estimate a confusion matrix for each annotator with expectation-maximization and weight each vote by the annotator's estimated reliability. A careless annotator then sways the result less than a careful annotator does. Aggregating Crowd Labels compares the two methods and the models built on Dawid-Skene.

A training run in Potato resolves labels with model_training.aggregation. The default, consensus, runs Dawid-Skene whenever the project has more than one annotator and falls back to majority vote if the estimate fails. The other value, majority, runs majority vote alone. Both methods treat an annotator's whole answer on a scheme as one vote. Two multiselect answers therefore count as the same vote only when both answers tick the same set of labels.

The run's consensus setting and the consensus export are separate features. The consensus export gives every annotator one vote, reports the count and fraction for every answer on every item, and marks each item retired or needs_review. Read the export before training when you want to see item by item where annotators split. MACE estimates a competence score for each annotator from the same votes.

Learning from disagreement

On subjective tasks such as offensiveness or emotion, annotators can disagree because each annotator reads the item differently, and a single voted label discards the split. Uma et al. (2021) survey methods that train on the disagreement instead of on one aggregated label. One family of these methods trains on soft labels. A soft label is a probability distribution over the labels, built from the share of annotators who chose each label.

hf-text-classification takes the parameter soft_targets: true. With the parameter set, an item that two or more annotators labeled trains toward the annotators' split by cross-entropy against the distribution. When two of three annotators said positive, the target is 0.67 positive and 0.33 negative. For a multiselect scheme, each label trains toward the share of annotators who ticked the label. An item with one annotator trains on that annotator's label as before. The model then stays unsure where people disagreed, and each prediction includes the model's probabilities. On held-out items with a vote split, the run reports three further metrics:

In a soft label scheme, each annotator gives a distribution with sliders. hf-text-classification always trains a soft_label scheme toward the annotators' mean distribution, and the pre-label sets the sliders to the predicted distribution. Soft targets need two or more labels on an item. A study that plans to train on disagreement therefore collects overlapping labels from the start, and How Many Annotators Do You Need? covers the overlap settings. Disagreement Is Signal, Not Noise gives the case for keeping the split in the data.

Train, validation, and test splits

The training, validation, and test splits do three different jobs. The training split fits the model's weights. The validation split chooses among candidate models, for example the epoch at which to stop. The test split estimates performance on unseen data and plays no part in either choice. Choosing a checkpoint on validation is a form of fitting to validation. The checkpoint kept for the best validation score has a validation score biased upward, and only the test score comes from items the choice never consulted.

A Potato run splits the labeled items 70/15/15 by default (splits). Each item's split comes from a seeded hash of the item's ID (split_seed). An item held out in one run therefore stays held out in the next run, as long as split_seed and splits stay the same. Changing either setting can move held-out items into training, and every score after the change then partly measures memorization. Set both once, at the start of the study.

The fine-tuning trainers, such as hf-text-classification, keep the checkpoint from the epoch that scored best on validation. The headline score of such a run comes from test. sklearn-text, setfit, rdkit-property, the decision-model trainers, and the inference bridges choose nothing on validation, and a run of one of those trainers takes the headline score from validation whenever validation has items. The tooltip on the Runs table's Held-out score column names the split and the metric behind each number. An item with a label and no prediction counts as wrong. A trainer cannot raise the score by skipping hard items, and <split>_coverage reports the share of items that got a prediction.

Corrected and pinned items

Two kinds of item go to the train split of every later run, whatever split the hash assigned. The first is an item a reviewer corrected in model review, and the second is an item someone pinned with Train on these beside a slice on the Catalog page. Corrections come from the items the model was worst at, and in validation or test the corrections would score the model on a sample of the model's own mistakes. A run that moves held-out items into training logs a warning with the count, and the run's held-out score is then not directly comparable with an earlier run's score. A run that trains on a corrected item an earlier run held out also logs a warning, because a score on the earlier run's validation or test items, or on a dataset saved from those items, then includes items the new model trained on. Corrections and pinned items always train gives the rules.

Recording what each model trained on

Comparing two runs requires knowing which items each run trained and scored on. Every Potato run writes a bundle under <output_annotation_dir>/training/runs/<run_id>/bundle/ with train.jsonl, val.jsonl, test.jsonl, labels.json, and manifest.json. The manifest records the split seed, the aggregation method, the item count in each split, the count of items forced into training, and a SHA-256 digest of the split files. Nothing in a bundle refers back to Potato, and a script outside Potato can read the bundle as a self-contained copy of the resolved labels and splits.

The project database, project.sqlite, keeps a ledger with one row per run. Each row records the trainer, the parameters, the split sizes, the split seed, the held-out metrics, and the bundle digest. A second table records which split each item was in for each run. Write-back reads that table and never pre-labels an item that any run still in the ledger trained on (retain_runs keeps five finished runs by default). A model is confident on items the model trained on, for reasons that say nothing about new data.

To compare later models on exactly the same items, Keep this split as a dataset on the run page saves the run's validation, test, or train items as a version of a dataset, with the agreed human labels. A saved version does not change afterwards. Each saved version contains one split and starts from scratch rather than adding to the previous version. Run 2's validation items saved on top of run 1's would otherwise include items run 2 trained on. An evaluation on a saved version cannot read the model's answer, because the saved inputs leave out the pre-label field. Saving needs datasets.enabled: true. When the model or the dataset leaves the project, Documenting Datasets and Models covers datasheets and model cards.

How much data a first model needs

Each Potato trainer refuses to fit with fewer than min_instances items in the train split. The defaults below are the counts at which a run starts:

The floor counts the train split only. At the default 70/15/15 split, a sklearn-text run starts once about 15 items have a label. The model training example in Potato's repository gives one measured point. With 120 of the example's 160 synthetic product reviews labeled once each, one sklearn-text run scored 0.84 macro-F1 on validation and published 80 pre-labels. The figure comes from one synthetic two-label task, and the held-out score of your own runs is the number to read for your task.

Start with sklearn-text. The trainer needs nothing beyond a plain install and no GPU. Retrain as labels arrive, and watch the held-out score from run to run with split_seed unchanged. Pre-labels anchor annotators. Leave writeback.schemas empty until the held-out score supports showing the model's answers. Model-Assisted Labeling covers what changes once annotators see pre-labels, and How to Tell Review from Rubber-Stamping covers how to check that annotators still read the items.

A config that sets each decision

The config below trains on a two-label sentiment scheme. Each key under model_training sets one of the decisions above:

annotation_task_name: "Review sentiment"
task_dir: "."
data_files:
  - "data/reviews.json"
item_properties:
  id_key: "id"
  text_key: "text"
output_annotation_dir: "annotation_output"
 
annotation_schemes:
  - annotation_type: "radio"
    name: "sentiment"
    description: "Is this review positive or negative?"
    labels: ["positive", "negative"]
 
model_training:
  enabled: true
  aggregation: consensus
  split_seed: 0
  splits: {train: 0.7, val: 0.15, test: 0.15}
  trainer_params:
    hf-text-classification:
      soft_targets: true

aggregation: consensus resolves overlapping labels with Dawid-Skene. split_seed and splits are written out at the default values because neither setting should change for the life of the study. soft_targets: true makes every hf-text-classification run on the project train on vote splits. That trainer needs pip install "potato-annotation[train-text]", and sklearn-text runs on the same config with no extra install. Without a writeback block, annotators see no pre-labels. Restart the server with the config, open /admin/training, pick a trainer and the sentiment scheme, and click Train now. The model training configuration reference lists every key, and the model-in-the-loop tutorial adds write-back and automatic retraining.

Further reading

• Model training overview
• Train your first model
• Model-assisted labeling
• Exporting annotations for machine learning
• Aggregating crowd labels
• Inter-annotator agreement
• Model training reference on GitHub, with the held-out metrics for every scheme type

References

• Dawid, A. P., and Skene, A. M. (1979). Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. Applied Statistics, 28(1), 20. doi.org/10.2307/2346806
• Uma, A. N., Fornaciari, T., Hovy, D., Paun, S., Plank, B., and Poesio, M. (2021). Learning from Disagreement: A Survey. Journal of Artificial Intelligence Research, 72, 1385–1470. doi.org/10.1613/jair.1.12752