# Training a Model on Annotation Data

Source: https://www.potatoannotator.com/docs/guides/training-models-from-annotations

**To train a model on annotation data, resolve each item's labels to one target or to the annotators' vote split, assign items to train, validation, and test splits by a rule that stays fixed between rounds, and report the test score whenever the validation split chose the checkpoint. Potato's model training runs each step on a study's own annotations and records which items every run trained on.**

*New in v2.14.0*

Annotation data rarely arrives in the shape a trainer reads. A study with overlap has several labels per item, and the labels often disagree. A study still in progress gains labels every day, and every retraining round sees more items than the round before. A trainer needs one target per item and a fixed set of items to score on, and the decisions below produce both. Each section below names the Potato setting that makes the decision. For the Training page itself, see the [model training overview](/docs/model-training/overview) and the tutorial that [trains a first model](/docs/model-training/first-model). To train in your own code instead, [Exporting Annotations for Machine Learning](/docs/guides/exporting-annotations-for-ml) covers the export formats, and the same decisions apply to the exported file.

## Resolving several annotators to one label

Aggregation turns the labels several annotators gave one item into the single label a model trains on. The simplest method is majority vote. Majority vote counts every annotator's answer once and keeps the most common answer. [Dawid and Skene (1979)](https://doi.org/10.2307/2346806) estimate a confusion matrix for each annotator with expectation-maximization and weight each vote by the annotator's estimated reliability. A careless annotator then sways the result less than a careful annotator does. [Aggregating Crowd Labels](/docs/guides/aggregating-crowd-labels) compares the two methods and the models built on Dawid-Skene.

A training run in Potato resolves labels with `model_training.aggregation`. The default, `consensus`, runs Dawid-Skene whenever the project has more than one annotator and falls back to majority vote if the estimate fails. The other value, `majority`, runs majority vote alone. Both methods treat an annotator's whole answer on a scheme as one vote. Two `multiselect` answers therefore count as the same vote only when both answers tick the same set of labels.

The run's `consensus` setting and the [consensus export](/docs/features/consensus-aggregation) are separate features. The consensus export gives every annotator one vote, reports the count and fraction for every answer on every item, and marks each item `retired` or `needs_review`. Read the export before training when you want to see item by item where annotators split. [MACE](/docs/features/mace) estimates a competence score for each annotator from the same votes.

## Learning from disagreement

On subjective tasks such as offensiveness or emotion, annotators can disagree because each annotator reads the item differently, and a single voted label discards the split. [Uma et al. (2021)](https://doi.org/10.1613/jair.1.12752) survey methods that train on the disagreement instead of on one aggregated label. One family of these methods trains on soft labels. A soft label is a probability distribution over the labels, built from the share of annotators who chose each label.

`hf-text-classification` takes the parameter `soft_targets: true`. With the parameter set, an item that two or more annotators labeled trains toward the annotators' split by cross-entropy against the distribution. When two of three annotators said `positive`, the target is 0.67 `positive` and 0.33 `negative`. For a `multiselect` scheme, each label trains toward the share of annotators who ticked the label. An item with one annotator trains on that annotator's label as before. The model then stays unsure where people disagreed, and each prediction includes the model's `probabilities`. On held-out items with a vote split, the run reports three further metrics:

| Metric | What the metric measures |
|---|---|
| `soft_jsd` | Jensen-Shannon divergence between the model's distribution and the annotators' split, in bits from 0 to 1 |
| `soft_cross_entropy` | Cross-entropy of the annotators' split under the model's distribution |
| `ece` | Expected calibration error of the top label against the voted label, over 10 confidence bins |

In a [soft label scheme](/docs/annotation-types/soft-label), each annotator gives a distribution with sliders. `hf-text-classification` always trains a `soft_label` scheme toward the annotators' mean distribution, and the pre-label sets the sliders to the predicted distribution. Soft targets need two or more labels on an item. A study that plans to train on disagreement therefore collects overlapping labels from the start, and [How Many Annotators Do You Need?](/docs/guides/how-many-annotators) covers the overlap settings. [Disagreement Is Signal, Not Noise](/blog/disagreement-is-signal-not-noise) gives the case for keeping the split in the data.

## Train, validation, and test splits

The [training, validation, and test splits](https://en.wikipedia.org/wiki/Training,_validation,_and_test_data_sets) do three different jobs. The training split fits the model's weights. The validation split chooses among candidate models, for example the epoch at which to stop. The test split estimates performance on unseen data and plays no part in either choice. Choosing a checkpoint on validation is a form of fitting to validation. The checkpoint kept for the best validation score has a validation score biased upward, and only the test score comes from items the choice never consulted.

A Potato run splits the labeled items 70/15/15 by default (`splits`). Each item's split comes from a seeded hash of the item's ID (`split_seed`). An item held out in one run therefore stays held out in the next run, as long as `split_seed` and `splits` stay the same. Changing either setting can move held-out items into training, and every score after the change then partly measures memorization. Set both once, at the start of the study.

The fine-tuning trainers, such as `hf-text-classification`, keep the checkpoint from the epoch that scored best on validation. The headline score of such a run comes from test. `sklearn-text`, `setfit`, `rdkit-property`, the decision-model trainers, and the inference bridges choose nothing on validation, and a run of one of those trainers takes the headline score from validation whenever validation has items. The tooltip on the Runs table's **Held-out score** column names the split and the metric behind each number. An item with a label and no prediction counts as wrong. A trainer cannot raise the score by skipping hard items, and `<split>_coverage` reports the share of items that got a prediction.

### Corrected and pinned items

Two kinds of item go to the train split of every later run, whatever split the hash assigned. The first is an item a reviewer corrected in model review, and the second is an item someone pinned with **Train on these** beside a slice on the [Catalog](/docs/agent-evaluation/semantic-curation) page. Corrections come from the items the model was worst at, and in validation or test the corrections would score the model on a sample of the model's own mistakes. A run that moves held-out items into training logs a warning with the count, and the run's held-out score is then not directly comparable with an earlier run's score. A run that trains on a corrected item an earlier run held out also logs a warning, because a score on the earlier run's validation or test items, or on a dataset saved from those items, then includes items the new model trained on. [Corrections and pinned items always train](/docs/model-training/configuration#corrections-and-pinned-items-always-train) gives the rules.

## Recording what each model trained on

Comparing two runs requires knowing which items each run trained and scored on. Every Potato run writes a bundle under `<output_annotation_dir>/training/runs/<run_id>/bundle/` with `train.jsonl`, `val.jsonl`, `test.jsonl`, `labels.json`, and `manifest.json`. The manifest records the split seed, the aggregation method, the item count in each split, the count of items forced into training, and a SHA-256 digest of the split files. Nothing in a bundle refers back to Potato, and a script outside Potato can read the bundle as a self-contained copy of the resolved labels and splits.

The project database, `project.sqlite`, keeps a ledger with one row per run. Each row records the trainer, the parameters, the split sizes, the split seed, the held-out metrics, and the bundle digest. A second table records which split each item was in for each run. Write-back reads that table and never pre-labels an item that any run still in the ledger trained on (`retain_runs` keeps five finished runs by default). A model is confident on items the model trained on, for reasons that say nothing about new data.

To compare later models on exactly the same items, **Keep this split as a dataset** on the run page saves the run's validation, test, or train items as a version of a [dataset](/docs/agent-evaluation/datasets-and-experiments), with the agreed human labels. A saved version does not change afterwards. Each saved version contains one split and starts from scratch rather than adding to the previous version. Run 2's validation items saved on top of run 1's would otherwise include items run 2 trained on. An evaluation on a saved version cannot read the model's answer, because the saved inputs leave out the pre-label field. Saving needs `datasets.enabled: true`. When the model or the dataset leaves the project, [Documenting Datasets and Models](/docs/guides/documenting-datasets-and-models) covers datasheets and model cards.

## How much data a first model needs

Each Potato trainer refuses to fit with fewer than `min_instances` items in the train split. The defaults below are the counts at which a run starts:

| Trainer | `min_instances` default | Note |
|---|---|---|
| `setfit` | 8 | Built for the first days of a project, from about eight items per label |
| `sklearn-text` | 10 | Calibrates confidences only from 50 training items up |
| `hf-audio-tagger` | 10 | |
| `hf-text-classification`, `hf-token-classification`, `hf-image-classification` | 20 | |
| `llm-sft`, `llm-dpo` | 20 | |

The floor counts the train split only. At the default 70/15/15 split, a `sklearn-text` run starts once about 15 items have a label. The model training example in Potato's repository gives one measured point. With 120 of the example's 160 synthetic product reviews labeled once each, one `sklearn-text` run scored 0.84 macro-F1 on validation and published 80 pre-labels. The figure comes from one synthetic two-label task, and the held-out score of your own runs is the number to read for your task.

Start with `sklearn-text`. The trainer needs nothing beyond a plain install and no GPU. Retrain as labels arrive, and watch the held-out score from run to run with `split_seed` unchanged. Pre-labels anchor annotators. Leave `writeback.schemas` empty until the held-out score supports showing the model's answers. [Model-Assisted Labeling](/docs/guides/model-assisted-labeling) covers what changes once annotators see pre-labels, and [How to Tell Review from Rubber-Stamping](/docs/guides/detecting-rubber-stamped-prelabels) covers how to check that annotators still read the items.

## A config that sets each decision

The config below trains on a two-label sentiment scheme. Each key under `model_training` sets one of the decisions above:

```yaml
annotation_task_name: "Review sentiment"
task_dir: "."
data_files:
  - "data/reviews.json"
item_properties:
  id_key: "id"
  text_key: "text"
output_annotation_dir: "annotation_output"

annotation_schemes:
  - annotation_type: "radio"
    name: "sentiment"
    description: "Is this review positive or negative?"
    labels: ["positive", "negative"]

model_training:
  enabled: true
  aggregation: consensus
  split_seed: 0
  splits: {train: 0.7, val: 0.15, test: 0.15}
  trainer_params:
    hf-text-classification:
      soft_targets: true
```

`aggregation: consensus` resolves overlapping labels with Dawid-Skene. `split_seed` and `splits` are written out at the default values because neither setting should change for the life of the study. `soft_targets: true` makes every `hf-text-classification` run on the project train on vote splits. That trainer needs `pip install "potato-annotation[train-text]"`, and `sklearn-text` runs on the same config with no extra install. Without a `writeback` block, annotators see no pre-labels. Restart the server with the config, open `/admin/training`, pick a trainer and the `sentiment` scheme, and click **Train now**. The [model training configuration](/docs/model-training/configuration) reference lists every key, and the [model-in-the-loop tutorial](/docs/model-training/model-in-the-loop) adds write-back and automatic retraining.

## Further reading

- [Model training overview](/docs/model-training/overview)
- [Train your first model](/docs/model-training/first-model)
- [Model-assisted labeling](/docs/guides/model-assisted-labeling)
- [Exporting annotations for machine learning](/docs/guides/exporting-annotations-for-ml)
- [Aggregating crowd labels](/docs/guides/aggregating-crowd-labels)
- [Inter-annotator agreement](/docs/guides/inter-annotator-agreement)
- [Model training reference on GitHub](https://github.com/davidjurgens/potato/blob/master/docs/ai-intelligence/model_training.md#held-out-scores), with the held-out metrics for every scheme type

## References

- Dawid, A. P., and Skene, A. M. (1979). Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. *Applied Statistics*, 28(1), 20. [doi.org/10.2307/2346806](https://doi.org/10.2307/2346806)
- Uma, A. N., Fornaciari, T., Hovy, D., Paun, S., Plank, B., and Poesio, M. (2021). Learning from Disagreement: A Survey. *Journal of Artificial Intelligence Research*, 72, 1385–1470. [doi.org/10.1613/jair.1.12752](https://doi.org/10.1613/jair.1.12752)