# Model Training Configuration and Routes

Source: https://www.potatoannotator.com/docs/model-training/configuration

*New in v2.14.0*

This page is the reference for the `model_training` block of a Potato config and for what happens to a run after you start the run. The page is written for the person who runs a study and wants to limit runs, send runs to another machine, show a model's predictions to annotators, or start training from a script. Every key, default, route, and message below was checked against Potato's code. For a first run, start with [Train your first model](/docs/model-training/first-model). The trainers and their parameters are listed in [Trainers](/docs/model-training/trainers).

## A complete config

The config below trains a sentiment classifier on product reviews. `auto_retrain` refits `sklearn-text` after every 25 new annotations. With the default TF-IDF features, `sklearn-text` trains on the CPU and needs nothing beyond a plain install of Potato. Frequent automatic runs therefore cost little. Write-back shows annotators every prediction with a confidence of at least 0.8. With `include_unlabeled`, a run also predicts on the reviews nobody has labeled yet. The `hf-text-classification` defaults and the `gpu-server` backend serve runs started by hand from the Training page:

```yaml
annotation_task_name: "Review sentiment with model training"
task_dir: "."
data_files:
  - "data/reviews.json"
item_properties:
  id_key: "id"
  text_key: "text"
output_annotation_dir: "annotation_output"
user_config:
  allow_all_users: true

annotation_schemes:
  - annotation_type: "radio"
    name: "sentiment"
    description: "Is this review positive or negative?"
    labels: ["positive", "negative"]

model_training:
  enabled: true
  split_seed: 0
  splits: {train: 0.7, val: 0.15, test: 0.15}
  aggregation: consensus
  retain_runs: 5
  queue_max: 10
  max_wall_seconds: 3600
  max_ram_mb: 8192
  num_threads: 4
  trainer_params:
    sklearn-text:
      classifier: sklearn.linear_model.LogisticRegression
      classifier_kwargs: {max_iter: 1000}
    hf-text-classification:
      model_name: answerdotai/ModernBERT-base
      epochs: 4
      lora: true
  auto_retrain:
    enabled: true
    trainer: sklearn-text
    schemas: [sentiment]
    min_interval_s: 600
    update_frequency: 25
  writeback:
    schemas: [sentiment]
    min_confidence: 0.8
    include_unlabeled: true
  backends:
    gpu-server:
      type: ssh
      host: gpu-server
      python: /opt/conda/bin/python3
      gpus: auto
      min_free_gpu_mb: 4000
```

Keep `split_seed` fixed for the life of the study. A new seed reassigns every item, so items that were held out end up in training and later scores stop being comparable with earlier scores. The `min_confidence` floor of 0.8 is a starting point, and the right floor depends on the first held-out score. Turn on `include_unlabeled` once that score supports showing pre-labels, because pre-labels anchor annotators to the model's answer.

## Keys in `model_training`

Every key sits under `model_training`:

| Key | Default | Meaning |
|---|---|---|
| `enabled` | `false` | Registers the Training page, the routes under `/admin/training`, and the agent routes under `/api/training`. Without `enabled: true` the routes do not exist. |
| `retain_runs` | `5` | Finished runs kept in the run ledger. An older run leaves the ledger unless the run's predictions are still stored. Run directories stay on disk. |
| `grace_seconds` | `20` | Seconds a cancelled run gets to exit before Potato kills the process. 0 or more, and sent with the job to backends of type `ssh`, `slurm`, and `agent`. |
| `queue_max` | `10` | Runs that may wait behind the run in progress. |
| `split_seed` | `0` | Seed for assigning items to the train, val, and test splits. |
| `splits` | `{train: 0.7, val: 0.15, test: 0.15}` | Split fractions, normalized to sum to 1. The names are `train`, `val`, and `test`, and `train` is required. |
| `aggregation` | `consensus` | How several annotators' labels on an item become one label. `consensus` fits the [Dawid-Skene](https://doi.org/10.2307/2346806) model when the study has more than one annotator and falls back to majority vote if the fit fails. `majority` takes the majority vote. |
| `max_wall_seconds` | unset | Time limit for one run. Potato checks a local run each time the run reports an event, and an ssh or agent run every second. |
| `max_ram_mb` | unset | Address-space limit for the training process, set with `setrlimit`. On a platform without the limit, the run logs a warning and continues. |
| `device` | `auto` | `auto`, `cpu`, `cuda`, `cuda:N`, or `mps`. `auto` takes CUDA, then Apple's Metal, then the CPU. |
| `num_threads` | unset | Sets `OMP_NUM_THREADS`, `MKL_NUM_THREADS`, and `OPENBLAS_NUM_THREADS` for the run, unless the environment already sets those variables. |
| `auto_retrain` | off | Retraining as annotations arrive. See [`auto_retrain`](#auto_retrain). |
| `writeback` | none | Publishing predictions as pre-labels. See [`writeback`](#writeback). |
| `backend` | `local` | The backend for runs that name no other. |
| `backends` | none | Named `ssh`, `slurm`, and `agent` backends. See [Backends](#backends). |
| `max_predictions_mb` | `512` | Largest `predictions.jsonl` a remote agent may upload. |
| `trainer_params` | none | Parameter defaults, keyed by trainer name. |
| `tiling` | `enabled: auto` | Cutting large images into tiles for the detection and segmentation trainers. See [`tiling`](#tiling). |

A run's own `params` override `trainer_params`, which override the trainer's defaults. A parameter name the trainer does not take is refused at server start when the name sits under `trainer_params`, and when the run starts otherwise. [Trainers](/docs/model-training/trainers#parameters) gives the order with an example.

### `auto_retrain`

Automatic retraining checks for a new run after every annotation save:

| Key | Default | Meaning |
|---|---|---|
| `enabled` | `false` | Turns the check on. |
| `trainer` | none | Trainer name. Required. |
| `schemas` | none | Schemes to train. Required. |
| `min_interval_s` | `600` | Seconds since the last run finished before another run starts. |
| `update_frequency` | `25` | New annotations since the last automatic run before another run starts. |
| `params` | `{}` | Trainer parameters, as in the `params` of an API run. |
| `backend` | `model_training.backend` | Backend for automatic runs. |
| `accept_licence` | `false` | Confirms the license of a trainer that asks for confirmation (`ultralytics-detect`, `ultralytics-segment`, `layoutlmv3`). Without the confirmation, automatic runs of those trainers are skipped. |

When a check decides against a run, the reason appears in the `skip_reason` field of `GET /admin/training/api/status`, e.g., "waiting for 3 more annotation(s)" or "a run is already in progress". Automatic runs never join the run queue. The next annotation saved after the queue empties checks again, with more annotations to train on. Potato keeps the annotation counts in memory, so after a restart a project that has trained before counts new annotations from the restart.

### `writeback`

Write-back decides which predictions annotators see:

| Key | Default | Meaning |
|---|---|---|
| `schemas` | `[]` | Schemes whose predictions annotators see. An empty list shows none. |
| `min_confidence` | `0.0` | Predictions below the floor are stored and not shown. A prediction with no confidence passes. |
| `field` | `predictions` | Item field the pre-labels are written to. |
| `include_unlabeled` | `false` | Also predicts on items nobody has labeled, and pre-labels those items. Without `include_unlabeled`, a run predicts only on the run's own `val` and `test` items. |

### `tiling`

The detection and segmentation trainers resize each image to a few hundred pixels. At that size the objects on a whole-slide image or a satellite scene are lost. For an `image_annotation`, `microscopy`, or `geo_annotation` scheme, Potato therefore cuts each large image into tiles when the run's training data is built. An image's split is decided before the image is cut, so tiles of one slide never land in two splits. Predictions are mapped back onto the whole image, and held-out scores are computed on whole images. The [vision tutorial](/docs/model-training/vision-models) covers slides and maps in practice.

| Key | Default | Meaning |
|---|---|---|
| `enabled` | `auto` | `auto` tiles slide formats (`.svs`, `.ndpi`, `.mrxs`, `.scn`, and others) and images whose longer side is at least `auto_min_side`. `true` tiles every image, and `false` none. |
| `auto_min_side` | `4096` | Longer-side length, in pixels, at which `auto` starts tiling an image. |
| `tile_size` | `1024` | Side of a tile, in the pixels the trainer sees. |
| `overlap` | `128` | Pixels neighboring tiles share, at most half of `tile_size`. Keep the overlap at least as large as the objects, or a large object is found in pieces. |
| `downsample` | `1` | Full-resolution pixels per tile pixel. |
| `mpp` | none | Microns per tile pixel. Overrides `downsample` for a slide that records the slide's resolution. |
| `background` | `0.25` | Empty tiles sampled per tile with an object. Set `background: 0` for slides annotated only in part. |
| `max_predict_tiles` | `2000` | Most tiles predicted per image. The tiles with the most tissue are kept. |
| `min_visible` | `0.4` | Share of an object that must lie inside a tile for the object to train there. |
| `tissue_threshold` | `0.05` | Share of a slide tile that must be tissue for the model to predict on the tile. |

Slides are read with openslide, installed by `pip install "potato-annotation[wsi]"`, on the machine that builds the training data. A remote backend downloads the tiles and never the slide.

## The run queue

One run executes at a time. A run started from the Training page or the API while another run is in progress waits rather than being refused. The start request returns `202` with `queued: true` and the run's `position`, and the Runs table lists the run as `waiting #N` with a **Cancel run** button. When the run in progress ends, the oldest waiting run starts.

Waiting runs are records in the run ledger, so the queue survives a restart, and the first waiting run starts when the server comes back. A run the previous server was in the middle of is marked `interrupted` at the same time. A waiting run's training data is built when the run starts, so the run trains on the annotations present at that moment. A start that would be refused, such as a start naming an unknown scheme, is refused at once rather than when the run's turn comes. A run whose scheme or trainer left the config while the run waited fails with `invalid_when_dequeued`, and the next run starts.

Send `"queue": false` in the request body to get `409` instead of a place in the queue. With `queue_max` runs already waiting, a new start gets `429`. `GET /admin/training/api/status` lists the waiting runs in order under `queue`.

## Checks before a run

`potato validate` reports every problem in the `model_training` block at once, and the server refuses to start on the same problems:

```bash
potato validate config.yaml --strict
```

The checks cover these cases:

- `splits` with a name other than `train`, `val`, and `test`, a fraction below 0 or not a number, fractions that sum to 0, or no `train`.
- A key that `writeback` or `auto_retrain` does not have, a `writeback.schemas` entry that is not a scheme, or a `min_confidence` outside 0 to 1. `accept_license` is reported as a misspelling of `accept_licence`.
- An enabled `pre_annotation` block whose `field` differs from `writeback.field`. No training pre-label would show under such a block.
- With `auto_retrain.enabled`, a trainer or scheme that does not exist, a scheme the trainer cannot learn, parameters the trainer does not take or of the wrong type, a `backend` missing from `backends`, or `decision-calibrate` on a backend other than `local`.
- A parameter under `trainer_params` that the trainer does not take, or of the wrong type. Inside `dp`, the check also covers each key and the key's range.
- A `grace_seconds` below 0 or not a number, or a `retain_runs` or `split_seed` below 0 or not a whole number.
- Under `backends`, a missing or unknown `type`, a key the type does not have, an ssh backend with no `host`, a `gpus` value other than `auto`, `none`, or GPU indices, a remote backend named `local`, or a `model_training.backend` that is not defined.

`potato training check` tests one backend before a run uses the backend. For `local`, the command prints the device a run would use, e.g., `local: this machine, Apple Metal (mps)` on a Mac with Apple silicon. For an ssh backend, the command reports the Python version, every GPU with the GPU's free memory, the GPU a run would get, and the training packages the remote interpreter lacks. The **Check** button on the Training page runs the same test, and `--json` prints the report as JSON:

```bash
potato training check local config.yaml
potato training check gpu-server config.yaml
```

A run started from the Training page or the API is checked again when the run starts. The status codes are listed under [Routes](#routes).

## Backends

A run executes on a backend, and the Training page's **Runs on** menu lists the configured backends. `model_training.backend` sets the default, and an API run or `auto_retrain` can name a different backend with `backend`. Potato has four backend types:

| Type | Where the trainer runs | Who starts the trainer |
|---|---|---|
| `local` | This machine, in a subprocess. Always available, and the default. | Potato |
| `ssh` | Another machine Potato can reach over ssh, such as a GPU server. | Potato, over ssh |
| `slurm` | A cluster's compute node, as a batch job. | Potato, by ssh to the login node and then `sbatch` |
| `agent` | A machine Potato cannot reach, such as a workstation behind NAT. | A person, with `potato training agent` |

An ssh backend uses key-based authentication with `BatchMode` on, so a password prompt fails instead of hanging. A slurm backend uses only `sbatch`, `squeue`, and `scancel`, plus `scontrol` on clusters that provide `scontrol`, and needs a `remote_dir` on storage the compute nodes share. An agent backend needs no inbound port, because the agent opens every connection. The name `local` is reserved for this machine, and `decision-calibrate` runs only on `local`. [Train on a GPU server, SLURM cluster, or machine behind NAT](/docs/model-training/remote-training) walks through all three remote types with their keys.

## Held-out scores

Every run scores the run's own predictions on the `val` and `test` items against the resolved human labels. Metric names carry the split, as in `val_macro_f1` or `test_f1`. An item with a label and no prediction counts as wrong, so leaving out hard items cannot raise a trainer's score. `<split>_coverage` reports the share of items that got a prediction.

The headline score appears in the Runs table's **Held-out score** column, with the split and metric in the tooltip. The headline is the first scheme's headline metric on the first of `val` and `test` that has items. When a trainer picked the best checkpoint by `val` score, Potato tries `test` first, because the trainer has already seen the `val` score. The fine-tuning trainers that keep the best epoch, e.g., `hf-text-classification`, `torchvision-detect`, and `llm-sft`, take the headline from `test`. `sklearn-text`, `setfit`, `rdkit-property`, the decision-model trainers, and the inference bridges choose nothing on `val`, and a run of one of those trainers takes the headline from `val` whenever `val` has items. A run with no `val` or `test` items is shown as "not scored".

The headline metric depends on the scheme's annotation type, as the table below shows for common types:

| Annotation type | Headline metric |
|---|---|
| `radio`, `select`, `likert` | macro-F1 (also accuracy, and mean absolute error for numeric `likert` labels) |
| `multiselect` | micro-F1 (also macro-F1 and exact match) |
| `span` | F1 over spans that match on start, end, and label |
| `text` | token F1 (also exact match) |
| `pairwise` in `binary` mode | accuracy over items not answered as a tie |
| `ranking` | Kendall's tau |
| `image_annotation`, `microscopy`, `geo_annotation` | COCO mAP at IoU 0.5 (`map_50`), with `map_50_95` also recorded |
| `audio_annotation` | F1 over segments with the same label at IoU 0.5 or more |
| `speech_transcript` | `word_accuracy`, 1 minus the word error rate |
| `volume_annotation` | `mean_dice` |

Token F1 credits an answer that uses the right words in the wrong order, so read token F1 together with exact match. The detection headline uses IoU 0.5 rather than the 0.5 to 0.95 average, because two annotators' boxes for one object rarely overlap at IoU 0.9. The [upstream held-out score table](https://github.com/davidjurgens/potato/blob/master/docs/ai-intelligence/model_training.md#held-out-scores) lists the metrics for every annotation type.

## Write-back

After a successful run, Potato stores the run's predictions and writes the predictions that pass four checks into each item's `predictions` field, or the field `writeback.field` names. The checks run in this order, and the run's write-back report counts each skip:

1. **Train-split items are never pre-labeled.** The check covers the train split of every run still in the ledger, and a pre-label an earlier run left on an item is taken down when a later run trains on the item.
2. **Only schemes in `writeback.schemas` are published.** The list is empty by default, so turning training on changes nothing annotators see.
3. **Rejected pairs are skipped.** An item and scheme pair with a `reject` verdict in [model review](https://github.com/davidjurgens/potato/blob/master/docs/ai-intelligence/model_review.md) is not shown again.
4. **The confidence floor applies.** A prediction below `min_confidence` is stored and not published.

Write-back never changes a human label, because predictions go into item data and annotations live in each annotator's user state. A pre-label shows as a pre-filled answer the annotator can edit, e.g., a choice as the selected option and a free-text answer in the scheme's text box, and no `pre_annotation` block is needed. The pre-label becomes the annotator's answer only when the annotator edits the pre-label or moves on. A pre-label nobody has edited or moved past therefore never becomes a training label. An annotator-item pair that `ai_support.holdout` places in the study's no-assistance comparison group shows no pre-label, and the item data sent for that pair leaves out both the `writeback.field` and the `pre_annotation.field`. The write-back report is the `writeback` field of `GET /admin/training/api/status`, and the report's `summary` says why a run published nothing.

Potato keeps one prediction per item and scheme, and a newer run's prediction replaces an older run's prediction. When the newer prediction is stored but not shown, because the prediction is under `min_confidence`, empty, or rejected, the older run's pre-label comes off the item at once, and the write-back report counts the item in `n_superseded_cleared`. Pre-labels supplied with the data are left alone. At startup the stored predictions for the schemes in `writeback.schemas` are written back into item data under the same four checks, so pre-labels survive a restart. To withdraw a run's pre-labels, send `DELETE /admin/training/api/runs/<run_id>/predictions`. The request removes the stored predictions and takes the pre-labels off the items.

### Corrections and pinned items always train

Two kinds of item go to the `train` split of every later run, whatever the item's place in the stable split:

- **Items a reviewer corrected.** A `correct` verdict in model review means a person fixed the model's answer. Corrections come from the items the model was worst at, so a corrected item in `val` or `test` would score the model on a sample of the model's own mistakes. `accept` and `reject` verdicts leave an item where the item was.
- **Pinned items.** **Train on these** beside a slice on the [Catalog](/docs/agent-evaluation/semantic-curation) page pins the items the slice matches, e.g., the built-in `model-found-nothing` slice. A pinned item trains once someone has labeled the item.

A run that moved held-out items into `train` logs a warning with the count. The run's held-out score is then computed without those items and is not directly comparable with an earlier run's score. The counts are under `stats.forced_train` in the training data's `manifest.json`.

### Saving a split as a dataset version

**Keep this split as a dataset** on a run's page saves the run's `val`, `test`, or `train` items with their agreed human labels as a new version of a [dataset](/docs/agent-evaluation/datasets-and-experiments). A dataset version never changes afterwards, so later models and LLM prompts can be compared on exactly those items. The feature needs `datasets.enabled: true`. Each saved version holds one split and starts from scratch rather than adding to the dataset's last version. Run 2's `val` items added to run 1's would otherwise include items run 2 trained on. Each example's `inputs` leave out the pre-label field, so an evaluation cannot read the model's answer.

### Where a pre-label came from

`GET /admin/model-review` includes `provenance` for each item in the review queue whose pre-label a training run wrote. For each scheme, the entry gives the run id, the trainer, and the run's held-out score. Pre-labels supplied with the data have no entry.

## Where runs are stored

Each run has a directory at `<task_dir>/<output_annotation_dir>/training/runs/<run_id>/`:

| Path | Contents |
|---|---|
| `bundle/` | The training data: `train.jsonl`, `val.jsonl`, `test.jsonl`, `unlabeled.jsonl` with `include_unlabeled`, `labels.json`, `manifest.json`, and `media/`. |
| `artifacts/` | `model_card.json`, `predictions.jsonl`, and the model, e.g., `model.pkl` for `sklearn-text` or a `model/` directory for the transformer trainers. |
| `spec.json` | What the run was asked to do. |
| `events.jsonl`, `stdout.log`, `stderr.log` | The run's progress events and output. |
| `agent.log` | The remote agent's own output, on ssh backends. |
| `bundle.tar.gz` | The training data as the agent downloaded the data, media included, on remote backends. |

The run ledger lives in the project database, `<task_dir>/project.sqlite`, and records the runs, each item's split per run, the items pinned to train, the stored predictions, agent tokens, and pushes to the Hugging Face Hub. `retain_runs` prunes the ledger and never prunes a run whose predictions are still stored, so every pre-label can be traced to the run that wrote the pre-label. Pruning leaves run directories on disk.

A LoRA run saves only the adapter, a few to a few hundred megabytes, and loads the base model by `model_name` when the run predicts. A full fine-tune saves the whole model. Set the `POTATO_MODEL_CACHE` environment variable to a directory, and the training process uses that directory for `HF_HOME`, `TRANSFORMERS_CACHE`, and `SENTENCE_TRANSFORMERS_HOME` unless those variables are already set. On an [air-gapped machine](/docs/guides/air-gapped-annotation), put the models the trainers name in that directory in advance.

## Routes

The Training page and the page's JSON routes sit under `/admin/training` and need admin access. Admin access means an admin session or the `X-API-Key` header. POST and DELETE requests from a browser must be same-origin. Potato's [OpenAPI spec](https://github.com/davidjurgens/potato/blob/master/docs/api-reference/openapi.json) marks all 26 training operations with `x-potato-requires-config: model_training.enabled: true`:

| Route | Purpose |
|---|---|
| `GET /admin/training` | The Training page, with the run form, the run list, and the trainers. |
| `GET /admin/training/runs/<run_id>` | One run's page, with status, metrics, events, and each item's split. |
| `GET /admin/training/api/trainers` | Every trainer, with the trainer's kinds, modalities, license, and install status. |
| `GET /admin/training/api/schemas` | The study's schemes, each scheme's kind, and the trainers that fit each scheme. |
| `GET /admin/training/api/status` | The current or last run, with state, progress, metrics, warnings, `skip_reason`, `writeback`, and `queue`. |
| `GET /admin/training/api/runs` | Run history. `?limit=` defaults to 50, with a maximum of 500. |
| `POST /admin/training/api/runs` | Starts a run. `200` when the run starts, and `202` with `queued: true` and `position` when the run waits. |
| `GET /admin/training/api/runs/<run_id>` | One run's record, plus `review`, the precision of the run's predictions under review verdicts. |
| `GET /admin/training/api/runs/<run_id>/events` | The run's event log. `?limit=` defaults to 200. |
| `POST /admin/training/api/runs/<run_id>/cancel` | Cancels the run in progress or a waiting run. `400` for any other run id. |
| `DELETE /admin/training/api/runs/<run_id>/predictions` | Withdraws the run's stored predictions and pre-labels. |
| `POST /admin/training/api/runs/<run_id>/push-to-hub` | Publishes the run's model to the Hugging Face Hub. See [Publishing to the Hugging Face Hub](#publishing-to-the-hugging-face-hub). |
| `POST /admin/training/api/runs/<run_id>/pin-split` | Saves a split as a dataset version, with the body `{"dataset": name, "split": "val"}`. |
| `GET /admin/training/api/backends` | The configured backends. |
| `POST /admin/training/api/backends/<name>/check` | Tests a backend, as `potato training check` does. |
| `GET /admin/training/api/pinned` | Pinned items, grouped by reason. |
| `POST /admin/training/api/pinned` | Pins items to `train`, with `{"slice": name}` or `{"instance_ids": [...]}`. |
| `DELETE /admin/training/api/pinned` | Unpins the listed `instance_ids`, or every pinned item when the body lists none. |

A run request takes `trainer`, `schemas` (a list or one name), `params`, `backend`, `accept_licence`, and `queue`. Every trainer fits only the first scheme named. The request below starts an `sklearn-text` run on the example's `sentiment` scheme:

```bash
curl -X POST -H "X-API-Key: $KEY" -H "Content-Type: application/json" \
  http://localhost:8000/admin/training/api/runs \
  -d '{"trainer": "sklearn-text", "schemas": ["sentiment"]}'
```

A refused start gives the reason in `error`:

| Status | When |
|---|---|
| `400` | No trainer or no scheme in the body, an unknown trainer or scheme, a scheme the trainer cannot learn, a trainer the project cannot run, or a trainer whose license needs confirming, with `needs_licence_ack: true`, until the request sets `accept_licence: true` |
| `409` | A parameter the trainer does not take or of the wrong type or range, an unknown backend, `decision-calibrate` on a backend other than `local`, or a run already in progress with `"queue": false` |
| `429` | `queue_max` runs already waiting |

### Agent protocol

`model_training.enabled` also registers the routes a training agent calls, under `/api/training`. An agent sends the agent's token as `Authorization: Bearer <token>`, and a missing or invalid token gets `401`. Registering a backend takes the admin API key instead. `potato/training/protocol.py` describes the protocol for anyone writing a new agent, and `potato/training/agent.py` is the reference implementation:

| Route | Purpose |
|---|---|
| `POST /api/training/backends/register` | Issues an agent token for a backend name, with `X-API-Key`. `potato training register` calls this route. |
| `POST /api/training/backends/<name>/heartbeat` | An idle agent reports that the agent is alive. `403` for a token issued to another backend. |
| `GET /api/training/jobs/next?wait=30` | Long poll. `200` with a job, or `204` when no job arrives within `wait` seconds. |
| `GET /api/training/jobs/<run_id>` | The job, the job's spec, and whether the job was cancelled. |
| `GET /api/training/jobs/<run_id>/bundle` | The training data as `tar.gz`, media included. The route answers `Range` requests, so an agent resumes a dropped download, up to five attempts. `410` once the bundle is gone. |
| `POST /api/training/jobs/<run_id>/progress` | Progress events. A cancel comes back in the response. |
| `POST /api/training/jobs/<run_id>/predictions` | `predictions.jsonl`. `413` above `max_predictions_mb`. |
| `POST /api/training/jobs/<run_id>/complete` | Exit code and final events. |

## QLoRA

`llm-sft`, `llm-dpo`, `hf-text-classification`, `vlm-sft`, and `reward-model` take `lora: {quantize: 4bit}`, the setup of [QLoRA](https://proceedings.neurips.cc/paper_files/paper/2023/hash/1feb87871436031bdc0f2beaa62a049b-Abstract-Conference.html) (Dettmers et al., 2023). The base model loads with 4-bit NF4 weights and double quantization, and a LoRA adapter trains on top in bf16 where the GPU supports bf16. `quantize: 8bit` is the halfway option, and predictions load the base model the same way.

```yaml
model_training:
  trainer_params:
    llm-sft:
      model_name: Qwen/Qwen2.5-7B-Instruct
      lora: {quantize: 4bit}
```

In 4 bits the base weights take a quarter of the memory the same weights take in bf16, and activations, the adapter, and the optimizer state come on top. Potato has not yet measured a QLoRA run's memory on a GPU, so plan from the run's own "Peak GPU memory" line. Every transformer run logs that line at the end. QLoRA needs a CUDA GPU and a `bitsandbytes` build for that machine's CUDA. The run checks both before loading the model by quantizing a few numbers on the GPU, and when the two versions do not match, the error names both versions. The [LLM fine-tuning tutorial](/docs/model-training/fine-tuning-llms) puts QLoRA to work on SFT and DPO runs.

## Publishing to the Hugging Face Hub

A successful run whose model is on the Potato server has **Publish to the Hugging Face Hub** on the run's page. The push uploads the model and a `README.md` model card written from the run, naming the trainer, the base model and adapter, the schemes and labels, the split sizes, the held-out score and the split the score was measured on, and the run and bundle ids. The card of a [`dp` run](/docs/model-training/differential-privacy) also has a "Differential privacy" row with the epsilon spent against the target, the delta, and the train split the guarantee covers. The row leaves out the target or the delta when the run did not record the value.

The upload is a fixed list of files rather than the whole `artifacts/` directory. The list contains the weight files, e.g., `.safetensors`, `.bin`, `.pt`, and `.pkl`, the config and tokenizer files in the saved `model/` directory, `model_card.json` with local paths replaced by file names, and `calibration.json`. The model's answers on the project's items (`predictions.jsonl`, and `raw_answers.jsonl` for the decision models) stay on the server. Checkpoints, optimizer state, the copies of the training data some trainers make (`yolo_data/`, `wev_data/`, `laya_data/`), and any symlink that points outside `artifacts/` are left out as well.

The form guards against an accidental or public push in several ways:

- You type the repository name twice, and the push goes ahead only when the two names match.
- **Private** is ticked by default. While **Private** is ticked, Potato does not push to an existing public repository, and the API answers `409`.
- The form names an environment variable on the server, `HF_TOKEN` by default, that holds a Hugging Face write token. The token itself is never typed into the page. The variable's name must contain `TOKEN` or start with `HF_`, and the value must start with `hf_`, so no other variable's value can reach the Hub.
- When the Hub cannot say whether the repository exists, nothing is pushed, and a repository the push created is deleted again if the upload then fails.
- The model card's license is the Hub license id you give, such as `apache-2.0`. With none, the card says the base model's license applies. A trainer whose license needs confirming, such as `ultralytics-detect` (AGPL-3.0), needs the same confirmation to push.
- A model left on a remote host with `fetch_artifacts: false`, or trained on an agent backend, cannot be pushed from the server, and the error names where the model is.

Each push is one commit. The run page lists every push with the repository, visibility, and commit, and the ledger records who made the push. Through the API, the body of `POST /admin/training/api/runs/<run_id>/push-to-hub` takes `repo_id`, `confirm_repo_id`, `private` (default `true`), `token_env` (default `HF_TOKEN`), `licence`, and `accept_licence`. The route returns `201` with the commit, `400` for a mismatch, a missing token variable, no model files, or an unconfirmed license, and `502` when the Hub refuses.

## Troubleshooting

The entries below cover the errors most likely on a first setup. The [upstream troubleshooting list](https://github.com/davidjurgens/potato/blob/master/docs/ai-intelligence/model_training.md#troubleshooting) covers GPU drivers, CUDA builds, and the trainer-specific messages.

- **`/admin/training` returns 404.** `model_training.enabled` is not `true`, and Potato registers the routes only when the key is `true`.
- **"Only N items in the train split".** Annotate more items, or lower `min_instances` in the run's `params`. The train split holds 70% of the labeled items by default.
- **"Schema 'x' has 1 distinct label(s) in the data".** A classifier needs at least two labels in use.
- **"No resolved annotations to train on".** The scheme named in the run differs from the scheme being annotated, or nobody has annotated anything yet.
- **The run published no pre-labels.** Read the `writeback` report in `GET /admin/training/api/status`. "No predictions were published: write-back is not enabled for any schema" means `writeback.schemas` is empty, and the skip counts show how many predictions each check removed.
- **`<trainer> does not take <name>`.** A parameter in the run or under `trainer_params` is misspelled or belongs to another trainer. The message lists the names the trainer takes.
- **The run was killed after the machine ran out of memory.** The process exited on SIGKILL, usually from the out-of-memory killer. Use a smaller model or fewer items, or set `max_ram_mb` so the run fails inside the run's own limit.
- **"Stopped: the run took longer than model_training.max_wall_seconds".** The run reached the time limit. Raise `max_wall_seconds`, or train on fewer items.
- **"The GPU ran out of memory".** Lower `batch_size` and raise `gradient_accumulation` to keep the effective batch, set `lora: true` or `lora: {quantize: 4bit}`, or for `llm-sft` set `gradient_checkpointing: true` or choose a smaller model.
- **"ssh to host failed: Permission denied".** Potato runs `ssh` with `BatchMode=yes`, so `ssh <host> true` must succeed from the server's account without a prompt. A key with a passphrase works when `ssh-agent` holds the key.
- **A run on an agent backend stays queued.** No agent with that backend name is polling. Start an agent with `potato training agent --backend <name>`, and check the token with **Check** on the Training page.

## Further reading

- [Model training on your own annotations](/docs/model-training/overview): the training loop, the Training page, and a quick start.
- [Trainers](/docs/model-training/trainers): every trainer, the schemes each trainer learns, and the install commands.
- [Train on a GPU server, SLURM cluster, or machine behind NAT](/docs/model-training/remote-training): ssh, SLURM, and polling-agent backends step by step.
- [Model in the loop](/docs/model-training/model-in-the-loop): write-back, review, `auto_retrain`, and dataset versions in one study.
- [Active learning](/docs/features/active-learning): ordering items by model uncertainty.
- [Detecting rubber-stamped pre-labels](/docs/guides/detecting-rubber-stamped-prelabels): checking whether annotators accept pre-labels without reading the pre-labels.
- [Upstream reference](https://github.com/davidjurgens/potato/blob/master/docs/ai-intelligence/model_training.md): the full parameter tables for every trainer.

## References

- Dawid, A. P., and Skene, A. M. (1979). Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. *Applied Statistics*, 28(1). https://doi.org/10.2307/2346806
- Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. *Advances in Neural Information Processing Systems 36 (NeurIPS 2023)*. https://proceedings.neurips.cc/paper_files/paper/2023/hash/1feb87871436031bdc0f2beaa62a049b-Abstract-Conference.html