Model Training Configuration and Routes

New in v2.14.0

This page is the reference for the model_training block of a Potato config and for what happens to a run after you start the run. The page is written for the person who runs a study and wants to limit runs, send runs to another machine, show a model's predictions to annotators, or start training from a script. Every key, default, route, and message below was checked against Potato's code. For a first run, start with Train your first model. The trainers and their parameters are listed in Trainers.

A complete config

The config below trains a sentiment classifier on product reviews. auto_retrain refits sklearn-text after every 25 new annotations. With the default TF-IDF features, sklearn-text trains on the CPU and needs nothing beyond a plain install of Potato. Frequent automatic runs therefore cost little. Write-back shows annotators every prediction with a confidence of at least 0.8. With include_unlabeled, a run also predicts on the reviews nobody has labeled yet. The hf-text-classification defaults and the gpu-server backend serve runs started by hand from the Training page:

annotation_task_name: "Review sentiment with model training"
task_dir: "."
data_files:
  - "data/reviews.json"
item_properties:
  id_key: "id"
  text_key: "text"
output_annotation_dir: "annotation_output"
user_config:
  allow_all_users: true
 
annotation_schemes:
  - annotation_type: "radio"
    name: "sentiment"
    description: "Is this review positive or negative?"
    labels: ["positive", "negative"]
 
model_training:
  enabled: true
  split_seed: 0
  splits: {train: 0.7, val: 0.15, test: 0.15}
  aggregation: consensus
  retain_runs: 5
  queue_max: 10
  max_wall_seconds: 3600
  max_ram_mb: 8192
  num_threads: 4
  trainer_params:
    sklearn-text:
      classifier: sklearn.linear_model.LogisticRegression
      classifier_kwargs: {max_iter: 1000}
    hf-text-classification:
      model_name: answerdotai/ModernBERT-base
      epochs: 4
      lora: true
  auto_retrain:
    enabled: true
    trainer: sklearn-text
    schemas: [sentiment]
    min_interval_s: 600
    update_frequency: 25
  writeback:
    schemas: [sentiment]
    min_confidence: 0.8
    include_unlabeled: true
  backends:
    gpu-server:
      type: ssh
      host: gpu-server
      python: /opt/conda/bin/python3
      gpus: auto
      min_free_gpu_mb: 4000

Keep split_seed fixed for the life of the study. A new seed reassigns every item, so items that were held out end up in training and later scores stop being comparable with earlier scores. The min_confidence floor of 0.8 is a starting point, and the right floor depends on the first held-out score. Turn on include_unlabeled once that score supports showing pre-labels, because pre-labels anchor annotators to the model's answer.

Keys in model_training

Every key sits under model_training:

A run's own params override trainer_params, which override the trainer's defaults. A parameter name the trainer does not take is refused at server start when the name sits under trainer_params, and when the run starts otherwise. Trainers gives the order with an example.

auto_retrain

Automatic retraining checks for a new run after every annotation save:

When a check decides against a run, the reason appears in the skip_reason field of GET /admin/training/api/status, e.g., "waiting for 3 more annotation(s)" or "a run is already in progress". Automatic runs never join the run queue. The next annotation saved after the queue empties checks again, with more annotations to train on. Potato keeps the annotation counts in memory, so after a restart a project that has trained before counts new annotations from the restart.

writeback

Write-back decides which predictions annotators see:

tiling

The detection and segmentation trainers resize each image to a few hundred pixels. At that size the objects on a whole-slide image or a satellite scene are lost. For an image_annotation, microscopy, or geo_annotation scheme, Potato therefore cuts each large image into tiles when the run's training data is built. An image's split is decided before the image is cut, so tiles of one slide never land in two splits. Predictions are mapped back onto the whole image, and held-out scores are computed on whole images. The vision tutorial covers slides and maps in practice.

Slides are read with openslide, installed by pip install "potato-annotation[wsi]", on the machine that builds the training data. A remote backend downloads the tiles and never the slide.

The run queue

One run executes at a time. A run started from the Training page or the API while another run is in progress waits rather than being refused. The start request returns 202 with queued: true and the run's position, and the Runs table lists the run as waiting #N with a Cancel run button. When the run in progress ends, the oldest waiting run starts.

Waiting runs are records in the run ledger, so the queue survives a restart, and the first waiting run starts when the server comes back. A run the previous server was in the middle of is marked interrupted at the same time. A waiting run's training data is built when the run starts, so the run trains on the annotations present at that moment. A start that would be refused, such as a start naming an unknown scheme, is refused at once rather than when the run's turn comes. A run whose scheme or trainer left the config while the run waited fails with invalid_when_dequeued, and the next run starts.

Send "queue": false in the request body to get 409 instead of a place in the queue. With queue_max runs already waiting, a new start gets 429. GET /admin/training/api/status lists the waiting runs in order under queue.

Checks before a run

potato validate reports every problem in the model_training block at once, and the server refuses to start on the same problems:

potato validate config.yaml --strict

The checks cover these cases:

• splits with a name other than train, val, and test, a fraction below 0 or not a number, fractions that sum to 0, or no train.
• A key that writeback or auto_retrain does not have, a writeback.schemas entry that is not a scheme, or a min_confidence outside 0 to 1. accept_license is reported as a misspelling of accept_licence.
• An enabled pre_annotation block whose field differs from writeback.field. No training pre-label would show under such a block.
• With auto_retrain.enabled, a trainer or scheme that does not exist, a scheme the trainer cannot learn, parameters the trainer does not take or of the wrong type, a backend missing from backends, or decision-calibrate on a backend other than local.
• A parameter under trainer_params that the trainer does not take, or of the wrong type. Inside dp, the check also covers each key and the key's range.
• A grace_seconds below 0 or not a number, or a retain_runs or split_seed below 0 or not a whole number.
• Under backends, a missing or unknown type, a key the type does not have, an ssh backend with no host, a gpus value other than auto, none, or GPU indices, a remote backend named local, or a model_training.backend that is not defined.

potato training check tests one backend before a run uses the backend. For local, the command prints the device a run would use, e.g., local: this machine, Apple Metal (mps) on a Mac with Apple silicon. For an ssh backend, the command reports the Python version, every GPU with the GPU's free memory, the GPU a run would get, and the training packages the remote interpreter lacks. The Check button on the Training page runs the same test, and --json prints the report as JSON:

potato training check local config.yaml
potato training check gpu-server config.yaml

A run started from the Training page or the API is checked again when the run starts. The status codes are listed under Routes.

Backends

A run executes on a backend, and the Training page's Runs on menu lists the configured backends. model_training.backend sets the default, and an API run or auto_retrain can name a different backend with backend. Potato has four backend types:

An ssh backend uses key-based authentication with BatchMode on, so a password prompt fails instead of hanging. A slurm backend uses only sbatch, squeue, and scancel, plus scontrol on clusters that provide scontrol, and needs a remote_dir on storage the compute nodes share. An agent backend needs no inbound port, because the agent opens every connection. The name local is reserved for this machine, and decision-calibrate runs only on local. Train on a GPU server, SLURM cluster, or machine behind NAT walks through all three remote types with their keys.

Held-out scores

Every run scores the run's own predictions on the val and test items against the resolved human labels. Metric names carry the split, as in val_macro_f1 or test_f1. An item with a label and no prediction counts as wrong, so leaving out hard items cannot raise a trainer's score. <split>_coverage reports the share of items that got a prediction.

The headline score appears in the Runs table's Held-out score column, with the split and metric in the tooltip. The headline is the first scheme's headline metric on the first of val and test that has items. When a trainer picked the best checkpoint by val score, Potato tries test first, because the trainer has already seen the val score. The fine-tuning trainers that keep the best epoch, e.g., hf-text-classification, torchvision-detect, and llm-sft, take the headline from test. sklearn-text, setfit, rdkit-property, the decision-model trainers, and the inference bridges choose nothing on val, and a run of one of those trainers takes the headline from val whenever val has items. A run with no val or test items is shown as "not scored".

The headline metric depends on the scheme's annotation type, as the table below shows for common types:

Token F1 credits an answer that uses the right words in the wrong order, so read token F1 together with exact match. The detection headline uses IoU 0.5 rather than the 0.5 to 0.95 average, because two annotators' boxes for one object rarely overlap at IoU 0.9. The upstream held-out score table lists the metrics for every annotation type.

Write-back

After a successful run, Potato stores the run's predictions and writes the predictions that pass four checks into each item's predictions field, or the field writeback.field names. The checks run in this order, and the run's write-back report counts each skip:

1. Train-split items are never pre-labeled. The check covers the train split of every run still in the ledger, and a pre-label an earlier run left on an item is taken down when a later run trains on the item.
2. Only schemes in writeback.schemas are published. The list is empty by default, so turning training on changes nothing annotators see.
3. Rejected pairs are skipped. An item and scheme pair with a reject verdict in model review is not shown again.
4. The confidence floor applies. A prediction below min_confidence is stored and not published.

Write-back never changes a human label, because predictions go into item data and annotations live in each annotator's user state. A pre-label shows as a pre-filled answer the annotator can edit, e.g., a choice as the selected option and a free-text answer in the scheme's text box, and no pre_annotation block is needed. The pre-label becomes the annotator's answer only when the annotator edits the pre-label or moves on. A pre-label nobody has edited or moved past therefore never becomes a training label. An annotator-item pair that ai_support.holdout places in the study's no-assistance comparison group shows no pre-label, and the item data sent for that pair leaves out both the writeback.field and the pre_annotation.field. The write-back report is the writeback field of GET /admin/training/api/status, and the report's summary says why a run published nothing.

Potato keeps one prediction per item and scheme, and a newer run's prediction replaces an older run's prediction. When the newer prediction is stored but not shown, because the prediction is under min_confidence, empty, or rejected, the older run's pre-label comes off the item at once, and the write-back report counts the item in n_superseded_cleared. Pre-labels supplied with the data are left alone. At startup the stored predictions for the schemes in writeback.schemas are written back into item data under the same four checks, so pre-labels survive a restart. To withdraw a run's pre-labels, send DELETE /admin/training/api/runs/<run_id>/predictions. The request removes the stored predictions and takes the pre-labels off the items.

Corrections and pinned items always train

Two kinds of item go to the train split of every later run, whatever the item's place in the stable split:

• Items a reviewer corrected. A correct verdict in model review means a person fixed the model's answer. Corrections come from the items the model was worst at, so a corrected item in val or test would score the model on a sample of the model's own mistakes. accept and reject verdicts leave an item where the item was.
• Pinned items. Train on these beside a slice on the Catalog page pins the items the slice matches, e.g., the built-in model-found-nothing slice. A pinned item trains once someone has labeled the item.

A run that moved held-out items into train logs a warning with the count. The run's held-out score is then computed without those items and is not directly comparable with an earlier run's score. The counts are under stats.forced_train in the training data's manifest.json.

Saving a split as a dataset version

Keep this split as a dataset on a run's page saves the run's val, test, or train items with their agreed human labels as a new version of a dataset. A dataset version never changes afterwards, so later models and LLM prompts can be compared on exactly those items. The feature needs datasets.enabled: true. Each saved version holds one split and starts from scratch rather than adding to the dataset's last version. Run 2's val items added to run 1's would otherwise include items run 2 trained on. Each example's inputs leave out the pre-label field, so an evaluation cannot read the model's answer.

Where a pre-label came from

GET /admin/model-review includes provenance for each item in the review queue whose pre-label a training run wrote. For each scheme, the entry gives the run id, the trainer, and the run's held-out score. Pre-labels supplied with the data have no entry.

Where runs are stored

Each run has a directory at <task_dir>/<output_annotation_dir>/training/runs/<run_id>/:

The run ledger lives in the project database, <task_dir>/project.sqlite, and records the runs, each item's split per run, the items pinned to train, the stored predictions, agent tokens, and pushes to the Hugging Face Hub. retain_runs prunes the ledger and never prunes a run whose predictions are still stored, so every pre-label can be traced to the run that wrote the pre-label. Pruning leaves run directories on disk.

A LoRA run saves only the adapter, a few to a few hundred megabytes, and loads the base model by model_name when the run predicts. A full fine-tune saves the whole model. Set the POTATO_MODEL_CACHE environment variable to a directory, and the training process uses that directory for HF_HOME, TRANSFORMERS_CACHE, and SENTENCE_TRANSFORMERS_HOME unless those variables are already set. On an air-gapped machine, put the models the trainers name in that directory in advance.

Routes

The Training page and the page's JSON routes sit under /admin/training and need admin access. Admin access means an admin session or the X-API-Key header. POST and DELETE requests from a browser must be same-origin. Potato's OpenAPI spec marks all 26 training operations with x-potato-requires-config: model_training.enabled: true:

A run request takes trainer, schemas (a list or one name), params, backend, accept_licence, and queue. Every trainer fits only the first scheme named. The request below starts an sklearn-text run on the example's sentiment scheme:

curl -X POST -H "X-API-Key: $KEY" -H "Content-Type: application/json" \
  http://localhost:8000/admin/training/api/runs \
  -d '{"trainer": "sklearn-text", "schemas": ["sentiment"]}'

A refused start gives the reason in error:

Agent protocol

model_training.enabled also registers the routes a training agent calls, under /api/training. An agent sends the agent's token as Authorization: Bearer <token>, and a missing or invalid token gets 401. Registering a backend takes the admin API key instead. potato/training/protocol.py describes the protocol for anyone writing a new agent, and potato/training/agent.py is the reference implementation:

QLoRA

llm-sft, llm-dpo, hf-text-classification, vlm-sft, and reward-model take lora: {quantize: 4bit}, the setup of QLoRA (Dettmers et al., 2023). The base model loads with 4-bit NF4 weights and double quantization, and a LoRA adapter trains on top in bf16 where the GPU supports bf16. quantize: 8bit is the halfway option, and predictions load the base model the same way.

model_training:
  trainer_params:
    llm-sft:
      model_name: Qwen/Qwen2.5-7B-Instruct
      lora: {quantize: 4bit}

In 4 bits the base weights take a quarter of the memory the same weights take in bf16, and activations, the adapter, and the optimizer state come on top. Potato has not yet measured a QLoRA run's memory on a GPU, so plan from the run's own "Peak GPU memory" line. Every transformer run logs that line at the end. QLoRA needs a CUDA GPU and a bitsandbytes build for that machine's CUDA. The run checks both before loading the model by quantizing a few numbers on the GPU, and when the two versions do not match, the error names both versions. The LLM fine-tuning tutorial puts QLoRA to work on SFT and DPO runs.

Publishing to the Hugging Face Hub

A successful run whose model is on the Potato server has Publish to the Hugging Face Hub on the run's page. The push uploads the model and a README.md model card written from the run, naming the trainer, the base model and adapter, the schemes and labels, the split sizes, the held-out score and the split the score was measured on, and the run and bundle ids. The card of a dp run also has a "Differential privacy" row with the epsilon spent against the target, the delta, and the train split the guarantee covers. The row leaves out the target or the delta when the run did not record the value.

The upload is a fixed list of files rather than the whole artifacts/ directory. The list contains the weight files, e.g., .safetensors, .bin, .pt, and .pkl, the config and tokenizer files in the saved model/ directory, model_card.json with local paths replaced by file names, and calibration.json. The model's answers on the project's items (predictions.jsonl, and raw_answers.jsonl for the decision models) stay on the server. Checkpoints, optimizer state, the copies of the training data some trainers make (yolo_data/, wev_data/, laya_data/), and any symlink that points outside artifacts/ are left out as well.

The form guards against an accidental or public push in several ways:

• You type the repository name twice, and the push goes ahead only when the two names match.
• Private is ticked by default. While Private is ticked, Potato does not push to an existing public repository, and the API answers 409.
• The form names an environment variable on the server, HF_TOKEN by default, that holds a Hugging Face write token. The token itself is never typed into the page. The variable's name must contain TOKEN or start with HF_, and the value must start with hf_, so no other variable's value can reach the Hub.
• When the Hub cannot say whether the repository exists, nothing is pushed, and a repository the push created is deleted again if the upload then fails.
• The model card's license is the Hub license id you give, such as apache-2.0. With none, the card says the base model's license applies. A trainer whose license needs confirming, such as ultralytics-detect (AGPL-3.0), needs the same confirmation to push.
• A model left on a remote host with fetch_artifacts: false, or trained on an agent backend, cannot be pushed from the server, and the error names where the model is.

Each push is one commit. The run page lists every push with the repository, visibility, and commit, and the ledger records who made the push. Through the API, the body of POST /admin/training/api/runs/<run_id>/push-to-hub takes repo_id, confirm_repo_id, private (default true), token_env (default HF_TOKEN), licence, and accept_licence. The route returns 201 with the commit, 400 for a mismatch, a missing token variable, no model files, or an unconfirmed license, and 502 when the Hub refuses.

Troubleshooting

The entries below cover the errors most likely on a first setup. The upstream troubleshooting list covers GPU drivers, CUDA builds, and the trainer-specific messages.

• /admin/training returns 404. model_training.enabled is not true, and Potato registers the routes only when the key is true.
• "Only N items in the train split". Annotate more items, or lower min_instances in the run's params. The train split holds 70% of the labeled items by default.
• "Schema 'x' has 1 distinct label(s) in the data". A classifier needs at least two labels in use.
• "No resolved annotations to train on". The scheme named in the run differs from the scheme being annotated, or nobody has annotated anything yet.
• The run published no pre-labels. Read the writeback report in GET /admin/training/api/status. "No predictions were published: write-back is not enabled for any schema" means writeback.schemas is empty, and the skip counts show how many predictions each check removed.
• <trainer> does not take <name>. A parameter in the run or under trainer_params is misspelled or belongs to another trainer. The message lists the names the trainer takes.
• The run was killed after the machine ran out of memory. The process exited on SIGKILL, usually from the out-of-memory killer. Use a smaller model or fewer items, or set max_ram_mb so the run fails inside the run's own limit.
• "Stopped: the run took longer than model_training.max_wall_seconds". The run reached the time limit. Raise max_wall_seconds, or train on fewer items.
• "The GPU ran out of memory". Lower batch_size and raise gradient_accumulation to keep the effective batch, set lora: true or lora: {quantize: 4bit}, or for llm-sft set gradient_checkpointing: true or choose a smaller model.
• "ssh to host failed: Permission denied". Potato runs ssh with BatchMode=yes, so ssh <host> true must succeed from the server's account without a prompt. A key with a passphrase works when ssh-agent holds the key.
• A run on an agent backend stays queued. No agent with that backend name is polling. Start an agent with potato training agent --backend <name>, and check the token with Check on the Training page.

Further reading

• Model training on your own annotations: the training loop, the Training page, and a quick start.
• Trainers: every trainer, the schemes each trainer learns, and the install commands.
• Train on a GPU server, SLURM cluster, or machine behind NAT: ssh, SLURM, and polling-agent backends step by step.
• Model in the loop: write-back, review, auto_retrain, and dataset versions in one study.
• Active learning: ordering items by model uncertainty.
• Detecting rubber-stamped pre-labels: checking whether annotators accept pre-labels without reading the pre-labels.
• Upstream reference: the full parameter tables for every trainer.

References

• Dawid, A. P., and Skene, A. M. (1979). Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. Applied Statistics, 28(1). https://doi.org/10.2307/2346806
• Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. Advances in Neural Information Processing Systems 36 (NeurIPS 2023). https://proceedings.neurips.cc/paper_files/paper/2023/hash/1feb87871436031bdc0f2beaa62a049b-Abstract-Conference.html