Model Trainers in Potato, Grouped by Task

New in v2.14.0

Potato's model training has 38 trainers, and each trainer fits one kind of model to the answers of one kind of annotation scheme. The tables on this page list every trainer by task, with the schemes the trainer learns from, the items the trainer reads, the install the trainer needs, the pre-label the trainer writes back, and the held-out score each run reports. The page is for a study owner choosing a trainer before a first run, or checking whether a scheme can be trained at all. Config keys, backends, and write-back rules are on the configuration page. The full parameter table for each trainer is in the trainer's section of the upstream reference, and each trainer name below links to that section.

Trainers offered for each scheme

The Training page at /admin/training offers each scheme only the trainers that fit the scheme's kind and the modality of the scheme's items, e.g., text, images, or audio. A trainer the project cannot run stays in the list with the reason, as decision-calibrate shows "no decision model is configured" until the project has a decision model. A trainer whose packages are missing is listed as "not installed", with the command that installs the packages. Until you pick a trainer, sklearn-text is selected wherever sklearn-text fits. A run started for a scheme that does not exist, or for a scheme the trainer cannot learn, is refused with HTTP 400 before anything is built.

Each run trains on one scheme. The Training page sends one scheme, and the training API accepts a list of schemes but every trainer fits only the first scheme in the list. To train two schemes, start two runs.

In the tables below, the Pre-label column says what a run writes onto an item once the scheme is listed under writeback.schemas. Items in a run's train split never get a pre-label, and a pre-label becomes an annotator's answer only when the annotator edits the pre-label or moves on. The Held-out score column names the headline metric for the run, computed on the val and test items against the annotators' resolved labels. An item with a label and no prediction counts as wrong. A trainer therefore cannot raise the score by skipping hard items. Held-out scores in the upstream reference defines every metric.

Installing trainers

sklearn-text needs nothing beyond a plain install of Potato, because scikit-learn is a core dependency. The text, image, audio, and LLM trainers need PyTorch and transformers. One extra installs the packages for all four groups:

pip install "potato-annotation[train]"

The narrower extras [train-text], [train-vision], [train-audio], and [train-llm] each install one group, and the Install column below names the narrowest extra that covers each trainer, as in pip install "potato-annotation[train-vision]". None of the training extras is part of potato-annotation[all], because PyTorch is a multi-gigabyte install. Some trainers need packages outside [train]:

• ultralytics-detect and ultralytics-segment need pip install ultralytics.
• cellpose needs pip install cellpose, and works with Cellpose 3 or Cellpose 4.
• laya-finetune needs [decision], and openjev-finetune needs pip install wev-ai.
• rdkit-property needs [chemistry], volume-unet needs [volumes], video-track needs opencv-python, and WFDB or Parquet signals need [signals]. Whole-slide images need [wsi] for openslide.
• The dp option of hf-text-classification and hf-image-classification needs pip install opacus.

ultralytics-detect, ultralytics-segment, and layoutlmv3 ask you to confirm a license before the first run. Ultralytics is AGPL-3.0, so a model trained and distributed with either YOLO trainer comes under the AGPL's terms, and Ultralytics sells a separate license for closed use. The LayoutLMv3 weights are CC BY-NC-SA 4.0 and are not for commercial use. For all three trainers, the Training page asks for confirmation, an API run needs accept_licence: true, and an automatic retrain needs auto_retrain.accept_licence: true. Cellpose's code is BSD-3-Clause, and the Cellpose authors publish Cellpose's pretrained weights under their own terms. Licenses also differ between the checkpoints you name in model_name. Qwen2.5-VL-3B is under the Qwen research license, for example, while Qwen2.5-VL-7B is Apache-2.0.

Parameters

A run's parameters come from three places, and each place overrides the one before:

1. the trainer's defaults, listed in the trainer's upstream section;
2. model_training.trainer_params.<trainer> in the config;
3. the run's own params, i.e., the Parameters field on the Training page, the params of an API run, or auto_retrain.params.

Add trainer_params to the model_training block to set a default for each trainer you use:

model_training:
  enabled: true
  trainer_params:
    hf-text-classification:
      model_name: answerdotai/ModernBERT-base
      epochs: 4
    torchvision-detect:
      model_name: fasterrcnn_resnet50_fpn_v2
      score_threshold: 0.4

The Parameters field on the Training page is JSON, filled in from trainer_params for the chosen trainer. A parameter name the trainer does not take is an error. A name under trainer_params stops the server at startup, and a name in a run's params stops the run before the run starts. Either error lists the names the trainer takes. For example, a misspelled score_thresh under torchvision-detect is reported with score_threshold in the list. Potato also checks each value against the parameter's type and range.

Text and image classifiers

The classifiers learn from the choice schemes radio, select, likert, and multiselect. A radio, select, or likert pre-label selects one answer, and a multiselect pre-label ticks a set of labels. sklearn-text runs on a CPU with no extra install, and the first-model tutorial walks through a run with sklearn-text. On an image corpus, sklearn-text fits the classifier to CLIP image embeddings. setfit is built for the first days of a project and works from about eight labeled items per label. The scheme pages for radio and multiselect, select, and Likert scales cover the annotation side.

hf-text-classification fine-tunes a transformer, and with lora: true also fine-tunes a decoder LLM such as Qwen or Llama. With soft_targets: true, hf-text-classification trains toward the annotators' vote split rather than toward one resolved label. hf-text-classification and hf-image-classification take an acquisition parameter that orders a single-label scheme's next items to label by margin, entropy, BALD, or coreset distance, as the model-in-the-loop tutorial shows. Both trainers also take dp for training with DP-SGD through Opacus, and the differential privacy tutorial walks through a dp run.

Spans, relations, and structure

The trainers in this group learn spans with character offsets, links between spans, answers on the nodes of a conversation tree, and interlinear glosses. hf-relation trains one encoder to tag the spans and to classify pairs of spans. A predicted link counts as found only when the tagger also found both of the link's spans with the right boundaries. When every training answer of a span_link scheme is a dependency tree, hf-relation decodes trees and adds UAS and LAS to the score. A run follows the same steps as the first-model tutorial, with a different trainer and scheme chosen.

LLMs and reward models

The LLM trainers learn from free-text answers, preferences, rankings, and corrected agent traces. llm-sft and llm-dpo train a LoRA adapter by default, and lora: {quantize: 4bit} loads the base model in 4 bits as QLoRA. reward-model trains a scalar reward model from pairwise, ranking, and best-worst answers, and each prediction keeps the per-option rewards beside the scheme's answer. prm-reward rates each step of an agent trace correct, neutral, or wrong. Steps past max_length (1,024 tokens by default) are neither trained on nor rated. A trajectory_edit prediction is scored and stored but never shown as a pre-label, and the server refuses to start with a trajectory_edit scheme under writeback.schemas. The LLM fine-tuning tutorial covers SFT, DPO, reward models, process reward models, and QLoRA.

Image detection and segmentation

The detectors and segmenters learn the shapes annotators draw in an image_annotation scheme, and most of the detectors and segmenters also read microscopy, map, and spectrogram schemes. A detector trains on each shape's bounding box, so a polygon or mask study still trains a box detector. A segmenter trains on outlines and leaves boxes out, and predicts polygons of at most 64 vertices that the annotator can edit point by point. cellpose segments without classes, so every predicted cell gets the label parameter, or the scheme's most common label when label is unset.

sam-finetune fine-tunes the mask decoder of Segment Anything to turn a box into an outline while the image encoder stays frozen. A sam-finetune run outlines the box pre-labels that an earlier detection run wrote back, so the usual order is a detection run with writeback.include_unlabeled: true, then a sam-finetune run on the same scheme. On a Mac, torchvision-detect and torchvision-segment train on the CPU rather than on Metal. The vision models tutorial walks through detection, segmentation, slides, microscopy, maps, Cellpose, and SAM.

The image trainers read local files inside the task directory or media_directory. A remote http(s) image is not downloaded, and a row whose image is remote or missing is left out of the run and gets no prediction. Image items in the upstream reference gives the fields Potato reads an image path from.

Slides, microscopy, maps, and spectrograms

The detection and segmentation trainers resize each image to a few hundred pixels. For an image_annotation, microscopy, or geo_annotation scheme, Potato therefore cuts a large image into tiles when the run's bundle is built. Tiling is on by default for slide formats such as .svs and .ndpi and for images whose longer side is at least 4,096 pixels. The tiles of one slide always land in one split, and predicted shapes are mapped back onto the whole image in the coordinates the deep-zoom viewer stores. The tiling table lists every key, and the vision models tutorial sets the tile size, overlap, and resolution for a slide.

A microscopy scheme's ROIs train the detection and segmentation trainers and cellpose. Each z-slice and time point with ROIs is rendered with the scheme's default channels and contrast, and predictions come back as ROIs with each ROI's z and t. A geo_annotation scheme trains on the item's raster basemap, i.e., a GeoTIFF or an image with an extent, and predictions come back as GeoJSON in EPSG:4326 that the map shows as suggestions. A map whose basemap comes from a tile server has no raster to train on, and the run leaves those items out with a warning. A spectrogram_annotation scheme trains torchvision-detect and ultralytics-detect on each recording rendered as a spectrogram, and the boxes come back in seconds and hertz. Microscopy and maps and Spectrograms in the upstream reference give the rendering details.

Documents and vision-language models

layoutlmv3 learns document layout from the words on a page and the words' positions, and vlm-sft fine-tunes a vision-language model on answers about images. Potato runs no OCR, so each layoutlmv3 item needs a words field with each word's text and box. A PAGE XML import in regions mode fills words, and words from a PDF's text layer or another OCR tool work the same way. vlm-sft trains a LoRA adapter by default, and real checkpoints need a GPU. A region_caption run is scored but not published, because the regions a caption describes are an annotator's answer. The vision models tutorial covers both trainers, and VLM grounding covers the grounding_eval scheme.

Audio and speech

The audio trainers read recordings from local files that soundfile opens, e.g., WAV or FLAC, and an item whose audio is a URL is left out. hf-audio-tagger tags labeled time segments in an audio_annotation scheme. whisper-asr fine-tunes Whisper on the transcript turns annotators corrected, and by default also writes a CTranslate2 copy of the model for potato transcripts --transcribe and faster-whisper. diarization fine-tunes a speech encoder (WavLM by default) to find who speaks when. Speaker names are local to each recording. Spectrogram boxes train the detectors in Image detection and segmentation. The audio models tutorial walks through Whisper, diarization, spectrogram detection, and audio tagging.

Volumes, point clouds, and video

The 3D and video trainers learn from painted volumes, point-cloud cuboids, and video tracks. volume-unet trains a 3D U-Net in plain PyTorch on the masks annotators painted in CT and MR volumes, and a full-size CT needs a GPU in practice. external-3d runs your own 3D detector through the train_command and predict_command parameters and reads the cuboids back. OpenPCDet and MMDetection3D both fit, and both need CUDA operators compiled on the GPU host. video-track fine-tunes a Faster R-CNN on frames sampled from the annotated tracks and links the detections across frames into tracks. The volumes, point clouds, and video tutorial walks through all three trainers.

Science

The science trainers learn from chemistry, spectroscopy, physiological signals, and robot episodes. rdkit-property answers a question about a molecule from the molecule's Morgan fingerprint and RDKit descriptors, with a scikit-learn classifier for choices or a regressor for a slider. peak-cnn predicts peak positions on a spectrum and does not predict peak labels or assignments. signal-cnn stages fixed epochs or finds labeled intervals in EEG, ECG, and other signals. episode-model predicts phases, an outcome, and a reward curve from a robot episode's joint and sensor series. The episode's video is not read. Peaks, intervals, and episode answers show as suggestions for the annotator to review rather than as the answer itself. The science models tutorial walks through all four trainers, and the scheme pages for chemical structures, spectra, time series, and robot episodes cover the annotation side.

Decision models

A decision model returns a probability for each option of a question, and Potato uses a decision model as a labeling engine source or as a labeling model in Solo Mode. The decision-model trainers fit a decision model to the study's labels on radio, select, likert, and slider schemes, and ask each question the way the labeling engine asks the question. A slider is asked as a rating with one option per position, up to 21 positions. decision-calibrate trains nothing. The trainer fits a calibration to the probabilities from Jev, OpenJev, laya-serve, or Laya, and writes the calibration to calibration.json for the engine source to apply. decision-calibrate runs on the local backend only. laya-finetune fine-tunes Laya, and openjev-finetune fine-tunes the 8B OpenJev model with wev's LoRA training. The OpenJev model card notes that some of OpenJev's training data is for research only. Check those terms before commercial use.

Inference bridges

The inference bridges use a model that already runs somewhere else, and train nothing. A bridge run sends the held-out items to the service and reads the answers back. With writeback.include_unlabeled: true, the run sends the unlabeled items as well. The answers then get a held-out score against your annotators, write-back, and review, the same as a trained model's answers. A bridge never takes an API key in the config. Instead, a parameter names the environment variable on the server that contains the key. label-studio-ml calls a Label Studio ML backend's /predict and does not send annotation events to the backend's /webhook. hf-inference and roboflow run on the local backend only. Boxes from openai-vlm-boxes have no confidence, because the vision-language model returns no score. The bring-your-own-model tutorial sets up each bridge.

Further reading

• The model training overview explains the loop from annotations to pre-labels and the Training page.
• The configuration page documents every model_training key, the backends, write-back, and the routes.
• The first-model tutorial runs sklearn-text end to end with no GPU.
• The remote training tutorial sends runs to a GPU server over ssh, to a SLURM cluster, or to a polling agent.
• The model-assisted labeling guide discusses when pre-labels from your own model help annotators and when pre-labels anchor annotators.
• The upstream model training reference has every trainer's parameter table.