Train Detection and Segmentation Models on Image Annotations

New in v2.14.0

This tutorial trains an object detector on a study's own bounding boxes and shows the detector's boxes to the next annotators as pre-labels to correct. The tutorial is for study owners with an image_annotation scheme who have read Train your first model or are comfortable with the admin Training page. The first half runs one complete loop with torchvision-detect (boxes on images, a training run, write-back, corrections, and a second run), and every step there ran on a laptop CPU. The second half adapts the loop to YOLO, outlines, SAM refinement, whole-image classes, whole slides, microscopy, maps, cells, document pages, and vision-language models, one section each.

Installing the vision trainers

torchvision-detect fine-tunes a torchvision Faster R-CNN detector. The trainer needs torch and torchvision. Potato leaves both out of the default install because torch is a multi-gigabyte download. Install the vision extra on the machine that trains:

pip install "potato-annotation[train-vision]"

The same extra covers torchvision-segment, hf-image-classification, sam-finetune, layoutlmv3, and vlm-sft. A trainer whose packages are missing appears on the Training page as "not installed", with the command that installs the packages.

A GPU is optional for the small project below. On a Mac with Apple silicon, torchvision-detect and torchvision-segment train on the CPU rather than on Metal (mps), and the run log records the switch with the warning "torchvision-detect does not run on Apple's Metal (mps) with this torch; using the CPU". In the Potato team's measurements, one Metal training step at the default image size took longer than a full CPU epoch. For larger images or a larger study, train on a GPU server as Remote training describes.

Setting up an image project

The project needs three things: the images, a data file that lists the images, and a config. Put the images in a folder named media inside the project folder, because Potato serves <task_dir>/media/ at the URL path /media/ by default. A different folder works when the config names the folder with media_directory.

shapes/
  config.yaml
  data/images.jsonl
  media/img_001.png
  media/img_002.png
  ...

Each line of data/images.jsonl is one item with an id and the image's URL path:

{"id": "img_001", "image_url": "/media/img_001.png"}

The config defines one image_annotation scheme with the bbox tool and two labels, and turns model training on with write-back. The config passes python -m potato.validate_cli --strict:

annotation_task_name: "Shape detection (model training)"
task_dir: "."
data_files:
  - "data/images.jsonl"
item_properties:
  id_key: "id"
  text_key: "image_url"
output_annotation_dir: "annotation_output"
 
require_password: false
admin_api_key: "demo-admin-key"
num_annotators_per_item: 1
 
annotation_schemes:
  - annotation_type: "image_annotation"
    name: "objects"
    description: "Draw a box around every circle and every square."
    source_field: "image_url"
    tools: ["bbox"]
    labels:
      - name: "circle"
        color: "#e6194b"
      - name: "square"
        color: "#4363d8"
 
model_training:
  enabled: true
  writeback:
    schemas: ["objects"]
    include_unlabeled: true
    min_confidence: 0.5

Four settings shape the loop:

• source_field: "image_url" tells the image widget and the trainer which item field holds the image.
• num_annotators_per_item: 1 gives every image one annotator, so the seed labels cover as many different images as possible.
• writeback.include_unlabeled: true has each run predict on the images nobody has labeled yet. Without the setting, a run predicts only on the held-out val and test images, and those images already have human boxes.
• writeback.min_confidence: 0.5 holds back an image's prediction when the mean score of the image's boxes is below 0.5. The floor applies to the whole image, so annotators never see half of an image's boxes.

Start the server from inside the project folder:

potato start config.yaml -p 8000

Labeling a seed set

torchvision-detect refuses to fit with fewer than 20 annotated images in the train split, and the default split puts 70% of the labeled images in train. Label at least 30 images, and more if the objects vary. Draw the boxes by hand at http://localhost:8000. An image with no boxes is left out of training, because a detector cannot tell an image with no objects from an image nobody finished. Draw every object on an image you submit.

To try the loop before any person labels, the user simulator can draw boxes from a gold file. Each gold entry gives the image id and the scheme's shapes in normalized coordinates:

[
  {"id": "img_001", "objects": [
    {"type": "bbox", "label": "circle",
     "coordinates": {"x": 0.158, "y": 0.849, "width": 0.106, "height": 0.146}}
  ]}
]

python -m potato.simulator --server http://localhost:8000 --users 3 \
  --strategy gold_standard --competence good=1.0 --max-annotations 20 \
  --gold-file data/gold.json --sequential --fast-mode --no-export

The simulator redraws each gold box with a few pixels of jitter, and at the good level the simulator also misses or mislabels some objects, as a careful annotator sometimes does. In our test on 80 synthetic images, three simulated annotators labeled 60 images in about one second.

Training torchvision-detect

Open http://localhost:8000/admin, enter the admin key demo-admin-key, and open http://localhost:8000/admin/training. Pick objects as the scheme and torchvision-detect as the trainer, leave Parameters empty, and click Train now. The same run starts from a script with:

curl -X POST -H "X-API-Key: demo-admin-key" -H "Content-Type: application/json" \
  http://localhost:8000/admin/training/api/runs \
  -d '{"trainer": "torchvision-detect", "schemas": ["objects"]}'

The run starts from torchvision's COCO weights and puts a new box head on the detector for the scheme's two labels. Torch downloads the COCO weights into torch's cache on the first run. The defaults are the small fasterrcnn_mobilenet_v3_large_fpn architecture, 15 epochs, 4 images per step, and early stopping after 4 epochs without a better val mAP. Every shape trains as the shape's bounding box, so a study whose annotators drew polygons still trains a box detector. With several annotators per image, the run trains on the one whole answer that model_training.aggregation picks for the image, because two people's boxes rarely match exactly.

On the Apple M4 Pro CPU we used, 37 training images of 320 by 240 pixels trained in about 106 seconds, and the run stopped after 5 epochs. The run's files land in annotation_output/training/runs/<run_id>/, with the fitted detector at artifacts/model/detector.pt. The run's settings and the device the run used are in artifacts/model_card.json.

Reading the held-out score

Every run scores the detector on the images the detector never trained on. The headline for an image_annotation scheme is map_50, COCO mean average precision with a predicted box counted as a match when the box overlaps an annotated object's bounding box at an IoU of 0.5 or more. The run page also records map_50_95, the average over IoU thresholds from 0.5 to 0.95. Two annotators' boxes for one object rarely overlap at IoU 0.9, so the stricter thresholds partly measure how consistently people draw. Potato therefore leads with the 0.5 threshold. Measuring agreement on bounding boxes shows how to check that consistency directly.

The detector keeps the checkpoint from the epoch with the best val score, so the headline comes from the test split when the test split has images. The Runs table shows the headline in the Held-out score column. The run page lists n_train, n_boxes, each split's size and coverage, and the number of objects annotated and predicted on val and test. In our test run on the 80 synthetic images of colored circles and squares, the detector scored 0.99 on test map_50. Synthetic shapes are far easier to find than real objects, so judge the score on your own images.

Checking the pre-labels

After a successful run, Potato writes the detector's boxes into each eligible image's predictions field. GET /admin/training/api/status reports the outcome in the response's writeback field, with n_written for the images that got pre-labels and a count for every reason an image was held back, e.g., n_skipped_low_confidence below min_confidence and n_skipped_train_split for images the run trained on. Train-split images never get pre-labels, because a model's confidence on the model's own training data says nothing about new images.

Log in at http://localhost:8000 with a new username. Each unlabeled image that passed the write-back checks opens with the detector's boxes already drawn. Each box is stored with source: ai and the detector's score as confidence. The annotator moves or resizes a box, changes a label, deletes a false detection, and draws any object the detector missed. The boxes become the annotator's answer when the annotator edits the boxes or moves on to the next image. A pre-label is never saved while the annotator only looks at the image. The model review queue serves pre-labeled images least confident first, and a reviewer's correct verdict puts the image in the train split of every later run.

Retraining on the corrected boxes

Each corrected image becomes a human label for the next run. Start another torchvision-detect run from the Training page once annotators have worked through a batch of pre-labeled images. A new run replaces the previous run's prediction on each image, and a pre-label on an image the new run trains on is taken down.

To retrain without clicking, add an auto_retrain block. This fragment extends the model_training block above and starts a run after every 25 new annotations, at most once every 10 minutes:

model_training:
  enabled: true
  auto_retrain:
    enabled: true
    trainer: torchvision-detect
    schemas: [objects]
    update_frequency: 25
    min_interval_s: 600
  writeback:
    schemas: [objects]
    include_unlabeled: true
    min_confidence: 0.5

Pre-labels anchor annotators to the model's boxes. Turn write-back on for annotators once the held-out score supports the pre-labels, and see Detecting rubber-stamped pre-labels for a way to check whether annotators accept boxes without looking.

Faster detection with Ultralytics YOLO

ultralytics-detect trains an Ultralytics YOLO detector from a COCO-pretrained checkpoint (yolo11n.pt by default) on the same boxes. YOLO trains and predicts faster than torchvision-detect, and the pre-labels are the same editable boxes. Ultralytics is licensed under AGPL-3.0, and a model trained and distributed with Ultralytics carries the AGPL's terms. Ultralytics sells a separate enterprise license for closed use. Potato's extras leave Ultralytics out, so install the package yourself:

pip install ultralytics

Potato asks you to confirm the license before every run. The Training page shows a confirmation, an API run needs "accept_licence": true in the request body, and an automatic retrain needs auto_retrain.accept_licence: true. Without the confirmation, the API refuses the run with HTTP 400, needs_licence_ack: true, and the message "ultralytics-detect is licensed AGPL-3.0". The request below confirms the license and starts the run:

curl -X POST -H "X-API-Key: demo-admin-key" -H "Content-Type: application/json" \
  http://localhost:8000/admin/training/api/runs \
  -d '{"trainer": "ultralytics-detect", "schemas": ["objects"], "accept_licence": true}'

During a run, the trainer turns off Ultralytics' usage analytics, Ultralytics HUB, the experiment-tracker integrations, and automatic package installs. The changes apply in memory for the run's process, and your Ultralytics settings file stays unchanged. Checkpoints download into ~/.cache/potato/ultralytics. The defaults are 50 epochs at 640 pixels with early stopping after 10 epochs.

Outlines instead of boxes

torchvision-segment trains a torchvision Mask R-CNN on polygon, ellipse, freeform, and brush-mask outlines. Boxes are left out of segmentation training, because a box does not say which pixels belong to the object. Give annotators an outline tool such as polygon or brush. How to annotate image segmentation masks covers when each tool fits.

Each predicted mask is traced into a polygon with at most 64 vertices, so the annotator corrects the pre-label point by point. Where a mask comes out in pieces, the largest piece is kept. The default score_threshold is 0.5 rather than the detector's 0.25. Mask R-CNN's second guesses for one object score between 0.25 and 0.6, and each guess would be one more shape for the annotator to delete. Held-out scores compare the predicted polygons with the annotated outlines. On a Mac, torchvision-segment trains on the CPU. With torch 2.8, the Potato team measured one training step on a 256-pixel image at 57 seconds on Metal and 2.3 seconds on the CPU. Use a GPU server for real studies. ultralytics-segment is the YOLO counterpart (yolo11n-seg.pt), with the same 64-vertex polygons and the same AGPL-3.0 license confirmation.

Turning boxes into outlines with sam-finetune

Boxes are quick to draw and outlines are slow. sam-finetune fine-tunes the mask decoder of Segment Anything (SAM) to turn a box into the outline your annotators would draw. The image encoder stays frozen, and each image's embedding is computed once, so an epoch costs little. Training prompts each annotated outline with the outline's bounding box. The run moves and resizes each prompt box by up to box_jitter (5% by default), so the decoder learns from boxes as loose as a detector's.

sam-finetune refines boxes and needs boxes to start from. The usual sequence on one scheme has two runs:

1. Run torchvision-detect or ultralytics-detect with writeback.include_unlabeled: true, so the unlabeled images carry box pre-labels.
2. Run sam-finetune on the same scheme. Each box pre-label becomes a polygon pre-label, and the polygons replace the boxes.

The scheme needs outlines for SAM to learn from, so give annotators an outline tool beside bbox. On held-out images the annotators' own boxes are the prompts, so val_mask_iou reports the outline's mean IoU when the box is right. Those held-out predictions are scored but never stored or published, because the predictions are made from another annotator's answer. The write-back report counts the held-out predictions in n_skipped_heldout_only. The default checkpoint is facebook/sam-vit-base (Apache-2.0). The fine-tuned decoder serves pre-labels only. The image tool's click-to-segment runs separate ONNX weights in the browser, and a sam-finetune run does not replace those weights.

Classifying whole images

For a question about the whole image, e.g., a radio scheme asking which animal a photo shows, use a classifier rather than a detector. hf-image-classification fine-tunes any model that Hugging Face's AutoModelForImageClassification loads, such as ViT, DeiT, BEiT, ConvNeXt, Swin, or DINOv2. The default model is google/vit-base-patch16-224-in21k. The trainer flips training images at random by default. Set flip to false in Parameters when a flip changes the label, as with text in the image or left and right. lora: true trains a LoRA adapter and the classification head instead of every weight.

A choice scheme counts as an image corpus when at least 80% of the scheme's labeled items carry an image path ending in .jpg, .jpeg, .png, .bmp, .webp, .tif, .tiff, or .gif. On such a corpus, sklearn-text with no named vectorizer fits a linear classifier on CLIP image embeddings. The CLIP baseline needs no GPU, only pip install "potato-annotation[embeddings]" pillow.

Image paths and remote images

The image trainers read the field that the scheme's source_field names. Without source_field, a trainer takes the first non-empty item field among image_url, image, image_path, file_name, url, path, and the audio and video equivalents. Images must be local files inside the project:

• A relative path resolves against task_dir, or against media_directory when the file is not found under task_dir.
• A URL path such as /media/img_001.jpg resolves against media_directory. The annotation page loads the image from the same place.
• A file that an absolute path, ../, or a symbolic link places outside the task directory and media_directory is left out with a warning.
• Remote http(s) images are never downloaded. A run leaves out every item whose image is remote or missing, the run log says how many, and those items get no prediction.

Download remote images into media_directory to train on the images. The vision formats page and potato import cover bringing existing image datasets into a project.

Whole slides and other large images

The detection and segmentation trainers resize each image to a few hundred pixels, and the objects on a 100,000-pixel pathology slide or a satellite scene vanish at that size. For an image_annotation, microscopy, or geo_annotation scheme, Potato therefore cuts large images into tiles when the run's training data is built. Tiling is automatic for slide formats (.svs, .ndpi, .mrxs, .scn, and others) and for images whose longer side is 4,096 pixels or more. Slides need openslide:

pip install "potato-annotation[wsi]"

Training uses the tiles that contain an annotated object plus a sample of empty tiles, and each object is clipped to the object's tile. Prediction covers every tile over tissue, up to max_predict_tiles per image. Each image's split is decided before the image is cut, so tiles of one slide never land in two splits. Tile predictions are merged back onto the whole image, and an object two overlapping tiles both found is kept once. The pre-labels use the whole image's coordinates, the same coordinates the deep-zoom viewer stores. Held-out scores are computed on whole images. A remote backend downloads the tiles rather than the slide.

The settings live under model_training.tiling. This fragment trains a 40x slide at 20x on 1,024-pixel tiles:

model_training:
  enabled: true
  tiling:
    tile_size: 1024
    overlap: 128
    downsample: 2

Keep overlap at least as large as the objects, since a larger object is found in pieces, one per tile. For a slide that records the slide's resolution, mpp sets microns per tile pixel and overrides downsample. Two settings need care on some slides. The empty training tiles assume every object on an annotated image was drawn, so set background: 0 when annotators drew only some regions of a slide. Tissue is detected by Otsu's threshold on the slide thumbnail, so a slide with no bare glass loses the slide's paler half, and tissue_threshold: 0 predicts on every tile. The tiling table lists every key and default.

Microscopy and maps

A microscopy scheme's ROIs train torchvision-detect, ultralytics-detect, torchvision-segment, ultralytics-segment, and cellpose. Each z-slice and time point with ROIs is rendered with the scheme's default channels and contrast and becomes a separate training image. The renderer is the compositor the viewer uses. An image with nothing drawn yet is predicted at the middle z-slice and the first time point, where the viewer opens. Predictions come back as microscopy ROIs with each ROI's z and t. Cell classifications refer to a label image rather than a shape, and the trainers skip cell classifications.

A geo_annotation scheme's features train the same detection and segmentation trainers on the item's raster basemap. A GeoTIFF is placed by the transform and EPSG code in the file's own tags and reprojected with pyproj when projected. A plain image with an extent is placed in longitude and latitude. Polygons, rectangles, lines, and points become shapes on the raster, with polygon holes filled. Predictions come back as GeoJSON features in EPSG:4326 with source: model. The map offers the features as suggestions, and the suggestions count as an answer only once the annotator edits or accepts one. A map whose basemap is a tile server or a GeoJSON outline has no raster to train on, and the run leaves those items out with a warning.

Cells with cellpose

cellpose fine-tunes Cellpose on cell or nucleus outlines in an image_annotation or microscopy scheme. Install Cellpose 3 or 4 separately:

pip install cellpose

The default base model is cyto3. nuclei and cyto2 also work with Cellpose 3, and cpsam with Cellpose 4. Set diameter to the typical cell diameter in pixels when Cellpose's own estimate is off. Cellpose segments without classes, so every predicted cell gets the label parameter, or the scheme's most common label when label is unset. Train one model per label when the classes look different. Each predicted mask comes back as a polygon with the cell's mean probability as the pre-label's confidence. cellpose needs only 5 annotated images in the train split, against 20 for the detectors. Cellpose's code is BSD-3-Clause, and the weights come from the Cellpose authors under the authors' terms.

Spectrograms

A spectrogram_annotation scheme's boxes train torchvision-detect and ultralytics-detect on each recording rendered as a spectrogram image. The audio models tutorial covers spectrogram detection end to end, with the rendering settings and the recording formats the trainers read.

Document layout with layoutlmv3

layoutlmv3 fine-tunes LayoutLMv3 to find the regions annotators draw on page images from the page's words and the words' positions. A region is any labeled block, e.g., a title, a table, or a total. Potato runs no OCR, so each item carries the page's words with boxes in the page image's pixels or from 0 to 1:

{"id": "page_001", "image": "pages/page_001.png",
 "words": [{"text": "INVOICE", "bbox": [40, 52, 180, 82]},
           {"text": "Total", "bbox": [300, 700, 380, 730]}]}

A PAGE XML import in regions mode fills words from the file's Word elements. Words from a PDF's text layer or another OCR tool work the same way. Each word is labeled by the region the word's center falls in, predicted words are merged back into regions, and each region becomes a box pre-label around the region's words. Held-out scores are box mAP. A predicted box ends at the region's words, so regions drawn much larger than the words score lower than the boxes deserve.

The LayoutLMv3 weights are licensed CC BY-NC-SA 4.0 and are not for commercial use. As with the YOLO trainers, the Training page asks for confirmation before a run, an API run needs accept_licence: true, and an automatic retrain needs auto_retrain.accept_licence: true.

Vision-language models with vlm-sft

vlm-sft fine-tunes a vision-language model on annotators' answers about images, with a LoRA adapter by default and Qwen/Qwen2.5-VL-3B-Instruct as the default model. The trainer learns from four schemes:

Coordinates are on a 0 to 1,000 grid of the image, so any checkpoint can learn the format from the examples. A text run publishes the generated answer as a text pre-label. A grounding run publishes a region or an absence for every expression, and VLM grounding covers the grounding_eval scheme. A GUI trajectory run suggests a verdict for each step from where the model would click, and Evaluating computer-use agents covers GUI traces. A caption run is scored with token F1 and BLEU-4 but never published, because the captioned regions are an annotator's answer.

Real checkpoints need a GPU. The Potato team trained two epochs of Qwen/Qwen3-VL-2B-Instruct with LoRA over 26 images in 46 seconds on a Mac's Metal GPU. {"lora": {"quantize": "4bit"}} in Parameters trains a QLoRA adapter over 4-bit base weights. Check the license of the model you pick. Qwen2.5-VL-3B carries the Qwen research license, and the 7B model is Apache-2.0. The LLM fine-tuning tutorial covers LoRA, QLoRA, and publishing a model to the Hugging Face Hub.

Further reading

• Model training overview, for the loop and the Training page
• Trainers, for every trainer grouped by task
• Configuration, for every key, held-out scores, write-back, and routes
• Model-in-the-loop labeling, for review, acquisition, and automatic retraining
• Remote training, for ssh GPU servers, SLURM, and the polling agent
• Image annotation, for the scheme's tools and options
• Model zoo and licenses, for the vision models that run in the browser and on the server
• Upstream reference for image items, whole slides, torchvision-detect, sam-finetune, and vlm-sft, with every parameter table

References

• Ren, S., He, K., Girshick, R., and Sun, J. (2017). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39, 1137–1149. doi:10.1109/TPAMI.2016.2577031
• He, K., Gkioxari, G., Dollár, P., and Girshick, R. (2017). Mask R-CNN. 2017 IEEE International Conference on Computer Vision (ICCV), 2980–2988. doi:10.1109/ICCV.2017.322
• Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., Dollár, P., and Girshick, R. (2023). Segment Anything. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), 3992–4003. doi:10.1109/ICCV51070.2023.00371
• Stringer, C., Wang, T., Michaelos, M., and Pachitariu, M. (2021). Cellpose: a generalist algorithm for cellular segmentation. Nature Methods, 18, 100–106. doi:10.1038/s41592-020-01018-x
• Huang, Y., Lv, T., Cui, L., Lu, Y., and Wei, F. (2022). LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking. Proceedings of the 30th ACM International Conference on Multimedia, 4083–4091. doi:10.1145/3503161.3548112