Pre-Labels and Held-Out Scores from a Model You Already Run

New in v2.14.0

This tutorial is for teams that already run a model outside Potato and want to know how well the model agrees with the team's own annotators before showing the model's answers to anyone. Potato's model training includes five trainers, called bridges, that train nothing. A bridge sends the study's held-out items to a model served elsewhere, scores the answers against the study's labels, and writes the answers back as pre-labels for review, as a run of a trained model does. The steps below cover a vision-language model behind an OpenAI-compatible API, a Roboflow model, a CVAT serverless function, a Hugging Face model, and a Label Studio ML backend, with a validated config for each.

What a bridge run sends and scores

A bridge is a trainer whose fit step only checks that the service answers. The run then sends the items in the run's val and test splits to the service, and with writeback.include_unlabeled: true the items nobody has labeled as well. Potato turns the answers into the scheme's own shapes, spans, or choices and scores the answers on held-out items against the resolved human labels, with the same held-out metrics a trained model gets (mAP50 for boxes, span F1 for spans, macro-F1 for single choices). Write-back, the review queue, and each pre-label's run id work as after any other run.

An item's split depends only on the item's id and model_training.split_seed, so a bridge run and a trained model's run score on the same val and test items while the seed stays fixed. The Runs table on /admin/training then puts a model you already have beside a model trained on the study's labels. Items in the train split get no prediction from a bridge, and write-back never pre-labels a train-split item. At the default split of 70/15/15, a bridge therefore scores on about 30% of the labeled items.

Before you start

The four bridges other than label-studio-ml need requests and Pillow on the Potato server, and the Training page lists a bridge as "not installed" until both packages import. label-studio-ml needs nothing beyond Potato. None of the five bridges installs a model, a GPU library, or torch, because the model runs in the service.

Images have to be local files inside the project, as for every image trainer. A bridge leaves out an item whose image is a remote http(s) URL, and an item with a label and no prediction counts as wrong in the held-out score. The run reports the share of held-out items that got a prediction as val_coverage and test_coverage, so a low coverage figure points to images the bridge could not send.

Keys stay in environment variables

A key never goes in the config. Each bridge that sends a key takes a parameter that names the environment variable holding the key (api_key_env, token_env for Hugging Face, or the two basic_auth_*_env names for Label Studio), and the variable has to be set in the environment of the Potato server before the server starts. cvat-nuclio sends no key and takes no such parameter. The four bridges that take a key refuse a value that is not a variable name, and the error does not repeat the value, since a value in that place is most likely the key itself. When a service's error body or a connection error echoes the request, the bridges replace the key's value with [redacted] before showing the error. Error bodies from a Label Studio ML backend's /predict are cleaned the same way. Anything shaped like a key in a query string or an Authorization header is replaced as well.

hf-inference and roboflow runs are local only. Both bridges read the key on the Potato server, and a run on an ssh, SLURM, or agent backend would have to ship the key to that host, so Potato refuses to start such a run there. openai-vlm-boxes, cvat-nuclio, and label-studio-ml run on any backend that can reach the service, with the key's variable set on that backend's host.

Step 1: An image study with three bridges

The config below sets up a study of site photos in which annotators box every hard hat and every bare head. model_training.trainer_params holds defaults for several trainers at once, so the one config carries the settings for all three image bridges, and each run uses the block for its own trainer. The Parameters field on the Training page fills in from these defaults when you choose a trainer.

annotation_task_name: "Hard hats on site photos"
task_dir: .
media_directory: media
output_annotation_dir: annotation_output
admin_api_key: "replace-with-a-long-random-string"
 
data_files:
  - data/photos.jsonl
 
item_properties:
  id_key: id
  text_key: image_url
 
annotation_schemes:
  - annotation_type: image_annotation
    name: ppe
    description: "Box every hard hat and every bare head."
    source_field: image_url
    tools: [bbox, polygon]
    labels:
      - name: helmet
      - name: head
 
model_training:
  enabled: true
  writeback:
    schemas: [ppe]
    include_unlabeled: true
  trainer_params:
    openai-vlm-boxes:
      url: http://localhost:8000/v1
      model: Qwen/Qwen2.5-VL-7B-Instruct
      coordinate_format: pixels
    roboflow:
      model_id: site-safety/3        # your Roboflow project and version
      url: https://detect.roboflow.com
      api_key_env: ROBOFLOW_API_KEY
      confidence: 0.4
      label_map: {hardhat: helmet, no-hardhat: head}   # the model's class names on the left
    cvat-nuclio:
      url: http://localhost:32768
      threshold: 0.5
      label_map: {person: null}

Each line of data/photos.jsonl names one image under media/:

{"id": "site_001", "image_url": "media/site_001.jpg"}

writeback.schemas is what makes the model's boxes visible to annotators, and include_unlabeled sends the photos nobody has labeled to the service as well. Leave both keys out for the first run if you want the held-out score before anyone sees a pre-label. label_map maps the service's class names to the scheme's labels, and a class mapped to null is dropped. A class without an entry goes to the label of the same name, and the match ignores case. Boxes under any other class are left out, and the run log counts the boxes left out.

Step 2: A vision-language model with openai-vlm-boxes

openai-vlm-boxes shrinks each image to at most 1,280 pixels on the longer side (max_image_side) and sends the image to the chat completions route with a prompt naming the scheme's labels. Potato reads the reply as JSON boxes. Any server with an OpenAI-compatible API works. The command below serves Qwen2.5-VL with vLLM, whose documentation connects clients at http://localhost:8000/v1:

vllm serve Qwen/Qwen2.5-VL-7B-Instruct

With vLLM on port 8000, start Potato on another port:

potato start config.yaml -p 9000

Vision-language models answer in their own coordinate convention whatever the prompt asks for, and two parameters tell Potato how to read the numbers. coordinate_format is fraction (0 to 1 of the image), per_mille (0 to 1,000), or pixels of the image as sent. box_order is xyxy for [x0, y0, x1, y1] or yxyx for [y0, x0, y1, x1]. Qwen2.5-VL answers in pixels, so the config above sets coordinate_format: pixels. Under auto, Potato guesses the format once per reply from the size of the numbers. Values up to 1.05 read as fractions, values past the sent image's size read as per-mille, and anything else reads as pixels. A per-mille box that fits inside a large image therefore reads as pixels, so name the format whenever you know the model's convention.

Gemma and Gemini answer per-mille boxes with y first, [ymin, xmin, ymax, xmax], even when the prompt asks for fractions. With both parameters on auto, Potato reads the boxes of a model whose name contains gemma or gemini as per-mille with y first. On a Gemma 4 12B test project of red and blue squares, the Potato team measured mAP50 at 0.0 with the boxes read x first and 1.0 with the boxes read y first. Ollama serves Gemma 3 under a name containing gemma, so auto applies:

ollama run gemma3:12b

Replace the openai-vlm-boxes block in the config with the block below:

model_training:
  trainer_params:
    openai-vlm-boxes:
      url: http://localhost:11434/v1
      model: gemma3:12b
      json_mode: object

A Gemma or Gemini fine-tune served under a name without either word gets the x-first reading under auto. Set coordinate_format: per_mille and box_order: yxyx for such a model. json_mode: object asks Ollama for JSON with response_format: json_object. The default, schema, sends a JSON schema with the labels as an enum. vLLM and OpenAI honor the schema. A run in which no reply parses as JSON fails, and the error suggests json_mode: object or a plainer prompt (prompt, where {labels} becomes the scheme's labels).

A vision-language model gives no score with its boxes, so the boxes carry no confidence. writeback.min_confidence passes every openai-vlm-boxes prediction, and model review cannot order the predictions from least to most confident. The openai-vlm-boxes reference lists api_key_env for a hosted API and the token and timeout settings.

Step 3: Start a run and read the score

Open http://localhost:9000/admin/training, choose openai-vlm-boxes and the ppe scheme, and click Train now. The same run starts from the API:

curl -X POST -H "X-API-Key: $KEY" -H "Content-Type: application/json" \
  http://localhost:9000/admin/training/api/runs \
  -d '{"trainer": "openai-vlm-boxes", "schemas": ["ppe"]}'

The fit step lists the server's models through /models and logs which model the run asks and how the run reads the model's boxes. The prediction pass then sends one image per request. When the run finishes, the held-out mAP50 appears in the Runs table, and the run page lists val_map_50, test_map_50, the coverage figures, and a "Not used" warning that counts replies that were not JSON, boxes under labels the scheme lacks, and boxes off the image. When the score is near 0 while coverage is high and the warning is empty, check coordinate_format and box_order first.

Roboflow models

roboflow posts each image to <url>/<model_id>, where model_id is the project and version as project/version. The key goes in ROBOFLOW_API_KEY by default (api_key_env), and the run refuses to start without the key:

export ROBOFLOW_API_KEY=...   # in the shell that starts Potato
potato start config.yaml -p 9000

Potato's default url is https://detect.roboflow.com, and the config above names the default. A run for instance segmentation sets url: https://outline.roboflow.com. url is a parameter, so a run can also name https://serverless.roboflow.com, the host Roboflow's object detection reference gives for the same project/version route. The Potato team has not tested that host with this bridge. A self-hosted Roboflow Inference server listens on port 9001 by default (url: http://localhost:9001).

Roboflow's API takes the key in the query string. Potato does not write the request URL to the run log or to an error message. Potato sends confidence and overlap (defaults 0.4 and 0.3) to the API as percentages, and Roboflow's reference documents both on a 0 to 100 scale. shapes: auto writes an outline where the model returns an outline and the scheme has a polygon tool, and a box otherwise. The health check posts an 8 × 8 white image, and Roboflow counts the request against the account's usage.

CVAT serverless functions

cvat-nuclio uses a detector already deployed for CVAT as a Nuclio function, such as the YOLOv7 or Faster R-CNN functions in cvat/serverless. Potato posts each image to the function's own URL as {"image": <base64>, "threshold": t}, without going through CVAT. nuctl get functions lists the deployed functions with their ports, and the command below is the form CVAT's documentation gives:

nuctl get functions --platform local

The function answers with shapes in pixels. A rectangle becomes a box, and a polygon becomes a polygon, or the polygon's bounding box when the scheme has no polygon tool. mask shapes are left out and counted. The function drops shapes scoring below threshold (default 0.5). The scheme has no person label, so label_map: {person: null} in the config drops a COCO detector's people.

Hugging Face models with hf-inference

hf-inference calls a model deployed on Hugging Face, either a dedicated Inference Endpoint by its https:// URL (endpoint_url) or a model id on the serverless HF Inference API (model). Set endpoint_url or model, and not both. The task follows the scheme's type:

• Text classification for radio, select, and multiselect. A single-choice scheme takes the model's top label. When that label is not one of the scheme's labels after label_map, the item gets no prediction rather than the runner-up. A multiselect scheme takes every label scoring at least threshold (default 0.5).
• Token classification for span, with word pieces grouped into entities by aggregation_strategy (simple, first, average, or max) and character offsets taken from the model.
• Object detection for image_annotation. Boxes scoring below threshold are dropped.

The config below asks dslim/bert-base-NER for spans. The model card describes a BERT model fine-tuned for named entities on English CoNLL-2003. HF Inference serves a subset of the Hub's models, so check that the model's page lists HF Inference before relying on the serverless API.

annotation_task_name: "Named entities in news sentences"
task_dir: .
output_annotation_dir: annotation_output
admin_api_key: "replace-with-a-long-random-string"
 
data_files:
  - data/sentences.jsonl
 
item_properties:
  id_key: id
  text_key: text
 
annotation_schemes:
  - annotation_type: span
    name: entities
    description: "Highlight every person, organization, and place."
    labels: [person, organization, location]
 
model_training:
  enabled: true
  writeback:
    schemas: [entities]
    include_unlabeled: true
  trainer_params:
    hf-inference:
      model: dslim/bert-base-NER
      token_env: HF_TOKEN
      aggregation_strategy: simple
      label_map: {PER: person, ORG: organization, LOC: location, MISC: null}

The model's PER, ORG, and LOC entities map to the scheme's labels, and MISC is dropped because the scheme has no such label. When an entity's full name matches nothing, Potato drops a B- or I- prefix and tries again. A sentiment classifier that answers LABEL_0 and LABEL_1 needs label_map: {LABEL_0: negative, LABEL_1: positive} in the same way. The health check is one real request, a short sentence or an 8 × 8 image, and wait_for_model: true (the default) waits while a serverless model loads instead of failing with a 503.

A Label Studio ML backend

label-studio-ml uses a running Label Studio ML backend. A backend started from the repository's examples with docker-compose up serves a model over HTTP at http://localhost:9090. The run checks the backend's GET /health, sends the items to POST /predict in batches of 16 (batch_size), and reads the returned regions with the same converter that potato import -f label_studio uses. Label Studio's UTF-16 span offsets become code points, and percent coordinates become the image widget's.

annotation_task_name: "Review sentiment"
task_dir: .
output_annotation_dir: annotation_output
admin_api_key: "replace-with-a-long-random-string"
 
data_files:
  - data/reviews.jsonl
 
item_properties:
  id_key: id
  text_key: text
 
annotation_schemes:
  - annotation_type: radio
    name: sentiment
    description: "Is this review positive or negative?"
    labels: [positive, negative]
 
model_training:
  enabled: true
  writeback:
    schemas: [sentiment]
    include_unlabeled: true
    min_confidence: 0.6
  trainer_params:
    label-studio-ml:
      url: http://localhost:9090
      basic_auth_user_env: LS_ML_USER
      basic_auth_pass_env: LS_ML_PASS

A stock backend's only authentication is HTTP Basic, set by BASIC_AUTH_USER and BASIC_AUTH_PASS on the backend. basic_auth_user_env and basic_auth_pass_env name the variables on the Potato server that hold the same two values, and the two parameters go together. api_key_env sends Authorization: Token <value> instead, for a backend behind a proxy that expects a token. A stock backend answers that header with a 500. Potato generates the labeling config sent with each request from the scheme (Choices, Labels, RectangleLabels, or PolygonLabels), and label_config replaces the generated XML with your own.

An image backend needs one more setting. Without media_url_prefix, Potato sends each image as an absolute path on the Potato machine, and a backend built on the Label Studio SDK reads data.image as a URL and fails on a bare path. For a backend on the same machine, add the prefix below and start the backend with LOCAL_FILES_DOCUMENT_ROOT set to the task directory:

model_training:
  trainer_params:
    label-studio-ml:
      url: http://localhost:9090
      media_url_prefix: /data/local-files/?d=

For a backend on another machine, serve the task directory over HTTP and give that server's URL as the prefix. An image outside the task directory has no path relative to the directory, so Potato does not send the image, and the run log names how many were left out.

Potato sends the backend prediction requests and no annotation events, so the backend's own training in its /webhook handler does not start from Potato. potato import -f label_studio moves a whole Label Studio project into Potato with the project's labeling config, annotators' answers, and stored predictions (Importing Projects). Migrating from Label Studio to Potato covers the tags the importer leaves out.

Review the pre-labels

A bridge's predictions pass through the same write-back checks as a trained model's. Write-back skips train-split items and schemes missing from writeback.schemas, skips item and scheme pairs a reviewer rejected, and holds back predictions under writeback.min_confidence. For boxes and spans, a prediction's confidence is the confidence of the least confident shape or span, so the floor holds back an item's whole prediction when any one box or span on the item scores lower. Roboflow's confidence and CVAT's threshold act per box instead, at the service, and hf-inference's threshold acts per box or per multiselect label. The model-in-the-loop tutorial covers the review queue, accept and reject verdicts, and retraining. The queue and the verdicts work the same way for a bridge.

One prediction is kept per item and scheme, so a later run of another bridge or of a trained model replaces the earlier run's pre-labels. To withdraw a run's pre-labels, send DELETE /admin/training/api/runs/<run_id>/predictions. The inference bridges reference lists every parameter of the four bridges, and the label-studio-ml reference lists the Label Studio parameters.

Further reading

• Model Training covers the loop and the Training page.
• Trainers lists all 38 trainers by task, for a model trained on the study's labels to compare with a bridge's score.
• Configuration documents splits, write-back, and the routes.
• Image Annotation and Span Annotation document the schemes the bridges fill.
• Visual AI Support shows a vision model's suggestions to an annotator on request, without a held-out score.