# Pre-Labels and Held-Out Scores from a Model You Already Run Source: https://www.potatoannotator.com/docs/model-training/bring-your-own-model *New in v2.14.0* This tutorial is for teams that already run a model outside Potato and want to know how well the model agrees with the team's own annotators before showing the model's answers to anyone. Potato's [model training](/docs/model-training/overview) includes five trainers, called bridges, that train nothing. A bridge sends the study's held-out items to a model served elsewhere, scores the answers against the study's labels, and writes the answers back as pre-labels for review, as a run of a trained model does. The steps below cover a vision-language model behind an OpenAI-compatible API, a Roboflow model, a CVAT serverless function, a Hugging Face model, and a Label Studio ML backend, with a validated config for each. ## What a bridge run sends and scores A bridge is a trainer whose fit step only checks that the service answers. The run then sends the items in the run's `val` and `test` splits to the service, and with `writeback.include_unlabeled: true` the items nobody has labeled as well. Potato turns the answers into the scheme's own shapes, spans, or choices and scores the answers on held-out items against the resolved human labels, with the same [held-out metrics](https://github.com/davidjurgens/potato/blob/master/docs/ai-intelligence/model_training.md#held-out-scores) a trained model gets (mAP50 for boxes, span F1 for spans, macro-F1 for single choices). Write-back, the review queue, and each pre-label's run id work as after any other run. | Trainer | Schemes | The model it calls | |---|---|---| | `openai-vlm-boxes` | `image_annotation` | A vision-language model behind an OpenAI-compatible chat API (vLLM, Ollama, LM Studio, llama.cpp's server, OpenAI) | | `roboflow` | `image_annotation` | A Roboflow detection or instance-segmentation model, hosted or on a self-hosted Inference server | | `cvat-nuclio` | `image_annotation` | A CVAT serverless function running on Nuclio | | `hf-inference` | `radio`, `select`, `multiselect`, `span`, `image_annotation` | A Hugging Face Inference Endpoint or a model on the serverless HF Inference API | | `label-studio-ml` | `radio`, `select`, `multiselect`, `span`, `image_annotation` | A running Label Studio ML backend | An item's split depends only on the item's id and `model_training.split_seed`, so a bridge run and a trained model's run score on the same `val` and `test` items while the seed stays fixed. The Runs table on `/admin/training` then puts a model you already have beside a model trained on the study's labels. Items in the `train` split get no prediction from a bridge, and write-back never pre-labels a train-split item. At the default split of 70/15/15, a bridge therefore scores on about 30% of the labeled items. ## Before you start The four bridges other than `label-studio-ml` need `requests` and Pillow on the Potato server, and the Training page lists a bridge as "not installed" until both packages import. `label-studio-ml` needs nothing beyond Potato. None of the five bridges installs a model, a GPU library, or torch, because the model runs in the service. Images have to be local files inside the project, as for every image trainer. A bridge leaves out an item whose image is a remote `http(s)` URL, and an item with a label and no prediction counts as wrong in the held-out score. The run reports the share of held-out items that got a prediction as `val_coverage` and `test_coverage`, so a low coverage figure points to images the bridge could not send. ### Keys stay in environment variables A key never goes in the config. Each bridge that sends a key takes a parameter that names the environment variable holding the key (`api_key_env`, `token_env` for Hugging Face, or the two `basic_auth_*_env` names for Label Studio), and the variable has to be set in the environment of the Potato server before the server starts. `cvat-nuclio` sends no key and takes no such parameter. The four bridges that take a key refuse a value that is not a variable name, and the error does not repeat the value, since a value in that place is most likely the key itself. When a service's error body or a connection error echoes the request, the bridges replace the key's value with `[redacted]` before showing the error. Error bodies from a Label Studio ML backend's `/predict` are cleaned the same way. Anything shaped like a key in a query string or an `Authorization` header is replaced as well. `hf-inference` and `roboflow` runs are local only. Both bridges read the key on the Potato server, and a run on an ssh, SLURM, or agent backend would have to ship the key to that host, so Potato refuses to start such a run there. `openai-vlm-boxes`, `cvat-nuclio`, and `label-studio-ml` run on any [backend](/docs/model-training/remote-training) that can reach the service, with the key's variable set on that backend's host. ## Step 1: An image study with three bridges The config below sets up a study of site photos in which annotators box every hard hat and every bare head. `model_training.trainer_params` holds defaults for several trainers at once, so the one config carries the settings for all three image bridges, and each run uses the block for its own trainer. The Parameters field on the Training page fills in from these defaults when you choose a trainer. ```yaml annotation_task_name: "Hard hats on site photos" task_dir: . media_directory: media output_annotation_dir: annotation_output admin_api_key: "replace-with-a-long-random-string" data_files: - data/photos.jsonl item_properties: id_key: id text_key: image_url annotation_schemes: - annotation_type: image_annotation name: ppe description: "Box every hard hat and every bare head." source_field: image_url tools: [bbox, polygon] labels: - name: helmet - name: head model_training: enabled: true writeback: schemas: [ppe] include_unlabeled: true trainer_params: openai-vlm-boxes: url: http://localhost:8000/v1 model: Qwen/Qwen2.5-VL-7B-Instruct coordinate_format: pixels roboflow: model_id: site-safety/3 # your Roboflow project and version url: https://detect.roboflow.com api_key_env: ROBOFLOW_API_KEY confidence: 0.4 label_map: {hardhat: helmet, no-hardhat: head} # the model's class names on the left cvat-nuclio: url: http://localhost:32768 threshold: 0.5 label_map: {person: null} ``` Each line of `data/photos.jsonl` names one image under `media/`: ```json {"id": "site_001", "image_url": "media/site_001.jpg"} ``` `writeback.schemas` is what makes the model's boxes visible to annotators, and `include_unlabeled` sends the photos nobody has labeled to the service as well. Leave both keys out for the first run if you want the held-out score before anyone sees a pre-label. `label_map` maps the service's class names to the scheme's labels, and a class mapped to `null` is dropped. A class without an entry goes to the label of the same name, and the match ignores case. Boxes under any other class are left out, and the run log counts the boxes left out. ## Step 2: A vision-language model with `openai-vlm-boxes` `openai-vlm-boxes` shrinks each image to at most 1,280 pixels on the longer side (`max_image_side`) and sends the image to the chat completions route with a prompt naming the scheme's labels. Potato reads the reply as JSON boxes. Any server with an OpenAI-compatible API works. The command below serves Qwen2.5-VL with [vLLM](https://docs.vllm.ai/en/latest/serving/online_serving/openai_compatible_server/), whose documentation connects clients at `http://localhost:8000/v1`: ```bash vllm serve Qwen/Qwen2.5-VL-7B-Instruct ``` With vLLM on port 8000, start Potato on another port: ```bash potato start config.yaml -p 9000 ``` Vision-language models answer in their own coordinate convention whatever the prompt asks for, and two parameters tell Potato how to read the numbers. `coordinate_format` is `fraction` (0 to 1 of the image), `per_mille` (0 to 1,000), or `pixels` of the image as sent. `box_order` is `xyxy` for `[x0, y0, x1, y1]` or `yxyx` for `[y0, x0, y1, x1]`. Qwen2.5-VL answers in pixels, so the config above sets `coordinate_format: pixels`. Under `auto`, Potato guesses the format once per reply from the size of the numbers. Values up to 1.05 read as fractions, values past the sent image's size read as per-mille, and anything else reads as pixels. A per-mille box that fits inside a large image therefore reads as pixels, so name the format whenever you know the model's convention. Gemma and Gemini answer per-mille boxes with y first, `[ymin, xmin, ymax, xmax]`, even when the prompt asks for fractions. With both parameters on `auto`, Potato reads the boxes of a model whose name contains `gemma` or `gemini` as per-mille with y first. On a Gemma 4 12B test project of red and blue squares, the Potato team measured mAP50 at 0.0 with the boxes read x first and 1.0 with the boxes read y first. [Ollama](https://docs.ollama.com/api/openai-compatibility) serves Gemma 3 under a name containing `gemma`, so `auto` applies: ```bash ollama run gemma3:12b ``` Replace the `openai-vlm-boxes` block in the config with the block below: ```yaml model_training: trainer_params: openai-vlm-boxes: url: http://localhost:11434/v1 model: gemma3:12b json_mode: object ``` A Gemma or Gemini fine-tune served under a name without either word gets the x-first reading under `auto`. Set `coordinate_format: per_mille` and `box_order: yxyx` for such a model. `json_mode: object` asks Ollama for JSON with `response_format: json_object`. The default, `schema`, sends a JSON schema with the labels as an enum. vLLM and OpenAI honor the schema. A run in which no reply parses as JSON fails, and the error suggests `json_mode: object` or a plainer prompt (`prompt`, where `{labels}` becomes the scheme's labels). A vision-language model gives no score with its boxes, so the boxes carry no confidence. `writeback.min_confidence` passes every `openai-vlm-boxes` prediction, and model review cannot order the predictions from least to most confident. The [`openai-vlm-boxes` reference](https://github.com/davidjurgens/potato/blob/master/docs/ai-intelligence/model_training.md#openai-vlm-boxes) lists `api_key_env` for a hosted API and the token and timeout settings. ## Step 3: Start a run and read the score Open `http://localhost:9000/admin/training`, choose `openai-vlm-boxes` and the `ppe` scheme, and click **Train now**. The same run starts from the API: ```bash curl -X POST -H "X-API-Key: $KEY" -H "Content-Type: application/json" \ http://localhost:9000/admin/training/api/runs \ -d '{"trainer": "openai-vlm-boxes", "schemas": ["ppe"]}' ``` The fit step lists the server's models through `/models` and logs which model the run asks and how the run reads the model's boxes. The prediction pass then sends one image per request. When the run finishes, the held-out mAP50 appears in the Runs table, and the run page lists `val_map_50`, `test_map_50`, the coverage figures, and a "Not used" warning that counts replies that were not JSON, boxes under labels the scheme lacks, and boxes off the image. When the score is near 0 while coverage is high and the warning is empty, check `coordinate_format` and `box_order` first. ## Roboflow models `roboflow` posts each image to `/`, where `model_id` is the project and version as `project/version`. The key goes in `ROBOFLOW_API_KEY` by default (`api_key_env`), and the run refuses to start without the key: ```bash export ROBOFLOW_API_KEY=... # in the shell that starts Potato potato start config.yaml -p 9000 ``` Potato's default `url` is `https://detect.roboflow.com`, and the config above names the default. A run for instance segmentation sets `url: https://outline.roboflow.com`. `url` is a parameter, so a run can also name `https://serverless.roboflow.com`, the host Roboflow's [object detection reference](https://docs.roboflow.com/deployment/legacy/legacy-serverless/object-detection) gives for the same `project/version` route. The Potato team has not tested that host with this bridge. A self-hosted [Roboflow Inference server](https://docs.roboflow.com/deployment/self-hosted/inference-server) listens on port 9001 by default (`url: http://localhost:9001`). Roboflow's API takes the key in the query string. Potato does not write the request URL to the run log or to an error message. Potato sends `confidence` and `overlap` (defaults 0.4 and 0.3) to the API as percentages, and Roboflow's reference documents both on a 0 to 100 scale. `shapes: auto` writes an outline where the model returns an outline and the scheme has a polygon tool, and a box otherwise. The health check posts an 8 × 8 white image, and Roboflow counts the request against the account's usage. ## CVAT serverless functions `cvat-nuclio` uses a detector already deployed for CVAT as a [Nuclio function](https://docs.cvat.ai/docs/administration/community/advanced/installation_automatic_annotation/), such as the YOLOv7 or Faster R-CNN functions in `cvat/serverless`. Potato posts each image to the function's own URL as `{"image": , "threshold": t}`, without going through CVAT. `nuctl get functions` lists the deployed functions with their ports, and the command below is the form CVAT's documentation gives: ```bash nuctl get functions --platform local ``` The function answers with shapes in pixels. A `rectangle` becomes a box, and a `polygon` becomes a polygon, or the polygon's bounding box when the scheme has no polygon tool. `mask` shapes are left out and counted. The function drops shapes scoring below `threshold` (default 0.5). The scheme has no person label, so `label_map: {person: null}` in the config drops a COCO detector's people. ## Hugging Face models with `hf-inference` `hf-inference` calls a model deployed on Hugging Face, either a dedicated [Inference Endpoint](https://huggingface.co/docs/inference-endpoints/index) by its `https://` URL (`endpoint_url`) or a model id on the serverless [HF Inference](https://huggingface.co/docs/inference-providers/providers/hf-inference) API (`model`). Set `endpoint_url` or `model`, and not both. The task follows the scheme's type: - **Text classification** for `radio`, `select`, and `multiselect`. A single-choice scheme takes the model's top label. When that label is not one of the scheme's labels after `label_map`, the item gets no prediction rather than the runner-up. A `multiselect` scheme takes every label scoring at least `threshold` (default 0.5). - **Token classification** for `span`, with word pieces grouped into entities by `aggregation_strategy` (`simple`, `first`, `average`, or `max`) and character offsets taken from the model. - **Object detection** for `image_annotation`. Boxes scoring below `threshold` are dropped. The config below asks [dslim/bert-base-NER](https://huggingface.co/dslim/bert-base-NER) for spans. The model card describes a BERT model fine-tuned for named entities on English CoNLL-2003. HF Inference serves a subset of the Hub's models, so check that the model's page lists HF Inference before relying on the serverless API. ```yaml annotation_task_name: "Named entities in news sentences" task_dir: . output_annotation_dir: annotation_output admin_api_key: "replace-with-a-long-random-string" data_files: - data/sentences.jsonl item_properties: id_key: id text_key: text annotation_schemes: - annotation_type: span name: entities description: "Highlight every person, organization, and place." labels: [person, organization, location] model_training: enabled: true writeback: schemas: [entities] include_unlabeled: true trainer_params: hf-inference: model: dslim/bert-base-NER token_env: HF_TOKEN aggregation_strategy: simple label_map: {PER: person, ORG: organization, LOC: location, MISC: null} ``` The model's `PER`, `ORG`, and `LOC` entities map to the scheme's labels, and `MISC` is dropped because the scheme has no such label. When an entity's full name matches nothing, Potato drops a `B-` or `I-` prefix and tries again. A sentiment classifier that answers `LABEL_0` and `LABEL_1` needs `label_map: {LABEL_0: negative, LABEL_1: positive}` in the same way. The health check is one real request, a short sentence or an 8 × 8 image, and `wait_for_model: true` (the default) waits while a serverless model loads instead of failing with a 503. ## A Label Studio ML backend `label-studio-ml` uses a running [Label Studio ML backend](https://github.com/HumanSignal/label-studio-ml-backend). A backend started from the repository's examples with `docker-compose up` serves a model over HTTP at `http://localhost:9090`. The run checks the backend's `GET /health`, sends the items to `POST /predict` in batches of 16 (`batch_size`), and reads the returned regions with the same converter that `potato import -f label_studio` uses. Label Studio's UTF-16 span offsets become code points, and percent coordinates become the image widget's. ```yaml annotation_task_name: "Review sentiment" task_dir: . output_annotation_dir: annotation_output admin_api_key: "replace-with-a-long-random-string" data_files: - data/reviews.jsonl item_properties: id_key: id text_key: text annotation_schemes: - annotation_type: radio name: sentiment description: "Is this review positive or negative?" labels: [positive, negative] model_training: enabled: true writeback: schemas: [sentiment] include_unlabeled: true min_confidence: 0.6 trainer_params: label-studio-ml: url: http://localhost:9090 basic_auth_user_env: LS_ML_USER basic_auth_pass_env: LS_ML_PASS ``` A stock backend's only authentication is HTTP Basic, set by `BASIC_AUTH_USER` and `BASIC_AUTH_PASS` on the backend. `basic_auth_user_env` and `basic_auth_pass_env` name the variables on the Potato server that hold the same two values, and the two parameters go together. `api_key_env` sends `Authorization: Token ` instead, for a backend behind a proxy that expects a token. A stock backend answers that header with a 500. Potato generates the labeling config sent with each request from the scheme (`Choices`, `Labels`, `RectangleLabels`, or `PolygonLabels`), and `label_config` replaces the generated XML with your own. An image backend needs one more setting. Without `media_url_prefix`, Potato sends each image as an absolute path on the Potato machine, and a backend built on the Label Studio SDK reads `data.image` as a URL and fails on a bare path. For a backend on the same machine, add the prefix below and start the backend with `LOCAL_FILES_DOCUMENT_ROOT` set to the task directory: ```yaml model_training: trainer_params: label-studio-ml: url: http://localhost:9090 media_url_prefix: /data/local-files/?d= ``` For a backend on another machine, serve the task directory over HTTP and give that server's URL as the prefix. An image outside the task directory has no path relative to the directory, so Potato does not send the image, and the run log names how many were left out. Potato sends the backend prediction requests and no annotation events, so the backend's own training in its `/webhook` handler does not start from Potato. `potato import -f label_studio` moves a whole Label Studio project into Potato with the project's labeling config, annotators' answers, and stored predictions ([Importing Projects](/docs/tools/importing-projects#label-studio)). [Migrating from Label Studio to Potato](/blog/label-studio-migration) covers the tags the importer leaves out. ## Review the pre-labels A bridge's predictions pass through the same write-back checks as a trained model's. Write-back skips train-split items and schemes missing from `writeback.schemas`, skips item and scheme pairs a reviewer rejected, and holds back predictions under `writeback.min_confidence`. For boxes and spans, a prediction's confidence is the confidence of the least confident shape or span, so the floor holds back an item's whole prediction when any one box or span on the item scores lower. Roboflow's `confidence` and CVAT's `threshold` act per box instead, at the service, and `hf-inference`'s `threshold` acts per box or per `multiselect` label. The [model-in-the-loop tutorial](/docs/model-training/model-in-the-loop) covers the review queue, accept and reject verdicts, and retraining. The queue and the verdicts work the same way for a bridge. One prediction is kept per item and scheme, so a later run of another bridge or of a trained model replaces the earlier run's pre-labels. To withdraw a run's pre-labels, send `DELETE /admin/training/api/runs//predictions`. The [inference bridges reference](https://github.com/davidjurgens/potato/blob/master/docs/ai-intelligence/model_training.md#inference-bridges) lists every parameter of the four bridges, and the [`label-studio-ml` reference](https://github.com/davidjurgens/potato/blob/master/docs/ai-intelligence/model_training.md#label-studio-ml) lists the Label Studio parameters. ## Further reading - [Model Training](/docs/model-training/overview) covers the loop and the Training page. - [Trainers](/docs/model-training/trainers) lists all 38 trainers by task, for a model trained on the study's labels to compare with a bridge's score. - [Configuration](/docs/model-training/configuration) documents splits, write-back, and the routes. - [Image Annotation](/docs/annotation-types/image-annotation) and [Span Annotation](/docs/annotation-types/span-annotation) document the schemes the bridges fill. - [Visual AI Support](/docs/features/visual-ai-support) shows a vision model's suggestions to an annotator on request, without a held-out score.