Using Visual AI to Speed Up Image and Video Annotation

Use a detection model for boxes and a vision-language model for judgments, then have annotators accept or reject what the model proposes instead of drawing every region by hand.

Potato 2.1 added visual AI support, which puts model suggestions inside the image and video annotation interface. A detector such as YOLO finds objects and draws candidate boxes for review. A vision-language model such as GPT-4o, Claude, or a local Ollama model classifies an image or a region and explains the classification. Nothing a model proposes becomes an annotation until an annotator accepts it.

This guide covers the four visual endpoints, the assistance modes each one supports, and how to pair a visual endpoint with a text endpoint in the same project. The visual AI source documentation has the full API reference.

What you need

Visual AI needs Potato 2.1.0 or later:

pip install --upgrade potato-annotation

The endpoint you pick decides the rest of the setup:

• YOLO runs locally: pip install ultralytics opencv-python
• Ollama runs locally: install from ollama.ai and pull a vision model
• OpenAI needs an API key with access to GPT-4o
• Anthropic needs an API key with access to Claude vision models

YOLO for object detection

Reach for YOLO when the task is bounding boxes on common objects and the images should not leave your machine. It detects people, cars, animals, and furniture in milliseconds per image, and it returns coordinates rather than prose, which is what a box-drawing task needs.

The scheme-level ai_support block decides which buttons an annotator sees, and the project-level block decides which model answers them. Keeping them separate is what lets one project run AI on the detection scheme and leave the rest of its schemes untouched.

annotation_task_name: "Object Detection with YOLO"
task_dir: "."
 
data_files:
  - data/images.json
 
item_properties:
  id_key: id
  text_key: image_url
 
instance_display:
  fields:
    - key: image_url
      type: image
      display_options:
        max_width: 800
        zoomable: true
 
annotation_schemes:
  - annotation_type: image_annotation
    name: objects
    description: "Draw a box around every object and label it"
    source_field: "image_url"
    tools:
      - bbox
    labels:
      - name: "person"
        color: "#FF6B6B"
      - name: "car"
        color: "#4ECDC4"
      - name: "dog"
        color: "#45B7D1"
      - name: "cat"
        color: "#96CEB4"
 
    ai_support:
      enabled: true
      features:
        detection: true
        pre_annotate: true
        hint: true
 
ai_support:
  enabled: true
  endpoint_type: "yolo"
  ai_config:
    model: "yolov8m.pt"
    confidence_threshold: 0.5
 
output_annotation_dir: "annotation_output/"
user_config:
  allow_all_users: true

The data file is one JSON object per line, with the image path or URL in the field named by text_key:

{"id": "img_001", "image_url": "images/street_scene_1.jpg"}
{"id": "img_002", "image_url": "images/park_photo.jpg"}
{"id": "img_003", "image_url": "https://example.com/images/office.jpg"}

Picking a model size

Model size trades latency against recall, and either end of that trade costs the annotator time. A model that misses objects turns pre-annotation back into a drawing task, and a model that takes seconds per image leaves the annotator waiting between items. The middle sizes are the usual choice because they keep both costs small rather than because they win on either one.

For objects outside YOLO's built-in classes, YOLO-World does open-vocabulary detection:

ai_config:
  model: "yolo-world"
  confidence_threshold: 0.3

Setting the confidence threshold

The threshold decides which kind of work you hand your annotators. A low threshold produces more boxes, so they spend their time rejecting false positives. A high threshold produces fewer, so they spend it drawing what the model missed. Start at 0.5, watch which correction dominates the first batch, and move it:

ai_config:
  confidence_threshold: 0.3  # more detections, more false positives

ai_config:
  confidence_threshold: 0.7  # fewer detections, higher precision

Ollama vision models for local scene understanding

Ollama runs vision-language models on your own hardware. Unlike YOLO, these models read the whole scene and write an explanation, which is what a classification task needs and a detector cannot give you. Nothing leaves the machine, so the same setup covers datasets that cannot be sent to a hosted API.

curl -fsSL https://ollama.ai/install.sh | sh
ollama pull llava

The project below classifies scene type with a local model. No scheme-level ai_support block appears, because the classification features are project-level here rather than attached to a drawing tool:

annotation_task_name: "Image Classification with Ollama Vision"
task_dir: "."
 
data_files:
  - data/images.json
 
item_properties:
  id_key: id
  text_key: image_url
 
instance_display:
  fields:
    - key: image_url
      type: image
      display_options:
        max_width: 600
        zoomable: true
 
annotation_schemes:
  - annotation_type: radio
    name: scene_type
    description: "What type of scene is shown?"
    labels:
      - indoor
      - outdoor_urban
      - outdoor_nature
      - aerial
      - underwater
 
ai_support:
  enabled: true
  endpoint_type: "ollama_vision"
  ai_config:
    model: "llava:latest"
    base_url: "http://localhost:11434"
    max_tokens: 500
    temperature: 0.1
 
output_annotation_dir: "annotation_output/"
user_config:
  allow_all_users: true

A low temperature matters more here than in a writing task. The same image shown twice should get the same suggested label, and a model sampling at a high temperature will sometimes disagree with itself, which annotators read as noise in the tool rather than uncertainty in the model.

Potato supports LLaVA in its 7B, 13B, and 34B sizes, LLaVA-LLaMA3, BakLLaVA, Llama 3.2 Vision at 11B and 90B, Qwen2.5-VL, and Moondream. The smaller models answer fast enough to keep an annotator moving; the larger ones are worth the wait when the label depends on fine detail.

OpenAI vision

GPT-4o handles images through the same ai_support block, with a detail setting that controls how much of the image is sent:

ai_support:
  enabled: true
  endpoint_type: "openai_vision"
  ai_config:
    api_key: "${OPENAI_API_KEY}"
    model: "gpt-4o"
    max_tokens: 1000
    detail: "auto"

export OPENAI_API_KEY="sk-..."

low sends a downscaled image, which is cheaper and fast enough for scene-level classification. high sends full resolution, which is what finding small objects needs. auto lets the API choose. The setting is a cost decision on a per-image basis, so it is worth fixing deliberately before a large batch rather than after the bill.

Anthropic vision

Claude's vision models read image context and write out the reasoning behind a classification, which makes them a good fit for tasks where the annotator needs an explanation rather than a box:

ai_support:
  enabled: true
  endpoint_type: "anthropic_vision"
  ai_config:
    api_key: "${ANTHROPIC_API_KEY}"
    model: "claude-sonnet-4-20250514"
    max_tokens: 1024

export ANTHROPIC_API_KEY="sk-ant-..."

Assistance modes

Each mode adds a button to the annotator's toolbar, and each one changes what the model is asked for. Turn on only the ones the task needs, since every extra button is a decision the annotator has to make on every item.

Detection finds objects matching your configured labels and shows them as dashed overlays. It is the mode to use when annotators work image by image and want a starting point per image:

ai_support:
  enabled: true
  features:
    detection: true

Pre-annotation runs the detector over everything in one pass, so annotators open each item with suggestions already on it. This is the mode that pays off on large datasets, and it is also the one that most needs a calibrated threshold, since nobody reviews the settings again once the pass has run:

ai_support:
  enabled: true
  features:
    pre_annotate: true

Classification labels a selected region or the whole image and returns a suggested label with a confidence score. Hint returns guidance without the answer, which is the mode to use during annotator training, where handing over the label would teach nothing:

ai_support:
  enabled: true
  features:
    classification: true
    hint: true

Accepting and rejecting suggestions

Suggestions arrive as dashed overlays and stay suggestions until an annotator acts on them. Double-clicking an overlay converts it into a real annotation, right-clicking dismisses it, and the toolbar carries an "Accept All" and a "Clear" for the items where every suggestion is right or every one is wrong.

The accept step is what keeps the annotation file honest. The export contains what a human accepted rather than what a model proposed, so a badly tuned threshold costs annotation time instead of quietly writing wrong labels into the dataset. That also makes the accept and reject counts worth watching during a pilot, since a run where annotators reject most suggestions is telling you about the threshold before it tells you about the model.

Video: scene segmentation

Video schemes take their own set of features, covering scene boundaries, frame classification, keyframes, and tracking. max_frames caps how many frames the model samples from the clip, which is the main lever on both accuracy and wait time:

annotation_schemes:
  - annotation_type: video_annotation
    name: scenes
    description: "Segment this video into scenes"
    mode: segment
    labels:
      - name: "intro"
        color: "#4ECDC4"
      - name: "main_content"
        color: "#FF6B6B"
      - name: "outro"
        color: "#45B7D1"
 
    ai_support:
      enabled: true
      features:
        scene_detection: true
        pre_annotate: true
        hint: true
 
ai_support:
  enabled: true
  endpoint_type: "ollama_vision"
  ai_config:
    model: "llava:latest"
    max_frames: 10

One endpoint for text, another for images

A project that mixes text schemes and image schemes usually wants different models for each, because the model that draws the best boxes is not the one that writes the best hint. visual_endpoint_type and visual_ai_config set the second endpoint. Add this block to a config that already has an ai_support section:

ai_support:
  enabled: true
 
  endpoint_type: "ollama"
  ai_config:
    model: "llama3.2"
    include:
      all: true
 
  visual_endpoint_type: "yolo"
  visual_ai_config:
    model: "yolov8m.pt"
    confidence_threshold: 0.5

The same pair works with a hosted vision model alongside a local text model:

ai_support:
  enabled: true
  endpoint_type: "ollama"
  visual_endpoint_type: "openai_vision"
  ai_config:
    model: "llama3.2"
  visual_ai_config:
    api_key: "${OPENAI_API_KEY}"
    model: "gpt-4o"

One caveat applies to the pair in 2.9.1. The AI cache reads both keys and the second endpoint works, but potato validate --strict reports them as unrecognized, so a config using them fails a strict validation run. Projects that gate deploys on --strict will want to keep the visual and text endpoints in separate projects until that is reconciled upstream.

A complete example: product photos

This project asks annotators to box products and defects, judge whether the photo can be published, and tag what is wrong with it. YOLO proposes the boxes, and the two judgment schemes stay human:

annotation_task_name: "Product Photo Annotation"
task_dir: "."
 
data_files:
  - data/product_photos.json
 
item_properties:
  id_key: sku
  text_key: photo_url
 
instance_display:
  layout:
    direction: horizontal
    gap: 24px
  fields:
    - key: photo_url
      type: image
      label: "Product Photo"
      display_options:
        max_width: 600
        zoomable: true
    - key: product_description
      type: text
      label: "Product Details"
 
annotation_schemes:
  - annotation_type: image_annotation
    name: product_regions
    description: "Draw boxes around products and defects"
    source_field: "photo_url"
    tools:
      - bbox
    labels:
      - name: "product"
        color: "#4ECDC4"
      - name: "defect"
        color: "#FF6B6B"
      - name: "label"
        color: "#45B7D1"
      - name: "packaging"
        color: "#96CEB4"
 
    ai_support:
      enabled: true
      features:
        detection: true
        pre_annotate: true
 
  - annotation_type: radio
    name: photo_quality
    description: "Is this photo suitable for the product listing?"
    labels:
      - Approved
      - Needs editing
      - Reshoot required
 
  - annotation_type: multiselect
    name: quality_issues
    description: "Select any issues present"
    labels:
      - Blurry
      - Poor lighting
      - Wrong angle
      - Background clutter
      - Color inaccurate
 
ai_support:
  enabled: true
  endpoint_type: "yolo"
  ai_config:
    model: "yolov8m.pt"
    confidence_threshold: 0.5
 
output_annotation_dir: "annotation_output/"
export_annotation_format: "jsonl"
user_config:
  allow_all_users: true

To add text hints on the two judgment schemes, give this config the visual_endpoint_type pair shown above, with the text model as endpoint_type and yolo as the visual endpoint. The image scheme keeps its boxes from YOLO, and the radio and multiselect schemes get their hints from the text model.

Sample data (data/product_photos.json):

{"sku": "SKU-001", "photo_url": "images/products/laptop_front.jpg", "product_description": "15-inch laptop, silver finish"}
{"sku": "SKU-002", "photo_url": "images/products/headphones_side.jpg", "product_description": "Over-ear wireless headphones, black"}
{"sku": "SKU-003", "photo_url": "images/products/backpack_full.jpg", "product_description": "40L hiking backpack, navy blue"}

Getting the setup right

• Run pre-annotation before annotators start on a large dataset. Reviewing and correcting is faster per item than drawing from scratch, and the gain scales with dataset size while the setup cost does not.
• Match the endpoint to the question. A detector answers "where is it", a vision-language model answers "what is this and why", and asking either one for the other's job produces suggestions annotators learn to ignore.
• Tune the threshold on a pilot batch. Start at 0.5 and move it based on which correction your annotators are making more often.
• Use hint mode for training. It gives a new annotator direction without handing over the label, so their judgment is still what gets recorded.
• Cache the results. Disk caching keeps a model from being paid for twice on the same image.

Troubleshooting

"No visual AI endpoint configured"

Check that ai_support.enabled is true and that endpoint_type names a vision endpoint: yolo, ollama_vision, openai_vision, or anthropic_vision. A text endpoint produces this message on an image scheme.

YOLO is not detecting your objects

YOLO's built-in classes cover 80 common object categories, so labels outside that set will not match no matter how the threshold is set. Switch to YOLO-World for an open vocabulary. If the labels do match YOLO's classes and the detections are merely sparse, lower confidence_threshold instead.

Ollama returns errors

Check that the server is running and that the model you named is both installed and vision-capable:

curl http://localhost:11434/api/tags
ollama list

Cloud API calls are slow

Turn on the disk cache so the same image is not analyzed twice across sessions. The path resolves relative to task_dir and has to stay inside it:

ai_support:
  cache_config:
    disk_cache:
      enabled: true
      path: "ai_cache/visual_cache.json"

Next steps

• Read the visual AI support documentation for the full API reference
• Set up instance display to show images alongside other content
• Explore text-based AI support for hints and keyword highlighting