Image Annotation

Configure image annotation in Potato with bounding boxes, polygons, masks, and landmarks. Display images with instance_display and export to COCO or YOLO.

Potato supports image annotation for classification, object detection, segmentation, and keypoint tasks.

Showing an Image

Displaying an image and collecting an annotation are two separate settings. instance_display says what the annotator sees, and annotation_schemes says what they answer. Keeping them apart means any content type can sit alongside any question type:

yaml

instance_display:
  fields:
    - key: image_url          # Field name in your data file
      type: image
      label: "Image to classify"
      display_options:
        max_width: 600
        zoomable: true

The image_annotation scheme is for drawing on an image, not for showing one. A classification task needs only the display field above plus a radio or multiselect scheme.

Display Options

Option	Description
`max_width`	Maximum rendered width in pixels
`max_height`	Maximum rendered height in pixels
`zoomable`	Enable click-to-zoom
`object_fit`	How the image fills its box
`alt_text`	Alternative text for screen readers

An option name Potato does not recognize is a configuration error rather than a silently ignored key, and the message suggests the nearest real one.

Data Format

Reference images in your data file:

json

{"id": "img_001", "image_url": "https://example.com/photo_001.jpg"}
{"id": "img_002", "image_url": "/static/images/photo_002.png"}

data_files is a list of paths, and item_properties maps your field names onto the ones Potato reads:

yaml

data_files:
  - "data/image_tasks.json"
 
item_properties:
  id_key: id
  text_key: image_url   # The field holding the image URL

Image Classification

Classify entire images:

yaml

annotation_schemes:
  - annotation_type: radio
    name: category
    description: "What is shown in this image?"
    labels:
      - Cat
      - Dog
      - Bird
      - Other
 
  - annotation_type: multiselect
    name: attributes
    description: "Select all that apply"
    labels:
      - Indoor
      - Outdoor
      - Multiple animals
      - Human present

Multi-Label Classification

yaml

annotation_schemes:
  - annotation_type: multiselect
    name: objects
    description: "What objects are visible?"
    labels:
      - Person
      - Car
      - Building
      - Tree
      - Animal
      - Furniture
      - Food
      - Electronic device

Image Quality Assessment

yaml

annotation_schemes:
  - annotation_type: likert
    name: quality
    description: "Overall image quality"
    size: 5
    min_label: "Very poor"
    max_label: "Excellent"
 
  - annotation_type: multiselect
    name: issues
    description: "Select any quality issues"
    labels:
      - Blurry
      - Overexposed
      - Underexposed
      - Noisy
      - Low resolution
      - Watermark visible

Drawing on Images

The image_annotation scheme requires both tools and labels. Giving each label a color and a key_value is worth the extra lines, because annotators assign labels by keystroke far faster than by clicking:

yaml

annotation_schemes:
  - annotation_type: image_annotation
    name: object_detection
    description: "Draw boxes around objects"
    tools:
      - bbox
      - polygon
    labels:
      - name: person
        color: "#FF6B6B"
        key_value: "1"
      - name: vehicle
        color: "#4ECDC4"
        key_value: "2"
      - name: animal
        color: "#45B7D1"
        key_value: "3"
    min_annotations: 1
    zoom_enabled: true
    pan_enabled: true

color must be hex (#rgb, #rrggbb, or #rrggbbaa). Named CSS colors and rgb() are not read: shapes would look right, but masks are painted pixel by pixel and fall back to red.

Available Tools

Tool	Description
`bbox`	Rectangular bounding boxes
`polygon`	Multi-point closed polygons
`polyline`	Multi-point open paths, for lane markings, vessels, cracks, coastlines
`ellipse`	Ellipses and circles, for cells, nuclei, wheels, faces
`keypoint_set`	Ordered skeletons, for human or animal pose and facial landmarks
`cuboid_2d`	3D boxes projected into the image, KITTI style
`sam`	Magic wand: click an object and a model segments it
`freeform`	Free-hand drawing
`landmark`	Single point markers
`brush`	Segmentation brush for pixel-level masks
`eraser`	Remove mask regions
`fill`	Flood fill for enclosed regions

A polygon is closed and encloses an area; a polyline is open and has length but no interior. The distinction matters on export, where a polyline is written with area: 0 and never treated as a region.

Polygon Annotation

For non-rectangular regions:

yaml

annotation_schemes:
  - annotation_type: image_annotation
    name: regions
    description: "Outline regions of interest"
    tools:
      - polygon
    labels:
      - name: Building
      - name: Road
      - name: Vegetation
      - name: Water

Segmentation Masks

For pixel-level work, enable the painting tools. Masks are stored as run-length encoding rather than per-pixel arrays, which is what keeps a full-image mask small:

yaml

annotation_schemes:
  - annotation_type: image_annotation
    name: segmentation
    description: "Paint regions using the brush tool"
    tools:
      - brush
      - eraser
      - fill
    labels:
      - name: road
        color: "#45B7D1"
      - name: sidewalk
        color: "#96CEB4"
    brush_size: 20
    mask_opacity: 0.5
    mask_mode: semantic

mask_mode: semantic merges every stroke of a class into one region. Use instance when two objects of the same class must stay apart.

Output Format

Annotations are stored as a flat JSON array, with every shape and mask in the same list. Coordinates are normalized to the range 0 to 1 against the image, so annotations stay correct when the image is served at a different size:

json

[
  {
    "type": "bbox",
    "label": "person",
    "color": "#FF6B6B",
    "coordinates": {"x": 0.125, "y": 0.104, "width": 0.3125, "height": 0.625}
  },
  {
    "type": "polygon",
    "label": "vehicle",
    "color": "#4ECDC4",
    "coordinates": [
      {"x": 0.02, "y": 0.04}, {"x": 0.20, "y": 0.04},
      {"x": 0.20, "y": 0.21}, {"x": 0.02, "y": 0.21}
    ]
  },
  {
    "type": "mask",
    "label": "road",
    "color": "#45B7D1",
    "rle": {"counts": [12, 40, 8], "size": [480, 640]}
  }
]

The geometry key varies by type:

Type	Geometry key	Shape
`bbox`	`coordinates`	`{x, y, width, height}`, normalized
`polygon`	`coordinates`	list of `{x, y}`, normalized
`landmark`	`coordinates`	`{x, y}`, normalized
`freeform`	`coordinates`	`{path, left, top, scaleX, scaleY}`
`mask`	`rle`	`{counts: [ints], size: [height, width]}`

rle.size is [height, width], in that order. To convert to and from these shapes in Python, use normalize_annotation_object() and to_client_object() from potato.export.cv_utils rather than reading the fields directly.

Where a Shape Came From

An accepted model detection and a hand-drawn box are otherwise identical once stored, so every shape records its own origin. Without it a project that pre-labels with a model cannot report its own acceptance rate from its own data:

json

{
  "type": "bbox",
  "label": "person",
  "coordinates": {"x": 0.125, "y": 0.104, "width": 0.3125, "height": 0.625},
  "source": "ai",
  "ai_model": "yolov8n",
  "confidence": 0.82,
  "edited": true
}

Key	Meaning
`source`	`human` (drawn in the browser), `ai` (a model proposed it and the annotator accepted it), `import` (it arrived with the data)
`ai_model`	Which model, when the endpoint reports one
`confidence`	The detector's own score for this shape
`edited`	The annotator has since moved, resized or relabeled a shape that was not theirs
`carried_over`	Copied forward from the previous image rather than made on this one

edited is what separates accepting a box as-is from accepting it and then correcting it. Only the first is agreement with the model.

Reviewing Existing Annotations

To review shapes that already exist, import them rather than adding them to a data field. potato import turns a COCO file into a runnable project, stamping each shape source: import:

bash

potato import --input instances_val2017.json \
    --image-dir /data/val2017 \
    --image-url-prefix /files \
    --output-dir my-project/ \
    --schema-name object_detection
 
potato start my-project/config.yaml -p 8000

Annotators then correct the imported shapes in place, and the export keeps the human and model split across the round trip.

Image Comparison

Compare two images with the pairwise scheme. items_key names the data field holding the pair:

yaml

annotation_schemes:
  - annotation_type: pairwise
    name: preference
    description: "Which image looks better?"
    mode: binary
    items_key: images
    show_labels: true
    labels:
      - "Original"
      - "Edited"
    allow_tie: true
    tie_label: "About the same"

Image Captioning

yaml

annotation_schemes:
  - annotation_type: text
    name: caption
    description: "Write a caption for this image"
    rows: 4
    placeholder: "Describe what you see..."

Caption Quality Review

Show the image and the generated caption side by side, then rate the caption:

yaml

instance_display:
  fields:
    - key: image_url
      type: image
      label: "Image"
    - key: generated_caption
      type: text
      label: "Generated caption"
 
annotation_schemes:
  - annotation_type: likert
    name: accuracy
    description: "How accurate is this caption?"
    size: 5
    min_label: "Very inaccurate"
    max_label: "Very accurate"
 
  - annotation_type: likert
    name: fluency
    description: "How natural is the language?"
    size: 5
    min_label: "Very awkward"
    max_label: "Very natural"
 
  - annotation_type: text
    name: improved_caption
    description: "Suggest a better caption (optional)"
    rows: 4
    label_requirement:
      required: false

Content Moderation

yaml

annotation_schemes:
  - annotation_type: radio
    name: safe_for_work
    description: "Is this image safe for work?"
    labels:
      - Safe
      - Questionable
      - Not Safe
 
  - annotation_type: multiselect
    name: violation_types
    description: "Select all violations (if any)"
    labels:
      - Violence
      - Adult content
      - Hate symbols
      - Graphic content
      - Spam/advertisement

Interchange Formats

Potato reads COCO JSON, including every segmentation encoding and crowd regions, and writes COCO, YOLO, Pascal VOC, and per-label mask PNGs:

bash

python -m potato.export -c my-project/config.yaml -f coco -o ./export/

YOLO and Pascal VOC carry bounding boxes only, so other shapes are reduced to their enclosing box.

Full Example: Object Detection Review

yaml

annotation_task_name: "Object Detection Verification"
task_dir: "."
 
data_files:
  - "data/detections.json"
 
item_properties:
  id_key: id
  text_key: image_url
 
instance_display:
  fields:
    - key: image_url
      type: image
      label: "Image"
      display_options:
        max_width: 1000
        zoomable: true
 
annotation_schemes:
  # Correct the boxes
  - annotation_type: image_annotation
    name: objects
    description: "Verify and correct object boxes"
    tools:
      - bbox
    labels:
      - name: Person
        key_value: "1"
      - name: Vehicle
        key_value: "2"
      - name: Animal
        key_value: "3"
      - name: Object
        key_value: "4"
    zoom_enabled: true
    pan_enabled: true
 
  # Overall assessment
  - annotation_type: radio
    name: prediction_quality
    description: "How accurate were the predictions?"
    labels:
      - All correct
      - Minor corrections needed
      - Major corrections needed
      - Mostly incorrect
 
  - annotation_type: number
    name: missed_objects
    description: "How many objects were missed?"
    min: 0
    max: 50
 
  - annotation_type: text
    name: notes
    description: "Any issues or comments?"
    rows: 4
    label_requirement:
      required: false
 
output_annotation_dir: "output/"
export_annotation_format: jsonl
 
user_config:
  allow_all_users: true

Start the server:

bash

potato start config.yaml -p 8000

Performance Tips

Resize before annotating - Large source images slow down loading without improving the annotation
Use JPEG for photos - Smaller files, faster loading
Use PNG for graphics - Better quality for diagrams and screenshots
Cap the rendered size - Set max_width so the browser is not scaling a huge image on every instance
Pre-process consistently - Normalize sizes and formats across the dataset