Image Annotation

Potato's image annotation system supports multiple annotation methods for computer vision tasks including object detection, segmentation, and keypoint annotation.

Supported Tools

A scheme lists the tools it needs in tools, and only those appear in the toolbar. Enabling fewer tools is the cheapest way to keep an annotator from producing the wrong geometry for the task.

The sam tool needs a downloaded model. Its button renders either way and reports what is missing, rather than failing silently.

Basic Configuration

Both tools and labels are required, and Potato refuses to load a scheme that omits either. A detection task needs nothing else, since zoom and pan are on by default:

annotation_schemes:
  - name: "objects"
    description: "Draw boxes around all objects"
    annotation_type: "image_annotation"
    tools:
      - bbox
    labels:
      - person
      - car
      - bicycle

Configuration Options

Label Configuration

Labels can be simple strings or detailed objects. Potato assigns colors automatically, so the detailed form is worth writing when the colors need to mean something to the annotator or when the labels need keys:

Simple labels take the palette colors in list order:

labels:
  - person
  - car
  - tree

The detailed form names a color and a key for each one:

labels:
  - name: "person"
    color: "#FF6B6B"
    key_value: "1"
  - name: "car"
    color: "#4ECDC4"
    key_value: "2"
  - name: "bicycle"
    color: "#45B7D1"
    key_value: "3"

Digits are a safe choice for key_value, because the default tool shortcuts are letters and a label that reuses a tool key fires both actions on one press.

Examples

Object Detection (Bounding Boxes)

Setting min_annotations: 1 blocks submission of an image with no box, which catches the annotator who advanced without looking rather than the image that genuinely has no objects. Use it only on data where every image has at least one instance:

annotation_schemes:
  - name: "detection"
    description: "Draw bounding boxes around all vehicles"
    annotation_type: "image_annotation"
    tools:
      - bbox
    labels:
      - name: "car"
        color: "#3B82F6"
        key_value: "1"
      - name: "truck"
        color: "#10B981"
        key_value: "2"
      - name: "motorcycle"
        color: "#F59E0B"
        key_value: "3"
    min_annotations: 1
    zoom_enabled: true

Instance Segmentation (Polygons)

Polygon work runs at high magnification, so zoom and pan carry the accuracy of the boundary:

annotation_schemes:
  - name: "segmentation"
    description: "Draw polygons around each object"
    annotation_type: "image_annotation"
    tools:
      - polygon
    labels:
      - name: "building"
        color: "#8B5CF6"
      - name: "road"
        color: "#64748B"
      - name: "vegetation"
        color: "#22C55E"
    zoom_enabled: true
    pan_enabled: true

Keypoint Annotation (Landmarks)

Each landmark is its own label, which is what keeps the points identifiable in the output. Give the paired points matching colors so a left/right swap is visible on the canvas:

annotation_schemes:
  - name: "facial_landmarks"
    description: "Mark facial keypoints"
    annotation_type: "image_annotation"
    tools:
      - landmark
    labels:
      - name: "left_eye"
        color: "#3B82F6"
      - name: "right_eye"
        color: "#3B82F6"
      - name: "nose"
        color: "#10B981"
      - name: "left_mouth"
        color: "#F59E0B"
      - name: "right_mouth"
        color: "#F59E0B"

Multiple Tools

A scheme can enable several tools when one image needs more than one kind of geometry:

annotation_schemes:
  - name: "mixed_annotation"
    description: "Use boxes for objects and polygons for regions"
    annotation_type: "image_annotation"
    tools:
      - bbox
      - polygon
    labels:
      - name: "object"
        color: "#3B82F6"
      - name: "region"
        color: "#10B981"

Keyboard Shortcuts

Tool keys come from the profile named in keybinding_profile. The default v7 profile follows the conventions of V7 Darwin and CVAT, so an annotator arriving from either keeps their muscle memory; legacy preserves the bindings Potato used before. Only the enabled tools get a key.

The remaining shortcuts are the same in both profiles:

Each label with a key_value adds its own key.

Data Format

Input Data

Your data file should include image paths or URLs:

[
  {
    "id": "img_001",
    "image_url": "https://example.com/images/photo1.jpg"
  },
  {
    "id": "img_002",
    "image_url": "/data/images/photo2.png"
  }
]

Point text_key at the field holding the image path, since that is the field Potato hands to the annotation widget:

item_properties:
  id_key: id
  text_key: image_url

The canvas requests each image with CORS so it can read the image's pixels. Many public hosts, cocodataset.org among them, send no Access-Control-Allow-Origin header and refuse that request, and from Potato 2.11 the canvas then loads the image again without CORS. On an image loaded that way, boxes, polygons, ellipses, points, freeform strokes and the brush all work. The tools that read the image's pixels cannot run on it, so flood fill says it cannot read the image and fills nothing, and SAM and text-prompt segmentation say the host sends no CORS headers. To use those tools, copy the images into the task's media_directory and reference them as /media/<file>.

Output Format

Drawn regions travel in their own image_annotations key on the export record, grouped by scheme name, rather than inside labels. The browser writes one shape for every tool: a type, the label, its color, and the geometry under coordinates, normalized to the image rather than in pixels. Masks are the exception and carry rle instead:

{
  "instance_id": "img_001",
  "user_id": "annotator1",
  "labels": {},
  "spans": {},
  "links": {},
  "image_annotations": {
    "objects": [
      {
        "type": "bbox",
        "label": "car",
        "color": "#FF6B6B",
        "coordinates": {"x": 0.12, "y": 0.30, "width": 0.25, "height": 0.20}
      },
      {
        "type": "polygon",
        "label": "building",
        "color": "#4ECDC4",
        "coordinates": [{"x": 0.06, "y": 0.10}, {"x": 0.19, "y": 0.10}, {"x": 0.19, "y": 0.40}]
      }
    ]
  }
}

Exporters do not read that shape directly. normalize_annotation_object turns each object into absolute pixels first — {type, label, color, bbox, points, rle, area, instance, iscrowd, warnings}, with bbox always present even for a polygon — and the vision formats build on that. The two shapes are told apart structurally, by whether a coordinates key is present, never by magnitude: a 1×1 pixel box at the origin is [0, 0, 1, 1] in both coordinate spaces.

The jsonl export does not carry this key: it writes instance_id, user_id, labels, spans and links only. To get geometry out, export to one of the vision formats — coco, yolo, pascal_voc, mask_png, cvat, labelme — which read image_annotations directly.

Supported Image Formats

• JPEG / JPG
• PNG
• GIF
• WebP

Segmentation Masks

New in v2.2.0

The fill, eraser, and brush tools produce pixel-level masks for tasks where a polygon boundary is too coarse. Enable all three together, because painting without an eraser leaves an annotator no way to correct an overrun:

annotation_schemes:
  - name: "mask_painting"
    description: "Paint segmentation masks"
    annotation_type: "image_annotation"
    tools:
      - fill
      - eraser
      - brush
    labels:
      - name: "foreground"
        color: "#3B82F6"
      - name: "background"
        color: "#6B7280"
    brush_size: 10
    eraser_size: 10
    mask_opacity: 0.5

Under the default v7 profile the three tools are b for brush, e for eraser, and f for fill, and [ and ] change the brush size without leaving the canvas. Lower mask_opacity when annotators need to see the image through a painted region.

PDF/Document Bounding Boxes

New in v2.2.0

Draw bounding boxes on PDF pages and document images. When combined with the pdf display type in instance_display, annotators can draw boxes on individual pages of PDF documents:

instance_display:
  fields:
    - key: pdf_url
      type: pdf
 
annotation_schemes:
  - name: "document_regions"
    description: "Draw boxes around figures and tables"
    annotation_type: "image_annotation"
    tools:
      - bbox
    labels:
      - name: "figure"
        color: "#3B82F6"
      - name: "table"
        color: "#10B981"
      - name: "equation"
        color: "#F59E0B"

Best Practices

1. Use appropriate tools - Bounding boxes for quick detection, polygons for precise boundaries, fill/brush for segmentation
2. Define clear labels - Use distinct, non-overlapping categories
3. Set annotation limits - Use min_annotations to ensure completeness
4. Enable zoom - Detailed work on high-resolution images depends on it
5. Give labels keyboard shortcuts - An annotator who never leaves the canvas to pick a label works faster
6. Provide label colors - Makes it easier to distinguish annotations visually