Polygon Annotation for Segmentation Tasks

This tutorial builds an image segmentation task in Potato where annotators trace object boundaries with polygons, and it covers the config, the keys annotators press, and the shape of the exported coordinates.

Polygons trace a boundary vertex by vertex, which is what semantic and instance segmentation need and what a bounding box cannot give you. The cost is annotator time, so the first decision is whether your task needs boundaries at all.

Choosing between polygons, boxes and masks

Three tools produce region annotations in Potato, and they differ in what they can represent rather than only in speed.

Use a box when rough localization is enough, which covers most detection training. A polygon earns its extra annotator time when the model consumes boundaries and the objects are solid. Once an object has holes or wisps, the brush mask is the honest tool, since forcing a polygon around a bicycle wheel produces a shape nobody can check. Potato also has an open-path polyline tool for linear features such as lane markings and vessels, which exports with zero area rather than an invented interior.

The task configuration

The polygon tool is enabled per scheme through tools, and the labels are the classes annotators can assign to a shape. item_properties.text_key names the field in your data file that holds the image URL, which is how Potato knows what to display:

annotation_task_name: "Urban Scene Segmentation"
task_description: "Trace every object in the street scene."
 
data_files:
  - data/images.json
 
item_properties:
  id_key: id
  text_key: image_url
 
annotation_schemes:
  - annotation_type: image_annotation
    name: segments
    description: "Draw a polygon around each object."
    tools: [polygon]
    labels:
      - name: road
        color: "#808080"
        key_value: "1"
      - name: sidewalk
        color: "#F5F5DC"
        key_value: "2"
      - name: building
        color: "#8B4513"
        key_value: "3"
      - name: vegetation
        color: "#228B22"
        key_value: "4"
      - name: sky
        color: "#87CEEB"
        key_value: "5"
      - name: vehicle
        color: "#FF6B6B"
        key_value: "6"
      - name: person
        color: "#4ECDC4"
        key_value: "7"
 
output_annotation_dir: annotation_output

The key_value entries matter more than they look. A seven-class task where the annotator reaches for the toolbar on every shape runs at a fraction of the speed of one where classes are on the number keys, and the cost falls entirely on the annotator. Colors must be hex, because masks are painted pixel by pixel and a named CSS color falls back to red.

Each item in the data file gives an id and the image:

{"id": "img_001", "image_url": "https://example.com/street_001.jpg"}
{"id": "img_002", "image_url": "/static/images/street_002.jpg"}

Start the server against the config:

potato start config.yaml -p 8000

Drawing a polygon and the keys that do it

Drawing a polygon is a click per vertex around the object, then a double-click to close it. A polygon needs at least three points to enclose anything, and Potato drops the duplicate vertices a double-click leaves behind rather than storing them.

The keys come from a keybinding profile. The default, v7, matches the conventions V7 Darwin and CVAT use, so annotators arriving from either tool keep their muscle memory:

Set keybinding_profile: legacy on the scheme if you have a study already collecting data, because v7 rebinds b, f and l, and a mid-study rebind costs accuracy. On a tablet, one finger draws, two fingers pan, and a two-finger pinch zooms.

To reshape a finished polygon, switch to select mode and drag its vertices. Annotators working on curved boundaries do better laying down a rough outline first and refining it than trying to place every vertex correctly on the first pass, and they should zoom in before tracing the fiddly parts rather than after.

Overlapping objects and instances

When two objects overlap, annotators draw two polygons that overlap. Potato stores each shape as its own entry, so a scene with four cars yields four polygons whether or not they occlude each other, and instance identity survives into the export. Nothing in the config needs to change between a semantic task and an instance task when you are drawing polygons; the difference is in how you consume the output, since a semantic model wants the shapes merged per class and an instance model wants them kept apart.

For brush masks the distinction is a setting rather than a consumer decision, because mask_mode: semantic merges every stroke of a class into one region and mask_mode: instance keeps objects separate.

The output format

Annotations come back as a flat JSON array, with every shape in the same list. Coordinates are normalized to the range 0 to 1 against the image, so the annotation stays correct if the image is later served at a different size:

[
  {
    "type": "polygon",
    "label": "vehicle",
    "color": "#FF6B6B",
    "coordinates": [
      {"x": 0.02, "y": 0.04}, {"x": 0.20, "y": 0.04},
      {"x": 0.20, "y": 0.21}, {"x": 0.02, "y": 0.21}
    ]
  },
  {
    "type": "bbox",
    "label": "person",
    "color": "#4ECDC4",
    "coordinates": {"x": 0.125, "y": 0.104, "width": 0.3125, "height": 0.625}
  }
]

Each shape type names its geometry differently. A polygon carries coordinates as a list of points, a box carries one object with width and height, and a brush mask carries rle with row-major run lengths and a size of [height, width]. To convert these shapes in Python, use normalize_annotation_object() and to_client_object() from potato.export.cv_utils rather than reading the fields directly, so your code keeps working if the storage changes.

Agreement on polygon boundaries

Mean IoU between annotators is the number most tools report and it answers the wrong question. A corpus where every image holds one large centered object scores around 0.95 whoever draws it, including someone who never looked at the image, because IoU has no chance correction in it.

Potato reports chance-corrected measures for geometry schemes at /admin/iaa, and splits the question three ways: detection_alpha for whether annotators found the same objects, classification_alpha for whether they agree what each one is, and sigma with ks for whether they drew it in the same place. The split is diagnostic. High detection with low localization means the drawing guidelines are too loose, and the reverse means the definition of an object is unclear. Measuring agreement on bounding boxes covers the method for all shape types.

The other thing worth checking is vertex count. A four-vertex polygon around a curved object is a box in disguise, and vertices_median in the drawing telemetry surfaces it.

Where to go next

• How to annotate image segmentation masks for boundary rules and click-to-segment
• Bounding box annotation for the cheaper tool and hybrid workflows
• Quality control strategies for the checks that run alongside a segmentation task
• Video frame annotation for temporal data
• Image Annotation for the full option list, and the source documentation for every tool the schema supports