Polygons let you trace object boundaries pixel by pixel, which is what semantic and instance segmentation need. This guide walks through the config, the drawing techniques, and what to do when the shapes get messy.

Polygons vs bounding boxes

Use Polygons When	Use Bounding Boxes When
Precise boundaries needed	Rough localization sufficient
Irregular object shapes	Objects are roughly rectangular
Training segmentation models	Training detection models
Overlapping objects	Non-overlapping objects
Higher accuracy required	Speed is priority

Basic polygon setup

yaml

annotation_task_name: "Image Segmentation"
 
data_files:
  - "data/images.json"
 
annotation_schemes:
  - annotation_type: image_annotation
    tools: [polygon]
    name: segments
    description: "Draw polygons around each object"
    labels:
      - name: road
        color: "#808080"
      - name: sidewalk
        color: "#F5F5DC"
      - name: building
        color: "#8B4513"
      - name: vegetation
        color: "#228B22"
      - name: sky
        color: "#87CEEB"
      - name: vehicle
        color: "#FF6B6B"
      - name: person
        color: "#4ECDC4"

Drawing polygons

The basic workflow

Pick a label from the toolbar
Click to drop vertices around the object
Double-click or press Enter to close the polygon
Drag vertices to adjust them
Click on an edge to add a point

Keyboard shortcuts

Potato ships with shortcuts for polygon editing:

Enter: close the current polygon
Escape: cancel drawing or deselect
Delete: remove the selected vertex or polygon
+/-: zoom in and out
Space + drag: pan the image

When things get complicated

Overlapping objects

When objects overlap, annotators just draw multiple polygons that overlap. Potato stores each one separately, so you get instance-level annotation even where objects occlude each other.

Tricky boundaries

For objects with awkward, curvy boundaries:

Add more vertices on curved edges so the shape comes out accurate
Zoom in before you trace the fiddly parts
Work your way around the perimeter in order rather than jumping around
Switch to edit mode afterward to nudge vertices into place

Semantic segmentation setup

For full-image segmentation:

yaml

annotation_task_name: "Semantic Segmentation"
 
data_files:
  - "data/images.json"
 
annotation_schemes:
  - annotation_type: image_annotation
    tools: [polygon]
    name: semantic_segments
    description: "Segment the entire image"
    labels:
      - name: background
        color: "#000000"
      - name: road
        color: "#808080"
      - name: sidewalk
        color: "#F5F5DC"
      - name: building
        color: "#8B4513"
      - name: vegetation
        color: "#228B22"
      - name: sky
        color: "#87CEEB"
      - name: person
        color: "#4ECDC4"
      - name: vehicle
        color: "#FF6B6B"

Instance segmentation setup

For individual object instances:

yaml

annotation_task_name: "Instance Segmentation"
 
data_files:
  - "data/images.json"
 
annotation_schemes:
  - annotation_type: image_annotation
    tools: [polygon]
    name: instances
    description: "Draw polygons around each object instance"
    labels:
      - name: person
        color: "#4ECDC4"
      - name: car
        color: "#FF6B6B"
      - name: dog
        color: "#FFEAA7"

Each polygon is stored as its own annotation, so you can track individual instances within each object class.

Complete configuration example

yaml

annotation_task_name: "Cityscapes-style Segmentation"
 
data_files:
  - "data/urban_scenes.json"
 
annotation_schemes:
  - annotation_type: image_annotation
    tools: [polygon]
    name: segments
    description: "Segment all objects following Cityscapes protocol"
    labels:
      # Flat
      - name: road
        color: "#804080"
      - name: sidewalk
        color: "#F423E8"
      # Construction
      - name: building
        color: "#464646"
      - name: wall
        color: "#666666"
      - name: fence
        color: "#BE9999"
      # Nature
      - name: vegetation
        color: "#6B8E23"
      - name: terrain
        color: "#98FB98"
      # Sky
      - name: sky
        color: "#4682B4"
      # Human
      - name: person
        color: "#DC143C"
      - name: rider
        color: "#FF0000"
      # Vehicle
      - name: car
        color: "#00008B"
      - name: truck
        color: "#000046"
      - name: bus
        color: "#003C64"
      - name: motorcycle
        color: "#0000E6"
      - name: bicycle
        color: "#770B20"

Output format

Polygon annotations land in the output with coordinates for every vertex:

json

{
  "id": "img001",
  "annotations": {
    "segments": [
      {
        "label": "road",
        "points": [[0, 600], [100, 580], [200, 590], [0, 600]]
      },
      {
        "label": "car",
        "points": [[450, 400], [520, 400], [520, 480], [450, 480]]
      }
    ]
  }
}

Tips for clean polygons

Zoom to 2x or more when you are tracing detailed boundaries.
Learn the keyboard shortcuts. They are much faster than reaching for the toolbar.
Lay down a rough outline first, then go back and refine it.
Use the fewest vertices that still capture the shape. More is not better.
Take breaks. Detailed polygon work wears you down fast.

Where to go next

Combine this with bounding boxes for hybrid workflows
Set up quality control for segmentation
Learn about video segmentation for temporal data

For everything the image annotation type can do (polygon, bbox, and the other drawing tools), see the source documentation.

See /docs/features/image-annotation for complete polygon documentation.