Polygon Annotation for Segmentation Tasks

Polygon annotation enables pixel-precise object boundaries for semantic and instance segmentation. This guide covers configuration, annotation techniques, and best practices for complex segmentation tasks.

When to Use Polygons vs Bounding Boxes

Use Polygons When	Use Bounding Boxes When
Precise boundaries needed	Rough localization sufficient
Irregular object shapes	Objects are roughly rectangular
Training segmentation models	Training detection models
Overlapping objects	Non-overlapping objects
Higher accuracy required	Speed is priority

Basic Polygon Setup

annotation_task_name: "Image Segmentation"
 
data_files:
  - "data/images.json"
 
annotation_schemes:
  - annotation_type: image_annotation
    tools: [polygon]
    name: segments
    description: "Draw polygons around each object"
    labels:
      - name: road
        color: "#808080"
      - name: sidewalk
        color: "#F5F5DC"
      - name: building
        color: "#8B4513"
      - name: vegetation
        color: "#228B22"
      - name: sky
        color: "#87CEEB"
      - name: vehicle
        color: "#FF6B6B"
      - name: person
        color: "#4ECDC4"

Drawing Polygons

Basic Workflow

Select a label from the toolbar
Click to place vertices around the object
Double-click or press Enter to close the polygon
Adjust vertices by dragging
Add points by clicking on edges

Keyboard Shortcuts

Potato provides built-in keyboard shortcuts for polygon editing:

Enter: Close the current polygon
Escape: Cancel drawing or deselect
Delete: Remove selected vertex or polygon
+/-: Zoom in/out
Space + drag: Pan the image

Handling Complex Scenarios

Overlapping Objects

For tasks with overlapping objects, annotators can draw multiple polygons that overlap. Each polygon is stored separately in the output, allowing for instance-level annotation where objects may occlude each other.

Tips for Complex Boundaries

When annotating objects with complex boundaries:

Use more vertices for curved edges to capture the shape accurately
Zoom in when tracing detailed boundaries
Work systematically around the object perimeter
Use the edit mode to refine vertex positions after initial drawing

Semantic Segmentation Setup

For full-image segmentation:

annotation_task_name: "Semantic Segmentation"
 
data_files:
  - "data/images.json"
 
annotation_schemes:
  - annotation_type: image_annotation
    tools: [polygon]
    name: semantic_segments
    description: "Segment the entire image"
    labels:
      - name: background
        color: "#000000"
      - name: road
        color: "#808080"
      - name: sidewalk
        color: "#F5F5DC"
      - name: building
        color: "#8B4513"
      - name: vegetation
        color: "#228B22"
      - name: sky
        color: "#87CEEB"
      - name: person
        color: "#4ECDC4"
      - name: vehicle
        color: "#FF6B6B"

Instance Segmentation Setup

For individual object instances:

annotation_task_name: "Instance Segmentation"
 
data_files:
  - "data/images.json"
 
annotation_schemes:
  - annotation_type: image_annotation
    tools: [polygon]
    name: instances
    description: "Draw polygons around each object instance"
    labels:
      - name: person
        color: "#4ECDC4"
      - name: car
        color: "#FF6B6B"
      - name: dog
        color: "#FFEAA7"

Each polygon drawn will be stored as a separate annotation, allowing you to track individual instances of each object class.

Complete Configuration Example

annotation_task_name: "Cityscapes-style Segmentation"
 
data_files:
  - "data/urban_scenes.json"
 
annotation_schemes:
  - annotation_type: image_annotation
    tools: [polygon]
    name: segments
    description: "Segment all objects following Cityscapes protocol"
    labels:
      # Flat
      - name: road
        color: "#804080"
      - name: sidewalk
        color: "#F423E8"
      # Construction
      - name: building
        color: "#464646"
      - name: wall
        color: "#666666"
      - name: fence
        color: "#BE9999"
      # Nature
      - name: vegetation
        color: "#6B8E23"
      - name: terrain
        color: "#98FB98"
      # Sky
      - name: sky
        color: "#4682B4"
      # Human
      - name: person
        color: "#DC143C"
      - name: rider
        color: "#FF0000"
      # Vehicle
      - name: car
        color: "#00008B"
      - name: truck
        color: "#000046"
      - name: bus
        color: "#003C64"
      - name: motorcycle
        color: "#0000E6"
      - name: bicycle
        color: "#770B20"

Output Format

Polygon annotations are saved in the annotation output with coordinates for each vertex:

{
  "id": "img001",
  "annotations": {
    "segments": [
      {
        "label": "road",
        "points": [[0, 600], [100, 580], [200, 590], [0, 600]]
      },
      {
        "label": "car",
        "points": [[450, 400], [520, 400], [520, 480], [450, 480]]
      }
    ]
  }
}

Tips for Quality Polygons

Zoom in: Work at 2x or higher for detailed boundaries
Use keyboard shortcuts: Much faster than clicking tools
Start with major features: Rough outline first, then refine
Vertex economy: Use minimum points needed for accuracy
Regular breaks: Detailed polygon work causes fatigue

Next Steps

Combine with bounding boxes for hybrid workflows
Set up quality control for segmentation
Learn about video segmentation for temporal data

See /docs/features/image-annotation for complete polygon documentation.