# Polygon Annotation for Segmentation Tasks

Source: https://www.potatoannotator.com/blog/polygon-annotation-guide

Polygons let you trace object boundaries pixel by pixel, which is what semantic and instance segmentation need. This guide walks through the config, the drawing techniques, and what to do when the shapes get messy.

## Polygons vs bounding boxes

| Use Polygons When | Use Bounding Boxes When |
|------------------|------------------------|
| Precise boundaries needed | Rough localization sufficient |
| Irregular object shapes | Objects are roughly rectangular |
| Training segmentation models | Training detection models |
| Overlapping objects | Non-overlapping objects |
| Higher accuracy required | Speed is priority |

## Basic polygon setup

```yaml
annotation_task_name: "Image Segmentation"

data_files:
  - "data/images.json"

annotation_schemes:
  - annotation_type: image_annotation
    tools: [polygon]
    name: segments
    description: "Draw polygons around each object"
    labels:
      - name: road
        color: "#808080"
      - name: sidewalk
        color: "#F5F5DC"
      - name: building
        color: "#8B4513"
      - name: vegetation
        color: "#228B22"
      - name: sky
        color: "#87CEEB"
      - name: vehicle
        color: "#FF6B6B"
      - name: person
        color: "#4ECDC4"
```

## Drawing polygons

### The basic workflow

1. Pick a label from the toolbar
2. Click to drop vertices around the object
3. Double-click or press Enter to close the polygon
4. Drag vertices to adjust them
5. Click on an edge to add a point

### Keyboard shortcuts

Potato ships with shortcuts for polygon editing:

- Enter: close the current polygon
- Escape: cancel drawing or deselect
- Delete: remove the selected vertex or polygon
- +/-: zoom in and out
- Space + drag: pan the image

## When things get complicated

### Overlapping objects

When objects overlap, annotators just draw multiple polygons that overlap. Potato stores each one separately, so you get instance-level annotation even where objects occlude each other.

### Tricky boundaries

For objects with awkward, curvy boundaries:

1. Add more vertices on curved edges so the shape comes out accurate
2. Zoom in before you trace the fiddly parts
3. Work your way around the perimeter in order rather than jumping around
4. Switch to edit mode afterward to nudge vertices into place

## Semantic segmentation setup

For full-image segmentation:

```yaml
annotation_task_name: "Semantic Segmentation"

data_files:
  - "data/images.json"

annotation_schemes:
  - annotation_type: image_annotation
    tools: [polygon]
    name: semantic_segments
    description: "Segment the entire image"
    labels:
      - name: background
        color: "#000000"
      - name: road
        color: "#808080"
      - name: sidewalk
        color: "#F5F5DC"
      - name: building
        color: "#8B4513"
      - name: vegetation
        color: "#228B22"
      - name: sky
        color: "#87CEEB"
      - name: person
        color: "#4ECDC4"
      - name: vehicle
        color: "#FF6B6B"
```

## Instance segmentation setup

For individual object instances:

```yaml
annotation_task_name: "Instance Segmentation"

data_files:
  - "data/images.json"

annotation_schemes:
  - annotation_type: image_annotation
    tools: [polygon]
    name: instances
    description: "Draw polygons around each object instance"
    labels:
      - name: person
        color: "#4ECDC4"
      - name: car
        color: "#FF6B6B"
      - name: dog
        color: "#FFEAA7"
```

Each polygon is stored as its own annotation, so you can track individual instances within each object class.

## Complete configuration example

```yaml
annotation_task_name: "Cityscapes-style Segmentation"

data_files:
  - "data/urban_scenes.json"

annotation_schemes:
  - annotation_type: image_annotation
    tools: [polygon]
    name: segments
    description: "Segment all objects following Cityscapes protocol"
    labels:
      # Flat
      - name: road
        color: "#804080"
      - name: sidewalk
        color: "#F423E8"
      # Construction
      - name: building
        color: "#464646"
      - name: wall
        color: "#666666"
      - name: fence
        color: "#BE9999"
      # Nature
      - name: vegetation
        color: "#6B8E23"
      - name: terrain
        color: "#98FB98"
      # Sky
      - name: sky
        color: "#4682B4"
      # Human
      - name: person
        color: "#DC143C"
      - name: rider
        color: "#FF0000"
      # Vehicle
      - name: car
        color: "#00008B"
      - name: truck
        color: "#000046"
      - name: bus
        color: "#003C64"
      - name: motorcycle
        color: "#0000E6"
      - name: bicycle
        color: "#770B20"
```

## Output format

Polygon annotations land in the output with coordinates for every vertex:

```json
{
  "id": "img001",
  "annotations": {
    "segments": [
      {
        "label": "road",
        "points": [[0, 600], [100, 580], [200, 590], [0, 600]]
      },
      {
        "label": "car",
        "points": [[450, 400], [520, 400], [520, 480], [450, 480]]
      }
    ]
  }
}
```

## Tips for clean polygons

1. Zoom to 2x or more when you are tracing detailed boundaries.
2. Learn the keyboard shortcuts. They are much faster than reaching for the toolbar.
3. Lay down a rough outline first, then go back and refine it.
4. Use the fewest vertices that still capture the shape. More is not better.
5. Take breaks. Detailed polygon work wears you down fast.

## Where to go next

- Combine this with [bounding boxes](/blog/bounding-box-annotation) for hybrid workflows
- Set up [quality control](/blog/quality-control-strategies) for segmentation
- Learn about [video segmentation](/blog/video-frame-annotation) for temporal data

For everything the image annotation type can do (polygon, bbox, and the other drawing tools), see the [source documentation](https://github.com/davidjurgens/potato/blob/master/docs/annotation-types/multimedia/image_annotation.md).

---

*See [/docs/features/image-annotation](/docs/features/image-annotation) for complete polygon documentation.*