Bounding Box, Polygon, Keypoint, and Cuboid Annotation
Pick the annotation shape your vision model needs, compare what each shape costs per object, and measure agreement between annotators on every shape in Potato.
Choose the annotation shape from what your model predicts. An object detector trains on bounding boxes, an instance segmentation model on polygons or masks, a pose model on keypoint sets, and a 3D detector on cuboids drawn in a point cloud. Start with boxes when the guidelines or the model are still changing, because a box is the cheapest region shape to draw and to check.
An annotation shape is the geometry an annotator draws to mark where an object sits in an image or a scan. A bounding box is the axis-aligned rectangle that encloses an object, and the box is stored as four numbers. Polygons are closed chains of vertices that trace an object's outline, while a polyline is an open polygonal chain with length and no area. Keypoint sets are ordered lists of named points joined by skeleton edges, and keypoint sets train models for pose estimation. Cuboids are boxes in three dimensions. An annotator draws a cuboid either as a projection on a camera image or as an oriented box in a point cloud.
Potato is a free, open-source annotation tool that runs on your own server, and the guide uses Potato's tools as the worked example. Every 2D shape below is a tool in Potato's image_annotation scheme, and 3D cuboids belong to the spatial_annotation scheme. How to Annotate Image Segmentation Masks covers masks in depth, How to Annotate 3D Point Clouds covers lidar work, and Image Annotation Tools Compared compares Potato with other tools.
Which shape each kind of model needs
A model learns only what the labels encode. The table below pairs each kind of model output with the shape to draw, the Potato tool, and the similarity Potato uses to decide whether two annotators drew the same object:
| The model predicts | Draw | Potato tool | How two annotators' shapes are compared |
|---|---|---|---|
| A class and a rectangle per object (object detection) | Bounding box | bbox | Box IoU |
| A class and an outline per object (instance segmentation, e.g., Mask R-CNN) | Polygon, or a mask for holes and thin parts | polygon, or brush with mask_mode: instance | Polygon IoU, or mask IoU |
| A class per pixel (semantic segmentation) | Mask | brush, fill, sam | Mask IoU |
| Round objects such as cells and nuclei | Ellipse | ellipse | IoU of a 36-vertex polygon |
| Lines, e.g., lane markings and cracks | Polyline | polyline | Boundary F1 within a distance tolerance |
| Joint positions for pose | Keypoint set with a skeleton | keypoint_set | Object Keypoint Similarity (OKS) |
| One location per object, for counting | Landmark | landmark | Distance, scored 0 at 5% of the image apart |
| A 3D box from one camera image | Projected cuboid | cuboid_2d | IoU of the front faces |
| A 3D box in lidar or depth data | 3D cuboid | cuboid_3d in spatial_annotation | Exact rotated 3D IoU |
A shape richer than the model needs costs annotator time and adds nothing at training time. A shape poorer than the model needs cannot be repaired later. A box has no outline, while the enclosing box of any outline can be computed at export, and Potato's COCO export writes a bbox for every polygon and mask.
Drawing cost per object
Annotation cost per object rises with the number of points an annotator places and checks. Published timings exist for boxes and polygons, each measured on a large crowdsourced dataset with that project's own interface:
- Bounding boxes. Drawing one high-quality box for the ILSVRC dataset took 35 seconds, as Papadopoulos et al. (2017) report. The same paper reports 7 seconds per box with extreme clicking. In extreme clicking, the annotator clicks the object's topmost, bottommost, leftmost, and rightmost points. Potato's
bboxtool uses the conventional drag of a rectangle. - Polygons. Outlining object instances for COCO took over 22 worker hours per 1,000 segmentations, or about 79 seconds per object (Lin et al., 2014).
- Keypoints and cuboids. The Potato team found no published per-object time for keypoint sets or 3D cuboids that could be checked against its source. Measure your own in a pilot.
Both timings come from different datasets and interfaces. Treat the two numbers as orders of magnitude rather than as a ratio for your data. Time a short pilot of each candidate shape on your own images. Potato's drawing telemetry logs each committed shape with a timestamp and its vertex count, and the annotation cost calculator turns a time per object into a budget.
Agreement measures for each shape
Agreement on shapes starts with matching. Potato pairs one annotator's objects with another annotator's objects on the same image, and the pairing keeps as many pairs above the threshold as the two sets allow. Two shapes count as the same object when the shape's similarity reaches 0.5. Potato takes the 0.5 threshold from the COCO detection convention. Boxes compare by IoU, polygons and ellipses by the IoU of their areas, and projected cuboids by the IoU of their front faces. Polylines have no interior for IoU to measure, and Potato scores polylines by boundary F1. Boundary F1 is the harmonic mean of the shares of each trace's points that lie near the other trace.
Keypoint sets compare by Object Keypoint Similarity (OKS). COCO defined OKS for keypoint evaluation, and OKS scales each joint's distance by the object's size and by a tolerance set per joint. COCO sets the constant at 0.025 for an eye and 0.107 for a hip, because annotators place an eye more consistently than a hip. Potato skips a joint that either annotator left unlabeled, because a missing joint and a misplaced joint are different disagreements.
The report at /admin/iaa then splits disagreement into three questions:
- Detection. Did the annotators find the same objects? Potato reports nominal Krippendorff's α over presence.
- Classification. Did matched objects get the same label? Potato reports nominal Krippendorff's α over the labels of matched objects.
- Localization. Are matched objects in the same place? σ compares the mean distance between matched objects on one image with the mean distance between objects on different images, and a Kolmogorov–Smirnov statistic compares the two distributions.
σ is 1 for perfect agreement and 0 when annotators agree no more than annotators of unrelated images would. Raw mean IoU reads high on a corpus of large centered objects whoever draws the boxes, and σ corrects for the easy task. In the benchmark of Braylan et al. (2022), Krippendorff's α computed with IoU distance ranked annotator quality worse than α computed with plain L2 distance. Potato reports σ instead of α over IoU distance. We have found no other annotation platform reporting chance-corrected agreement over spatial labels. How to Measure Inter-Annotator Agreement on Bounding Boxes explains the method, and Agreement over Geometry documents the report.
The default distance inside σ is GIoU over the box that encloses each matched object. For polygons, polylines, and keypoint sets, the localization σ therefore measures where the enclosing boxes sit. The shape's own similarity enters through matching and the detection score.
3D cuboids get a separate report. Potato computes the volumetric IoU of two cuboids exactly for any rotation, and two cuboids count as one object at a 3D IoU of 0.25. An unmatched cuboid counts as full disagreement, a matched cuboid with a different label earns half credit, and the report gives the mean over pairs of annotators. The 3D report has no chance baseline. For sequences, Potato scores each frame with boxes interpolated between keyframes and adds MOTA, IDF1, and identity switches between each pair of annotators' tracks. As a reference point, the KITTI 3D object benchmark requires a 3D overlap of 70% for cars and 50% for pedestrians and cyclists.
Starting with boxes and refining later
Boxes are the right first shape when the class list or the guidelines are still moving. A box is quick to redraw after a guideline change, and box agreement shows early whether annotators find the same objects at all. A low detection Krippendorff's α points at an unclear object definition. A more precise shape does not fix an unclear definition.
Three routes lead from boxes to finer shapes:
- Box to mask. With the
samtool, dragging a box asks a segmentation model for a mask inside the box. The model runs in the annotator's browser afterpotato download-models onnxruntimeandpotato download-models mobile_sam. - Outlines for a detector. Roboflow's post of July 19, 2023 reports training YOLOv8 from scratch on a 408-image fire hydrant dataset labeled both ways. The polygon labels gave the higher mAP50. The Potato team read the post on November 17, 2026. Roboflow's result covers one dataset and one model. Test the gain on your own data before paying for outlines.
- A shape per class. One
image_annotationscheme can enable several tools. A project can box most classes and outline only the classes whose boundaries the model needs.
Potato never matches shapes of different types. A class that one annotator boxes and another annotator outlines therefore scores as complete disagreement. Write the shape for each class into the annotation guidelines.
Each shape in Potato
The first config below is complete. Each later block replaces the annotation_schemes list of the first config.
Bounding boxes
Potato stores a box as {x, y, width, height} in coordinates normalized to the image. The box is axis-aligned. A four-vertex polygon records the angle of a rotated object such as a ship seen from above. Each line of the data file holds an id and an image_url:
annotation_task_name: "Street scene detection"
task_dir: "."
data_files:
- "data/images.jsonl"
item_properties:
id_key: id
text_key: image_url
output_annotation_dir: "annotation_output/"
annotation_schemes:
- annotation_type: image_annotation
name: objects
description: "Draw a tight box around every car, person, and bicycle."
tools: [bbox]
min_annotations: 1
labels:
- {name: car, color: "#FF6B6B", key_value: "1"}
- {name: person, color: "#4ECDC4", key_value: "2"}
- {name: bicycle, color: "#45B7D1", key_value: "3"}
agreement_metrics:
enabled: true
num_annotators_per_item: 2num_annotators_per_item: 2 sends each image to two annotators. Agreement needs at least two annotations of each image. min_annotations: 1 refuses an empty submission, and suits a corpus where every image holds at least one object. Export for a detector with the export CLI:
python -m potato.export --config config.yaml --format coco --output ./export/
python -m potato.export --config config.yaml --format yolo --output ./export/The first command writes a COCO annotations.json, and the second writes one YOLO label file per image with data.yaml and classes.txt. To draw boxes from a typed object name, see How to Label Objects by Typing Their Name.
Polygons and ellipses
An annotator draws a polygon with a click per vertex and a double-click to close. A polygon needs at least three vertices. Use a polygon when the model consumes boundaries and the objects are solid. An ellipse fits a round object with one drag, and Potato stores the ellipse as a center, two radii, and an angle. Exports write each ellipse as a 36-vertex polygon, and every format that reads polygons reads the ellipses as well.
annotation_schemes:
- annotation_type: image_annotation
name: nuclei
description: "Outline every nucleus. Use the ellipse for round nuclei and the polygon for irregular ones."
tools: [polygon, ellipse]
labels:
- {name: nucleus, color: "#E6194B", key_value: "1"}The BBBC038 nucleus outlines design and the SpaceNet building footprints design are complete polygon tasks.
Polylines
A polyline is an open path of two or more points, and an annotator finishes a polyline with a double-click. Lane markings, cracks, vessels, cables, and shorelines are polyline objects, because a lane marking or a crack has length and no interior. Closing a polyline into a polygon would claim an area the annotator never drew. COCO has no open-path type, and Potato's COCO export writes each polyline with an empty segmentation, area: 0, and the points under a non-standard polyline key.
annotation_schemes:
- annotation_type: image_annotation
name: lanes
description: "Trace each lane marking from the bottom of the image to where the marking ends."
tools: [polyline]
labels:
- {name: solid_line, color: "#F1C40F", key_value: "1"}
- {name: dashed_line, color: "#3498DB", key_value: "2"}Keypoint sets and skeletons
A keypoint set is one annotation that holds an ordered list of points. The skeletons block names each point and lists the edges between points by index, and the order gives each point its meaning. Annotators place the points in the declared order, and the set completes after the last name. A double-click finishes a set early when the remaining joints are out of frame.
annotation_schemes:
- annotation_type: image_annotation
name: pose
description: "Place each joint of every visible person in the skeleton's order."
tools: [keypoint_set]
labels: [person]
skeletons:
coco_person:
names: [nose, left_eye, right_eye, left_ear, right_ear,
left_shoulder, right_shoulder, left_elbow, right_elbow,
left_wrist, right_wrist, left_hip, right_hip,
left_knee, right_knee, left_ankle, right_ankle]
edges: [[0,1],[0,2],[1,3],[2,4],[5,6],[5,7],[7,9],[6,8],[8,10],
[5,11],[6,12],[11,12],[11,13],[13,15],[12,14],[14,16]]Potato's COCO export writes each set as a COCO keypoints array with visibility flags. Agreement uses OKS with COCO's per-joint constants. COCO set the constants for the 17-point person skeleton above, and a custom skeleton's joints take the constants of COCO's joints at the same index. Separate landmark points with one label per joint lose the grouping, and two people in one image become one pile of points.
Segmentation masks
A mask labels pixels and suits objects with holes or thin parts. Potato's brush, eraser, fill, and sam tools paint masks. mask_mode: semantic merges every stroke of a class into one region, and mask_mode: instance keeps each object separate.
annotation_schemes:
- annotation_type: image_annotation
name: instances
description: "Paint each animal as its own object."
tools: [brush, eraser, fill]
mask_mode: instance
labels: [cat, dog]How to Annotate Image Segmentation Masks covers boundary rules and click-to-segment, and Mask Consensus with STAPLE covers how to merge several annotators' masks.
Projected cuboids
The cuboid_2d tool draws a 3D box on a single camera image in the KITTI style. The annotator drags the visible front face, then clicks once to set the depth offset. Potato builds the back face by shifting the front face by the depth offset and stores both faces as quadrilaterals. The drawn box therefore has no perspective scaling and no depth in metres. Use cuboid_2d when only images exist and the model needs an object's visible extent and rough orientation. A 3D detector that predicts position in metres needs cuboids in a point cloud.
annotation_schemes:
- annotation_type: image_annotation
name: vehicles
description: "Drag the face of each vehicle nearest the camera, then click to set the depth."
tools: [cuboid_2d]
labels:
- {name: car, color: "#FF6B6B", key_value: "1"}
- {name: truck, color: "#4ECDC4", key_value: "2"}3D cuboids in point clouds
The spatial_annotation scheme with the cuboid_3d tool draws oriented boxes in a lidar or depth point cloud. Each box stores a center and a size in metres and a rotation, all in the sensor frame. With cuboid_3d on, the viewer adds three orthographic slab views, and a new box snaps its height to the points inside its footprint. A calibration file named in calibration_field turns on camera panels that project each box into the images for a 2D check. Each item names its point cloud in a point_cloud field and its calibration in a calibration field:
annotation_schemes:
- annotation_type: spatial_annotation
name: objects
description: "Put an oriented 3D box around every vehicle and pedestrian."
source_field: point_cloud
calibration_field: calibration
tools: [cuboid_3d]
labels:
- {name: car, color: "#FF6B6B", key_value: "1"}
- {name: pedestrian, color: "#FFD93D", key_value: "2"}python -m potato.export --config config.yaml --format kitti_3d --output ./export/ writes KITTI label files for the cuboids. The KITTI design and the nuScenes 3D cuboid design are complete tasks, and 3D Point Clouds and Cuboids lists every option.
Frequently asked questions
What is the difference between a bounding box and a polygon annotation?
A bounding box records an object's position and extent as one axis-aligned rectangle, and a polygon records the object's outline as a closed chain of vertices. A box trains object detectors, while a polygon trains instance segmentation models and also yields the enclosing box. Published crowdsourcing timings put a box at 35 seconds and a COCO polygon at about 79 seconds per object, with a different interface behind each figure. Potato compares two annotators' boxes by box IoU and two polygons by the IoU of their areas.
What is 3D cuboid annotation?
3D cuboid annotation draws an oriented box around each object in a lidar point cloud or other three-dimensional data. The same task is also called 3D bounding box annotation. Each cuboid records the object's center and size in metres and the object's rotation, and 3D object detectors train on the cuboids. Benchmarks score cuboids by 3D IoU, and KITTI requires a 70% overlap for cars. A projected cuboid drawn on a camera image costs less to draw and has no depth in metres.
What is polyline annotation?
Polyline annotation traces a linear object as an open chain of points with length and no enclosed area. Lane markings, road edges, cracks, and blood vessels are the usual polyline objects in images. Agreement between annotators on polylines uses boundary F1 over the points of the two traces, because IoU needs an interior that a polyline lacks.
Further reading
- Image Annotation reference, for every
image_annotationoption - Image Annotation feature page
- Geometry Primitives, for tool keys and the keybinding profiles
- How to Annotate Image Segmentation Masks
- How to Annotate 3D Point Clouds
- How to Measure Inter-Annotator Agreement on Bounding Boxes
- Agreement over Geometry
- Image Annotation Tools Compared
- Exporting Annotations for ML
References
- Braylan, A., Alonso, O., and Lease, M. (2022). Measuring Annotator Agreement Generally across Complex Structured, Multi-object, and Free-text Annotation Tasks. Proceedings of the ACM Web Conference 2022. https://doi.org/10.1145/3485447.3512242
- He, K., Gkioxari, G., Dollár, P., and Girshick, R. (2017). Mask R-CNN. 2017 IEEE International Conference on Computer Vision (ICCV). https://doi.org/10.1109/ICCV.2017.322
- Lin, T.-Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C. L., and Dollár, P. (2014). Microsoft COCO: Common Objects in Context. Computer Vision – ECCV 2014. https://doi.org/10.1007/978-3-319-10602-1_48
- Papadopoulos, D. P., Uijlings, J. R. R., Keller, F., and Ferrari, V. (2017). Extreme Clicking for Efficient Object Annotation. 2017 IEEE International Conference on Computer Vision (ICCV). https://doi.org/10.1109/ICCV.2017.528
- Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., and Savarese, S. (2019). Generalized Intersection over Union: A Metric and a Loss for Bounding Box Regression. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/CVPR.2019.00075
उदाहरण डिज़ाइन
हर डिज़ाइन एक चालू Potato कॉन्फ़िगरेशन है, जिसे आप डाउनलोड करके चला सकते हैं।
- BBBC038 2018 Data Science Bowl Outlines of Every NucleusDraw one polygon per nucleus on ten real images from the CC0 2018 Data Science Bowl set, with Cellpose fine-tuning that writes outlines back as pre-labels.
- SpaceNet Building Footprint Outlines on Satellite ScenesOutline building footprints as closed polygons on satellite-style scenes with the geo_annotation scheme. The ten scenes are illustrative synthetic images.
- KITTI - 3D Object Annotation on Lidar Point CloudsAutonomous-driving benchmark from Geiger, Lenz and Urtasun (CVPR 2012). Annotators place oriented 3D cuboids on the lidar scan, with calibration projecting every box into the camera images for verification in 2D.
- nuScenes 3D Cuboid Annotation on Lidar Point CloudsDraw 3D cuboids on a lidar point cloud and label each with one of 23 classes plus attributes. The ten point clouds are synthetic and illustrative.