Potato annotates images, video, gigapixel scans, and 3D point clouds with boxes, polygons, masks, and oriented cuboids, segments objects with SAM from a click, and reports agreement between annotators on every spatial label, chance-corrected on 2D shapes. Potato is free and open source, runs on your own computer or server, and exports COCO, YOLO, Pascal VOC, KITTI, and other standard formats.
Most of Potato's vision support arrived in version 2.8, built around one addition. Every spatial label in Potato comes with agreement between annotators, from a 2D box to a 3D cuboid. On 2D boxes, polygons, and masks the agreement is chance-corrected, and on 3D cuboids Potato reports the mean matched 3D IoU. We have found no other annotation platform that reports a chance-corrected coefficient for spatial labels. The usual measure is raw IoU or percent match, which runs high whenever one answer dominates, as one answer nearly always does. Measurement and integrity explains what that means for boxes, masks and cuboids.
Computer vision tasks in Potato
Each row is a surface that ships with Potato, and the linked page has its configuration.
| Task | How Potato does it | Docs |
|---|---|---|
| Detection and segmentation labels | Boxes, polygons, polylines, ellipses, keypoints, and masks on an image_annotation scheme | Geometry primitives |
| Click-to-segment | SAM-family models run in the annotator's browser through ONNX Runtime Web | Interactive segmentation |
| Label from a text prompt | Type a phrase and an open-vocabulary detector boxes every match | Text prompting |
| Video object tracking | Draw a mask on one frame and SAM 2 propagates it, with keyframes you correct | Video tracking |
| Gigapixel images | A tile pyramid with annotations that work at any zoom, for slides, aerial surveys, and scanned maps | Deep zoom |
| Lidar and point clouds | Oriented 3D cuboids, points, polylines, and per-point segments, projected into the camera images | Point clouds |
| Image classification | Radio and multiselect labels on each image, with keyboard shortcuts | Image annotation |
Models run on your hardware
Interactive segmentation and text prompting run in the annotator's browser, so there is no GPU to provision and no network call per click. Video mask propagation runs on the Potato server instead, because its cost grows with the number of frames. No model is bundled. potato download-models fetches the weights you choose once, and after that the tool runs air-gapped. The model zoo states each model's license, and download-models refuses to fetch a non-commercial model without --accept-licence.
potato download-models --list
potato download-models mobile_samFormats in and out
potato import reads 16 image and video formats, among them COCO (with RLE masks, crowd regions and keypoints), YOLO, Pascal VOC, CVAT XML, Labelbox, LabelMe, KITTI, MOT, DAVIS, Cityscapes, and Hugging Face datasets. The exporters write COCO, YOLO, Pascal VOC, CVAT, V7 Darwin, LabelMe, KITTI 2D and 3D, MOTChallenge, DAVIS, Cityscapes, PNG masks, and NIfTI or DICOM SEG for medical volumes. Formats differ in which geometry each one keeps, and Computer vision formats says what each one keeps.
Working alongside CVAT, Label Studio, and Roboflow
Drawing tools are not what sets Potato apart. CVAT, Label Studio, Roboflow, V7, and Supervisely all draw boxes and polygons, and several draw them better than Potato does. CVAT in particular is built for high-volume vision labeling. Potato fits when the labels are part of a study, with several annotators per image, agreement reported on every spatial label, attention and gold checks, and the text, audio, or survey questions of the same project in one tool. Image annotation tools compared goes through the differences.
Frequently asked questions
Can Potato export annotations in COCO or YOLO format?
Yes. python -m potato.export --format coco writes COCO JSON for detection and segmentation, and --format yolo writes YOLO label files. The COCO exporter converts the stored normalized coordinates to pixels and turns polygons, ellipses and brush masks into COCO segmentation. Pascal VOC, CVAT, KITTI, Cityscapes, and PNG masks are also available.
Can SAM pre-segment objects so annotators only correct the masks?
Yes. With the sam tool on an image_annotation scheme, an annotator clicks an object and gets a mask to accept or edit. The model runs in the annotator's browser once its weights are downloaded with potato download-models. For video, SAM 2 propagates a mask from one frame through the following frames on the server.
How do I measure agreement between annotators on bounding boxes?
Assign each image to at least two annotators and Potato reports chance-corrected agreement on the boxes, separating whether annotators found the same objects from how closely their boxes match. Measuring agreement on bounding boxes explains the method, and the span and box agreement calculator runs the same computation on an export in the browser.
Can I label 3D point clouds and lidar scans?
Yes. A spatial_annotation scheme shows a point cloud in the browser with oriented 3D cuboids, points, polylines, and per-point segments. Octree level of detail keeps a scan of several million points interactive on a laptop, and camera calibration projects each box into the images for checking in 2D. The kitti_3d exporter writes KITTI labels.
Further reading
- Image segmentation annotation, for masks, SAM, and quality checks
- Point cloud annotation, for a 3D labeling workflow end to end
- Video annotation tools compared, for tracking and temporal labels across tools
- Robotics and embodied AI, ecology, and healthcare, for fields that build on the same vision tools