Vision and Spatial Annotation
Potato annotates images, video, gigapixel scans, 3D point clouds, depth maps and robot episodes, and reports chance-corrected agreement over all of them. Overview of the vision and spatial surface.
Potato annotates images, video, gigapixel scans, 3D point clouds, depth maps, robot episodes and generative-video rollouts, all from one YAML config, and reports chance-corrected agreement over every one of them. This page is the map; each surface has its own page below.
Version 2.8 is where most of this arrived. Before it, Potato was a text annotation platform with an image mode.
What is here
| Surface | Schema | Page |
|---|---|---|
| Boxes, polygons, polylines, ellipses, cuboids, keypoints, masks | image_annotation | Geometry primitives |
| Click an object, get a mask | image_annotation with the sam tool | Interactive segmentation |
| Type a phrase, get every match boxed | image_annotation with text_prompt | Text prompting |
| Track an object through a clip | video_annotation | Video tracking |
| Gigapixel scans at full resolution | viewer: deepzoom | Deep zoom |
| 3D point clouds and oriented cuboids | spatial_annotation | Point clouds |
| Depth maps with a metres readout | depth_map display | Depth maps |
| Robot demonstrations on a timeline | episode_annotation | Robot episodes |
| Generative-video rollouts | rollout_evaluation | World-model evaluation |
| Referring expressions and pointing | grounding_eval, region_caption | VLM grounding |
| 15 import and 29 export formats | — | CV formats |
| Which models, and under which licence | — | Model zoo |
The part that is unusual
Drawing tools are not what distinguishes Potato here. CVAT, Label Studio, Roboflow, V7 and Supervisely all draw boxes and polygons, and several of them do it better; the comparison guide says where.
What Potato adds is that every one of these surfaces reports chance-corrected agreement. We have found no other annotation platform that reports a chance-corrected coefficient for spatial labels at all: the standard is raw IoU or percent match, which is inflated whenever one answer dominates, and one answer nearly always dominates. See Measurement and Integrity for what that means for boxes, masks, 3D cuboids, event timings and captions.
Models run locally
Interactive segmentation and open-vocabulary text prompting both run in the annotator's browser through ONNX Runtime Web. There is no GPU to provision, no model server to keep up, and no network call per click. Once the weights are on the machine, the whole thing works air-gapped.
Video mask propagation is the exception and runs on the server, by choice: its cost is per frame rather than per prompt, and a hundred frames of a video model in a browser tab would be minutes of frozen UI.
Nothing is bundled and nothing downloads unless you ask:
potato download-models --list
potato download-models mobile_samThe model zoo page states each model's licence, and download-models refuses to fetch a non-commercial model without --accept-licence.
Getting started
The fastest route is a runnable example. Each of these lives in the Potato repository and runs from the repository root with no downloads:
examples/image/interactive-segmentation/— click to segmentexamples/image/text-prompt-labeling/— type a phrase, label from itexamples/video/mask-propagation/— track through an occlusionexamples/spatial/kitti-cuboids/— 3D boxes on lidarexamples/embodied/lerobot-episode/— a robot demonstration
Further reading
- Measurement and Integrity — agreement over geometry, time and 3D
- Image annotation reference
- Video annotation reference
- Release notes for 2.8