# Vision and Spatial Annotation

Source: https://www.potatoannotator.com/docs/vision-spatial/overview

**Potato annotates images, video, gigapixel scans, 3D point clouds, depth maps, robot episodes and generative-video rollouts, all from one YAML config, and reports chance-corrected agreement over every one of them.** This page is the map; each surface has its own page below.

Version 2.8 is where most of this arrived. Before it, Potato was a text annotation platform with an image mode.

## What is here

| Surface | Schema | Page |
|---|---|---|
| Boxes, polygons, polylines, ellipses, cuboids, keypoints, masks | `image_annotation` | [Geometry primitives](/docs/vision-spatial/geometry-primitives) |
| Click an object, get a mask | `image_annotation` with the `sam` tool | [Interactive segmentation](/docs/vision-spatial/segmentation) |
| Type a phrase, get every match boxed | `image_annotation` with `text_prompt` | [Text prompting](/docs/vision-spatial/text-prompting) |
| Track an object through a clip | `video_annotation` | [Video tracking](/docs/vision-spatial/video-tracking) |
| Gigapixel scans at full resolution | `viewer: deepzoom` | [Deep zoom](/docs/vision-spatial/deep-zoom) |
| 3D point clouds and oriented cuboids | `spatial_annotation` | [Point clouds](/docs/vision-spatial/point-clouds) |
| Depth maps with a metres readout | `depth_map` display | [Depth maps](/docs/vision-spatial/depth-maps) |
| Robot demonstrations on a timeline | `episode_annotation` | [Robot episodes](/docs/vision-spatial/robot-episodes) |
| Generative-video rollouts | `rollout_evaluation` | [World-model evaluation](/docs/vision-spatial/world-model-evaluation) |
| Referring expressions and pointing | `grounding_eval`, `region_caption` | [VLM grounding](/docs/vision-spatial/vlm-grounding) |
| 15 import and 29 export formats | — | [CV formats](/docs/vision-spatial/cv-formats) |
| Which models, and under which licence | — | [Model zoo](/docs/vision-spatial/model-zoo) |

## The part that is unusual

Drawing tools are not what distinguishes Potato here. CVAT, Label Studio, Roboflow, V7 and Supervisely all draw boxes and polygons, and several of them do it better; the [comparison guide](/docs/guides/annotation-tools-compared) says where.

What Potato adds is that every one of these surfaces reports **chance-corrected agreement**. We have found no other annotation platform that reports a chance-corrected coefficient for spatial labels at all: the standard is raw IoU or percent match, which is inflated whenever one answer dominates, and one answer nearly always dominates. See [Measurement and Integrity](/docs/measurement/overview) for what that means for boxes, masks, 3D cuboids, event timings and captions.

## Models run locally

Interactive segmentation and open-vocabulary text prompting both run in the annotator's browser through [ONNX Runtime Web](https://onnxruntime.ai/docs/tutorials/web/). There is no GPU to provision, no model server to keep up, and no network call per click. Once the weights are on the machine, the whole thing works air-gapped.

Video mask propagation is the exception and runs on the server, by choice: its cost is per frame rather than per prompt, and a hundred frames of a video model in a browser tab would be minutes of frozen UI.

Nothing is bundled and nothing downloads unless you ask:

```bash
potato download-models --list
potato download-models mobile_sam
```

The [model zoo](/docs/vision-spatial/model-zoo) page states each model's licence, and `download-models` refuses to fetch a non-commercial model without `--accept-licence`.

## Getting started

The fastest route is a runnable example. Each of these lives in the Potato repository and runs from the repository root with no downloads:

- [`examples/image/interactive-segmentation/`](https://github.com/davidjurgens/potato/tree/main/examples/image/interactive-segmentation) — click to segment
- [`examples/image/text-prompt-labeling/`](https://github.com/davidjurgens/potato/tree/main/examples/image/text-prompt-labeling) — type a phrase, label from it
- [`examples/video/mask-propagation/`](https://github.com/davidjurgens/potato/tree/main/examples/video/mask-propagation) — track through an occlusion
- [`examples/spatial/kitti-cuboids/`](https://github.com/davidjurgens/potato/tree/main/examples/spatial/kitti-cuboids) — 3D boxes on lidar
- [`examples/embodied/lerobot-episode/`](https://github.com/davidjurgens/potato/tree/main/examples/embodied/lerobot-episode) — a robot demonstration

## Further reading

- [Measurement and Integrity](/docs/measurement/overview) — agreement over geometry, time and 3D
- [Image annotation reference](/docs/annotation-types/image-annotation)
- [Video annotation reference](/docs/annotation-types/video-annotation)
- [Release notes for 2.8](https://github.com/davidjurgens/potato/blob/main/docs/releasenotes/v2.8.0.md)