# Video Tracking and Mask Propagation

Source: https://www.potatoannotator.com/docs/vision-spatial/video-tracking

**Draw a mask on one frame, press Track forward, and SAM 2 follows the object through the frames that follow, each result arriving as a keyframe you can scrub through and correct.** Measured against known ground truth, per-frame IoU holds at **0.974 to 0.979 with no decay** from the first frame to the last.

```yaml
annotation_schemes:
  - annotation_type: video_annotation
    name: tracks
    description: "Draw the object once, then press Track forward."
    source_field: video_url
    mode: tracking
    labels:
      - {name: subject, color: "#d1495b"}
    tracking_options:
      interpolation: linear
      auto_advance_frames: 5
    video_fps: 12
    show_timecode: true
    frame_stepping: true
```

Runnable example: [`examples/video/mask-propagation/`](https://github.com/davidjurgens/potato/tree/main/examples/video/mask-propagation).

## This one runs on the server

Propagation uses [SAM 2](https://arxiv.org/abs/2408.00714) with its memory modules (bank ordering, repeat-padding, temporal encodings, object-pointer chunking) rather than re-prompting each frame independently.

It runs **server-side**, deliberately. The cost is per frame rather than per prompt, the model is 181 MB across five graphs, and the video is already there. A hundred frames of it in a browser tab would be minutes of frozen UI. Roughly 1.32 seconds per frame on CPU, faster on a GPU.

Potato also has a lighter in-browser carry-forward path that re-prompts frame to frame. The two are different things and it is worth not blurring them: the capability is available on the free, self-hosted product either way, but the memory-based propagation is the server-side one.

## Occlusion is answered, not guessed

The model decides for itself when an object is occluded and returns an **empty frame** rather than a guess. That is the behaviour you want while correcting: a guessed mask on a hidden frame is work to undo, and a plausible wrong mask is worse than an obvious gap.

Masks on interpolated frames are **held, not blended**, and held frames are marked in the timeline. An honest representation beats a smooth wrong one.

## Tubelets and polygon interpolation

Alongside mask propagation, Potato interpolates polygon tracks by arc-length resampling with rotational alignment. That handles the two cases naive interpolation gets wrong: differing vertex counts between keyframes, and differing tracing start points around the same shape.

## A frame-indexing bug worth knowing about

`getCurrentFrame` was off by one after every seek, on every clip, in every release before 2.8. HTML video reports `currentTime` truncated to six decimal places, so frame 14 of a 12 fps clip read back as 13.

If you have frame-indexed video annotations from an earlier version, they may be shifted by one frame after any seek.

## Related

- [Interactive segmentation](/docs/vision-spatial/segmentation)
- [Temporal agreement](/docs/measurement/temporal-agreement) — agreement on *when* something happens
- [Guide: video object tracking](/docs/guides/video-object-tracking)
- [Video annotation reference](/docs/annotation-types/video-annotation)