Skip to content
Diese Seite ist in Ihrer Sprache noch nicht verfügbar. Englische Version wird angezeigt.

Video Tracking and Mask Propagation

Draw a mask on one frame and SAM 2 follows the object through the clip, measured at 0.974 to 0.979 IoU per frame with no decay. Runs server-side by design.

Draw a mask on one frame, press Track forward, and SAM 2 follows the object through the frames that follow, each result arriving as a keyframe you can scrub through and correct. Measured against known ground truth, per-frame IoU holds at 0.974 to 0.979 with no decay from the first frame to the last.

yaml
annotation_schemes:
  - annotation_type: video_annotation
    name: tracks
    description: "Draw the object once, then press Track forward."
    source_field: video_url
    mode: tracking
    labels:
      - {name: subject, color: "#d1495b"}
    tracking_options:
      interpolation: linear
      auto_advance_frames: 5
    video_fps: 12
    show_timecode: true
    frame_stepping: true

Runnable example: examples/video/mask-propagation/.

This one runs on the server

Propagation uses SAM 2 with its memory modules (bank ordering, repeat-padding, temporal encodings, object-pointer chunking) rather than re-prompting each frame independently.

It runs server-side, deliberately. The cost is per frame rather than per prompt, the model is 181 MB across five graphs, and the video is already there. A hundred frames of it in a browser tab would be minutes of frozen UI. Roughly 1.32 seconds per frame on CPU, faster on a GPU.

Potato also has a lighter in-browser carry-forward path that re-prompts frame to frame. The two are different things and it is worth not blurring them: the capability is available on the free, self-hosted product either way, but the memory-based propagation is the server-side one.

Occlusion is answered, not guessed

The model decides for itself when an object is occluded and returns an empty frame rather than a guess. That is the behaviour you want while correcting: a guessed mask on a hidden frame is work to undo, and a plausible wrong mask is worse than an obvious gap.

Masks on interpolated frames are held, not blended, and held frames are marked in the timeline. An honest representation beats a smooth wrong one.

Tubelets and polygon interpolation

Alongside mask propagation, Potato interpolates polygon tracks by arc-length resampling with rotational alignment. That handles the two cases naive interpolation gets wrong: differing vertex counts between keyframes, and differing tracing start points around the same shape.

A frame-indexing bug worth knowing about

getCurrentFrame was off by one after every seek, on every clip, in every release before 2.8. HTML video reports currentTime truncated to six decimal places, so frame 14 of a 12 fps clip read back as 13.

If you have frame-indexed video annotations from an earlier version, they may be shifted by one frame after any seek.