How to Annotate Video Object Tracking
Track an object through a clip with model-assisted mask propagation, handle occlusion honestly, and measure temporal agreement with a tolerance sweep.
Tracking annotation means labelling the same object across frames, and the expensive part is not the first mask — it is the hundreds that follow. Model-assisted propagation removes most of that, provided the model is honest about when it has lost the object.
The workflow
- Mask the object on one frame.
- Propagate forward. Each result arrives as a keyframe.
- Scrub the keyframes, correct where the mask has drifted. Correcting a frame re-anchors the track from that point.
annotation_schemes:
- annotation_type: video_annotation
name: tracks
description: "Draw the object once, then press Track forward."
source_field: video_url
mode: tracking
labels:
- {name: subject, color: "#d1495b"}
tracking_options:
interpolation: linear
auto_advance_frames: 5
video_fps: 12
frame_stepping: trueOcclusion should return nothing
The single most useful property of a propagation model in an annotation loop is that it declines when the object is hidden, returning an empty frame rather than its best guess.
A plausible wrong mask on an occluded frame is worse than a gap: the annotator has to notice it, decide it is wrong, and remove it. An empty frame is immediately readable as "the object is not visible here", which is also the correct label.
SAM 2 does this, and its memory modules are what make propagation hold up across an occlusion rather than losing the object at the far side. Measured against known ground truth, per-frame IoU stays at 0.974 to 0.979 with no decay across the sequence.
Held versus blended
On frames between keyframes, a mask is either held (repeated from the last keyframe) or blended (interpolated). Held is the honest choice, and held frames should be visually marked.
A blended mask looks smoother and is wrong in a way nobody can see. If your consumer needs interpolated geometry, interpolate at export time where it is a stated transformation, rather than in the interface where it is invisible.
Polygon interpolation is harder than it looks
Interpolating between two polygon keyframes fails in two ways naive implementations get wrong:
- The two keyframes may have different vertex counts.
- The annotator may have started tracing at a different point around the shape.
Arc-length resampling with rotational alignment handles both. Without it, an interpolated polygon can turn inside out between keyframes.
Where propagation should run
On the server, if the model has memory. The cost is per frame rather than per prompt, and a hundred frames of a video model in a browser tab is minutes of frozen UI. Browser-side carry-forward that re-prompts frame to frame is a different, lighter thing — useful, but not the same capability, and worth not conflating in your own documentation.
Measuring agreement
Two annotators will not choose the same frame for an event boundary, so temporal agreement depends on a matching tolerance. Report the sweep, not one threshold:
- High at 2 s, low at 0.25 s: they agree an event happened, not precisely when. Usually a guidelines problem.
- Low at both: they disagree about whether it happened.
For the masks themselves, use the mask agreement machinery per frame.
A bug worth checking for in your own tooling
HTML video reports currentTime truncated to six decimal places. Converting that to a frame index with a naive multiply-and-floor gives an off-by-one after every seek: frame 14 of a 12 fps clip reads back as 13.
This was live in Potato before 2.8 and affected every annotator on every clip. If you have frame-indexed video annotations from any tool, it is worth verifying against a known frame.