Video Annotation

Annotate video in Potato with temporal segments, frame classification, keyframes, and object tracking, using frame-by-frame navigation along the timeline.

Potato supports video annotation across temporal segmentation, frame classification, keyframe marking, and object tracking.

Annotation Modes

The mode you pick decides what the annotator produces and which controls appear, so choose it from the unit of analysis your study needs.

Mode	Description	Use Case
`segment`	Mark temporal ranges	Scene detection, speaker turns
`frame`	Classify individual frames	Frame-level labeling
`keyframe`	Mark important moments	Highlight detection
`tracking`	Track objects across frames	Object tracking
`combined`	All modes in one interface	Complex annotation tasks

Basic Configuration

A minimal scheme names the labels and leaves every display option at its default, which puts the annotator on the segment timeline with zoom, speed control, and frame stepping already available:

yaml

annotation_schemes:
  - name: "video_segments"
    description: "Mark segments where the speaker changes"
    annotation_type: "video_annotation"
    labels:
      - name: "Speaker A"
        color: "#FF6B6B"
      - name: "Speaker B"
        color: "#4ECDC4"

Configuration Options

Field	Type	Default	Description
`name`	string	Required	Unique identifier for the annotation
`description`	string	Required	Instructions shown to annotators
`annotation_type`	string	Required	Must be `"video_annotation"`
`labels`	list	Required	Available annotation labels
`mode`	string	`"segment"`	Annotation mode
`min_segments`	integer	0	Minimum required annotations
`max_segments`	integer	null	Maximum allowed annotations
`timeline_height`	integer	70	Timeline height in pixels
`overview_height`	integer	40	Overview bar height in pixels
`zoom_enabled`	boolean	true	Enable timeline zoom
`playback_rate_control`	boolean	true	Show playback speed selector
`frame_stepping`	boolean	true	Enable frame-by-frame navigation
`show_timecode`	boolean	true	Display frame number and time
`video_fps`	integer	30	Video frame rate for calculations
`source_field`	string	`""`	Display field this schema annotates, when the item shows more than one video

Label Configuration

Labels can be simple strings or detailed objects with colors and keyboard shortcuts. A key_value binds the label to a key, which matters most in segment work where the annotator's hands stay on playback controls:

yaml

labels:
  - name: "intro"
    color: "#4ECDC4"
    key_value: "1"
  - name: "content"
    color: "#3B82F6"
    key_value: "2"
  - name: "outro"
    color: "#8B5CF6"
    key_value: "3"

Examples by Mode

Segment Mode (Default)

Mark temporal ranges with start and end times. Zoom and playback speed are the two controls that decide how precisely an annotator can place a boundary, so scene work leaves both on:

yaml

annotation_schemes:
  - name: "scene_detection"
    description: "Mark each scene in the video"
    annotation_type: "video_annotation"
    mode: "segment"
    labels:
      - name: "indoor"
        color: "#3B82F6"
        key_value: "1"
      - name: "outdoor"
        color: "#22C55E"
        key_value: "2"
      - name: "transition"
        color: "#F59E0B"
        key_value: "3"
    zoom_enabled: true
    playback_rate_control: true

Frame Mode

Classify individual frames. Frame stepping and the timecode readout let the annotator land on one frame and record which frame they judged, which is what makes the labels reproducible:

yaml

annotation_schemes:
  - name: "frame_quality"
    description: "Classify each frame's quality"
    annotation_type: "video_annotation"
    mode: "frame"
    labels:
      - name: "good"
        color: "#22C55E"
      - name: "blurry"
        color: "#F59E0B"
      - name: "occluded"
        color: "#EF4444"
    frame_stepping: true
    show_timecode: true

Keyframe Mode

Mark important moments. A keyframe is a single point rather than a range, so frame stepping carries the accuracy of the whole task:

yaml

annotation_schemes:
  - name: "highlights"
    description: "Mark key moments in the video"
    annotation_type: "video_annotation"
    mode: "keyframe"
    labels:
      - name: "goal"
        color: "#22C55E"
      - name: "foul"
        color: "#EF4444"
      - name: "highlight"
        color: "#F59E0B"
    frame_stepping: true

Tracking Mode

Track objects across frames with bounding boxes. Set video_fps to the frame rate of your own files, because Potato converts times to frame numbers with this value and a mismatch shifts every box:

yaml

annotation_schemes:
  - name: "object_tracking"
    description: "Track the ball throughout the video"
    annotation_type: "video_annotation"
    mode: "tracking"
    labels:
      - name: "ball"
        color: "#3B82F6"
      - name: "player"
        color: "#22C55E"
    frame_stepping: true
    video_fps: 30

Combined Mode

Combined mode puts segments, frame labels, keyframes, and tracking in one interface. It asks the most of an annotator per video, so use it when the same pass has to produce several kinds of annotation:

yaml

annotation_schemes:
  - name: "full_pass"
    description: "Full video annotation"
    annotation_type: "video_annotation"
    mode: "combined"
    labels:
      - name: "action"
        color: "#3B82F6"
      - name: "dialogue"
        color: "#22C55E"
      - name: "transition"
        color: "#F59E0B"
    zoom_enabled: true
    playback_rate_control: true
    frame_stepping: true
    show_timecode: true

Video Display

For video playback without annotation (e.g., to accompany other annotation types), use the video type. It renders a player and collects nothing, so the annotator watches the clip and answers whatever other schemes the task defines:

yaml

annotation_schemes:
  - name: "video_player"
    description: "Watch the video clip"
    annotation_type: "video"
    video_path: "{{video_url}}"
    controls: true
    autoplay: false
    loop: false
    muted: false

Subtitles and Captions

If the video has captions, you can annotate those instead of, or alongside, the video timeline. Potato reads WebVTT, SubRip, TTML/DFXP, SubStation Alpha, and YouTube's json3 and srv1/srv2/srv3 formats, detecting each from the file contents rather than the extension.

bash

yt-dlp --write-auto-subs --sub-langs en --sub-format vtt --skip-download <URL>

json

{"id": "talk_01", "conversation": {"audio": "media/talk_01.mp3",
                                   "transcript": "captions/talk_01.vtt"}}

Note: Auto-generated captions have no punctuation and no speaker labels, and their cue boundaries fall where the caption window filled up rather than where a sentence ended. Do not build a sentence-level scheme over them. See Annotating YouTube Subtitles for what to do instead.

The full format list is in Transcript Formats.

Keyboard Shortcuts

Potato generates the shortcut list from the scheme, so the frame, segment, keyframe, and classification keys appear only in the modes that use them.

Key	Action
`Space`	Play/pause
`←` / `→`	Seek 5 seconds
`,` / `.`	Previous/next frame (when `frame_stepping` is on)
`[`	Set segment start (segment and combined modes)
`]`	Set segment end (segment and combined modes)
`Enter`	Create segment (segment and combined modes)
`K`	Mark keyframe (keyframe and combined modes)
`C`	Classify current frame (frame and combined modes)
`Del`	Delete selected
`+` / `-`	Zoom in/out
`0`	Fit to view

Each label with a key_value adds its own key, so 1, 2, and 3 select labels in the examples above.

Data Format

Input Data

Your data file should include video paths or URLs:

json

[
  {
    "id": "video_1",
    "video_url": "https://example.com/videos/sample1.mp4",
    "description": "Meeting recording - identify speakers"
  },
  {
    "id": "video_2",
    "video_url": "/data/videos/sample2.mp4",
    "description": "Activity video - label actions"
  }
]

Output Format

Output varies by mode:

Segment Mode:

json

{
  "id": "video_1",
  "annotations": {
    "scene_detection": [
      {
        "start": 0.0,
        "end": 5.2,
        "start_frame": 0,
        "end_frame": 156,
        "label": "indoor"
      }
    ]
  }
}

Frame Mode:

json

{
  "id": "video_1",
  "annotations": {
    "frame_quality": [
      {
        "frame": 120,
        "time": 4.0,
        "label": "good"
      }
    ]
  }
}

Keyframe Mode:

json

{
  "id": "video_1",
  "annotations": {
    "highlights": [
      {
        "frame": 450,
        "time": 15.0,
        "label": "goal",
        "note": "First goal of the match"
      }
    ]
  }
}

Tracking Mode:

json

{
  "id": "video_1",
  "annotations": {
    "object_tracking": [
      {
        "id": "track_1",
        "label": "ball",
        "frames": [
          {"frame": 0, "bbox": {"x": 100, "y": 200, "w": 30, "h": 30}},
          {"frame": 1, "bbox": {"x": 105, "y": 198, "w": 30, "h": 30}}
        ]
      }
    ]
  }
}

Supported Video Formats

MP4 (recommended)
WebM
MOV
OGG

Video Object Tracking with Keyframe Interpolation

New in v2.2.0

Tracking mode interpolates between keyframes. An annotator marks an object's position on a few frames and Potato fills in the frames between them, so a tracked object costs a handful of boxes rather than one box per frame. Set video_fps to the frame rate of the source file, since the interpolation is computed over frame numbers:

yaml

annotation_schemes:
  - name: "tracking"
    description: "Track objects with keyframe interpolation"
    annotation_type: "video_annotation"
    mode: "tracking"
    labels:
      - name: "person"
        color: "#3B82F6"
      - name: "vehicle"
        color: "#22C55E"
    frame_stepping: true
    video_fps: 30

Workflow

Navigate to a keyframe and draw a bounding box around the object
Skip ahead several frames and reposition the bounding box
The system automatically interpolates positions for intermediate frames
Review and adjust interpolated positions as needed

The full set of valid annotation modes is: segment, frame, keyframe, tracking, and combined.

Technical Notes

Uses Peaks.js for timeline visualization
Standard HTML5 video elements (no additional server dependencies)
Timeline supports drag-to-select for segment creation

Best Practices

Use compressed videos - Large files slow down loading
Set appropriate FPS - Match video_fps to your actual video frame rate
Keep frame stepping on - Frame-level work depends on it
Use playback rate control - Slow motion helps with detailed work
Provide clear segment definitions - Define what constitutes segment boundaries
Use distinct colors - Make labels visually distinguishable
Consider timeline height - Increase for complex segmentation tasks