Video Annotation

Potato supports video annotation across temporal segmentation, frame classification, keyframe marking, and object tracking.

Annotation Modes

Basic Configuration

annotation_schemes:
  - name: "video_segments"

Configuration Options

Label Configuration

Labels can be simple strings or detailed objects with colors and keyboard shortcuts:

labels:
  - name: "intro"
    color: "#4ECDC4"
    key_value: "1"
  - name: "content"
    color: "#3B82F6"
    key_value: "2"
  - name: "outro"
    color: "#8B5CF6"
    key_value: "3"

Examples by Mode

Segment Mode (Default)

Mark temporal ranges with start and end times:

annotation_schemes:
  - name: "scene_detection"

Frame Mode

Classify individual frames:

annotation_schemes:
  - name: "frame_quality"

Keyframe Mode

Mark important moments:

annotation_schemes:
  - name: "highlights"

Tracking Mode

Track objects across frames with bounding boxes:

annotation_schemes:
  - name: "object_tracking"

Combined Mode

Use multiple annotation types in one interface:

annotation_schemes:
  - name: "comprehensive"

Video Display

For simple video playback without annotation (e.g., to accompany other annotation types), use the video type:

annotation_schemes:
  - name: "video_player"

Subtitles and Captions

If the video has captions, you can annotate those instead of, or alongside, the video timeline. Potato reads WebVTT, SubRip, TTML/DFXP, SubStation Alpha, and YouTube's json3 and srv1/srv2/srv3 formats, detecting each from the file contents rather than the extension.

yt-dlp --write-auto-subs --sub-langs en --sub-format vtt --skip-download <URL>

{"id": "talk_01", "conversation": {"audio": "media/talk_01.mp3",
                                   "transcript": "captions/talk_01.vtt"}}

Note: Auto-generated captions have no punctuation and no speaker labels, and their cue boundaries fall where the caption window filled up rather than where a sentence ended. Do not build a sentence-level scheme over them. See Annotating YouTube Subtitles for what to do instead.

The full format list is in Transcript Formats.

Keyboard Shortcuts

Data Format

Input Data

Your data file should include video paths or URLs:

[
  {
    "id": "video_1",
    "video_url": "https://example.com/videos/sample1.mp4",
    "description": "Meeting recording - identify speakers"
  },
  {
    "id": "video_2",
    "video_url": "/data/videos/sample2.mp4",
    "description": "Activity video - label actions"
  }
]

Output Format

Output varies by mode:

Segment Mode:

{
  "id": "video_1",
  "annotations": {
    "scene_detection": [
      {
        "start": 0.0,
        "end": 5.2,
        "start_frame": 0,
        "end_frame": 156,
        "label": "indoor"
      }
    ]
  }
}

Frame Mode:

{
  "id": "video_1",
  "annotations": {
    "frame_quality": [
      {
        "frame": 120,
        "time": 4.0,
        "label": "good"
      }
    ]
  }
}

Keyframe Mode:

{
  "id": "video_1",
  "annotations": {
    "highlights": [
      {
        "frame": 450,
        "time": 15.0,
        "label": "goal",
        "note": "First goal of the match"
      }
    ]
  }
}

Tracking Mode:

{
  "id": "video_1",
  "annotations": {
    "object_tracking": [
      {
        "id": "track_1",
        "label": "ball",
        "frames": [
          {"frame": 0, "bbox": {"x": 100, "y": 200, "w": 30, "h": 30}},
          {"frame": 1, "bbox": {"x": 105, "y": 198, "w": 30, "h": 30}}
        ]
      }
    ]
  }
}

Supported Video Formats

• MP4 (recommended)
• WebM
• MOV
• OGG

Video Object Tracking with Keyframe Interpolation

New in v2.2.0

The tracking mode now supports keyframe interpolation, allowing annotators to mark object positions at key frames and have intermediate frames automatically interpolated. This significantly speeds up object tracking tasks.

annotation_schemes:
  - name: "tracking"

Workflow

1. Navigate to a keyframe and draw a bounding box around the object
2. Skip ahead several frames and reposition the bounding box
3. The system automatically interpolates positions for intermediate frames
4. Review and adjust interpolated positions as needed

The full set of valid annotation modes is: segment, frame, keyframe, tracking, and combined.

Technical Notes

• Uses Peaks.js for timeline visualization
• Standard HTML5 video elements (no additional server dependencies)
• Timeline supports drag-to-select for segment creation

Best Practices

1. Use compressed videos - Large files slow down loading
2. Set appropriate FPS - Match video_fps to your actual video frame rate
3. Enable frame stepping - Essential for precise frame-level annotation
4. Use playback rate control - Slow motion helps with detailed work
5. Provide clear segment definitions - Define what constitutes segment boundaries
6. Use distinct colors - Make labels visually distinguishable
7. Consider timeline height - Increase for complex segmentation tasks