Video Annotation with Frame-by-Frame Controls

This tutorial builds a video annotation task in Potato with temporal segments, frame-by-frame navigation, keyframe markers, and per-segment questions. For the full video scheme reference, see the source documentation.

Four modes, one schema

Video work in Potato runs through a single annotation type, video_annotation, whose mode decides what the annotator marks. Choosing the mode first saves reworking the config later, because the timeline, the keyboard bindings, and the output all follow from it.

A minimal segment task

The schema needs a name, a description shown to the annotator, and the label set. Everything else has a default, so the shortest working config is short. item_properties.text_key names the field holding the video URL, which is how Potato finds the media for each item.

annotation_task_name: "Video Action Recognition"
task_dir: "."
 
data_files:
  - "data/videos.json"
 
item_properties:
  id_key: id
  text_key: video_url
 
output_annotation_dir: "annotation_output/"
export_annotation_format: "jsonl"
 
annotation_schemes:
  - annotation_type: video_annotation
    name: action
    description: "Mark each stretch of video where one action is happening"
    mode: segment
    labels:
      - name: walking
        color: "#4ECDC4"
        key_value: "1"
      - name: running
        color: "#FF6B6B"
        key_value: "2"
      - name: jumping
        color: "#45B7D1"
        key_value: "3"
      - name: sitting
        color: "#FFEAA7"
        key_value: "4"
 
user_config:
  allow_all_users: true

Giving each label a key_value binds it to a number key, which matters more here than in a text task. The annotator's other hand is already on the transport controls, so reaching for the mouse to pick a label costs more than the label choice itself. To create a segment, they navigate to the start, press [, navigate to the end, press ], select a label, and press Enter, then drag the boundaries if the first pass was close rather than exact.

Requiring segments

An annotator who scrubs through a clip and clicks Next leaves you an empty record that looks like a real one. min_segments sets the floor, and it only blocks the Next button when the scheme is also marked required:

    min_segments: 1
    label_requirement:
      required: true

Without label_requirement.required: true, min_segments is recorded and enforces nothing. With it, Next is blocked until the annotator has marked at least that many segments and labeled every one, and the message names what is missing.

Frame-level classification

Frame mode classifies single frames rather than stretches, which is what sequence models trained on per-frame labels need. frame_stepping turns on the , and . keys for single-frame movement, and video_fps tells Potato how to convert times to frame numbers, so it has to match the source footage or every frame number in the export is off:

annotation_schemes:
  - annotation_type: video_annotation
    name: frame_label
    description: "Classify the current frame"
    mode: frame
    frame_stepping: true
    show_timecode: true
    video_fps: 30
    labels:
      - name: action
        color: "#FF6B6B"
        key_value: "1"
      - name: no_action
        color: "#9CA3AF"
        key_value: "2"

The annotator steps to a frame with , and ., then presses C to classify it. Frame mode is the slowest of the four per minute of footage, so it pays to cut clips down to the stretches that need per-frame labels before handing them out.

Keyframes for single moments

Some events have no useful duration. The moment a ball crosses a line is an instant, and asking an annotator to give it a start and an end invents a boundary that nobody can reproduce. Keyframe mode records the position and an optional note:

annotation_schemes:
  - annotation_type: video_annotation
    name: key_moments
    description: "Mark the moment each action begins, peaks, and ends"
    mode: keyframe
    labels:
      - name: action_start
        color: "#22C55E"
        key_value: "1"
      - name: action_peak
        color: "#EF4444"
        key_value: "2"
      - name: action_end
        color: "#3B82F6"
        key_value: "3"

Pressing K marks the current position, and each keyframe carries a label and an optional note. Because a keyframe is a position rather than a span, two annotators can be compared on how far apart their marks fall, which is a cleaner disagreement to read than two overlapping segments.

Questions attached to each segment

A label alone cannot carry the detail some studies need, so segment_schemes hangs a full annotation scheme off every segment. The annotator marks a region, selects it, and answers in the segment panel, and the answers are stored on the segment rather than on the clip:

annotation_schemes:
  - annotation_type: video_annotation
    name: plays
    description: "Mark each play and describe it"
    mode: segment
    labels:
      - name: attack
        color: "#EF4444"
      - name: defense
        color: "#3B82F6"
    segment_schemes:
      - annotation_type: radio
        name: outcome
        description: "How did this play end?"
        labels: ["scored", "turnover", "out of play"]

Two limits are worth knowing before you build a task around this. Keyboard shortcuts are not bound inside a segment scheme, since the fields only exist while that segment is selected, so those answers are given by clicking. label_requirement is not enforced on a sub-scheme either, so completeness has to be checked in analysis rather than at the Next button.

Clip-level labels alongside segments

Ordinary annotation types work on the same item as the video schema, which is how a clip gets both a timeline and a verdict. A multiselect covers tags that are not mutually exclusive:

annotation_schemes:
  - annotation_type: multiselect
    name: clip_tags
    description: "Tags that apply to this clip as a whole"
    labels:
      - Highlight worthy
      - Good camera angle
      - Multiple players
      - Close-up
      - Wide shot
      - Slow motion available

A complete sports configuration

Putting the pieces together gives a task that marks game events on the timeline, tags the clip as a whole, and keeps the rules on screen. annotation_instructions renders as a collapsible panel above every item, and markdown in it is rendered:

annotation_task_name: "Sports Video Analysis"
task_dir: "."
 
data_files:
  - "data/sports_clips.json"
 
item_properties:
  id_key: id
  text_key: video_url
 
output_annotation_dir: "annotations/"
export_annotation_format: "jsonl"
 
annotation_instructions: |
  ## Event marking
 
  - Mark an event from the frame it **starts**, not from the frame it becomes obvious.
  - Include the full play sequence in the segment.
  - For a goal, run the segment from the shot to the ball crossing the line.
 
  ## Navigation
 
  - `Space` plays and pauses, `,` and `.` step one frame.
  - `[` and `]` set the segment boundaries, `Enter` creates the segment.
 
annotation_schemes:
  - annotation_type: video_annotation
    name: game_events
    description: "Mark every game event and its duration"
    mode: segment
    frame_stepping: true
    show_timecode: true
    playback_rate_control: true
    video_fps: 30
    min_segments: 1
    label_requirement:
      required: true
    labels:
      - name: goal
        color: "#22C55E"
        key_value: "1"
      - name: foul
        color: "#EF4444"
        key_value: "2"
      - name: corner_kick
        color: "#3B82F6"
        key_value: "3"
      - name: free_kick
        color: "#F59E0B"
        key_value: "4"
      - name: penalty
        color: "#EC4899"
        key_value: "5"
      - name: offside
        color: "#8B5CF6"
        key_value: "6"
 
  - annotation_type: multiselect
    name: clip_tags
    description: "Tags that apply to this clip as a whole"
    labels:
      - Highlight worthy
      - Good camera angle
      - Multiple players
      - Close-up
      - Wide shot
      - Slow motion available
 
user_config:
  allow_all_users: true

Keyboard shortcuts

Video annotation is paced by the transport controls, so the shortcuts are worth putting in front of annotators on day one:

What the output file holds

Video annotations are written as one object per item, with the segments, frame labels, and keyframes kept separate and each segment carrying both times and frame numbers:

{
    "video_metadata": {
        "duration": 120.5,
        "fps": 30,
        "width": 1920,
        "height": 1080
    },
    "segments": [
        {
            "id": "segment_1",
            "start_time": 0.0,
            "end_time": 10.5,
            "start_frame": 0,
            "end_frame": 315,
            "label": "goal"
        }
    ],
    "frame_annotations": {
        "450": {
            "frame": 450,
            "time": 15.0,
            "label": "scene_change"
        }
    },
    "keyframes": [
        {
            "id": "kf_1",
            "frame": 900,
            "time": 30.0,
            "label": "important_moment",
            "note": "Key dialogue"
        }
    ],
    "tracking": {}
}

Frame numbers are computed from video_fps, so a config that claims 30 fps against 25 fps footage writes plausible-looking numbers that point at the wrong frames. Check the frame rate of the source files before the first batch goes out, because the times stay correct while the frame numbers silently do not, and nothing in the export flags the mismatch.

Working faster

Watching the clip once at normal speed before marking anything costs a minute and prevents the common failure, which is a first segment drawn before the annotator knows what the clip contains. For the boundary itself, turn on playback_rate_control and slow the video down rather than scrubbing at full speed, since a boundary set while scrubbing is usually a frame or two late. Video work is also more tiring than text work, so plan for shorter sessions than you would for a classification task and write down each edge case as it is settled, while the clip that raised it is still open.

Next steps

• Learn image comparison for video quality assessment
• Set up crowdsourcing for large-scale video annotation
• Read the image annotation documentation for the video section's full option list