# Video Annotation

Source: https://www.potatoannotator.com/docs/annotation-types/video-annotation

Potato supports video annotation across temporal segmentation, frame classification, keyframe marking, and object tracking.

## Annotation Modes

| Mode | Description | Use Case |
|------|-------------|----------|
| `segment` | Mark temporal ranges | Scene detection, speaker turns |
| `frame` | Classify individual frames | Frame-level labeling |
| `keyframe` | Mark important moments | Highlight detection |
| `tracking` | Track objects across frames | Object tracking |
| `combined` | All modes in one interface | Complex annotation tasks |

## Basic Configuration

```yaml
annotation_schemes:
  - name: "video_segments"
    description: "Mark segments where the speaker changes"
    annotation_type: "video_annotation"
    labels:
      - name: "Speaker A"
        color: "#FF6B6B"
      - name: "Speaker B"
        color: "#4ECDC4"
```

## Configuration Options

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `name` | string | Required | Unique identifier for the annotation |
| `description` | string | Required | Instructions shown to annotators |
| `annotation_type` | string | Required | Must be `"video_annotation"` |
| `labels` | list | Required | Available annotation labels |
| `mode` | string | `"segment"` | Annotation mode |
| `min_segments` | integer | 0 | Minimum required annotations |
| `max_segments` | integer | null | Maximum allowed annotations |
| `timeline_height` | integer | 70 | Timeline height in pixels |
| `overview_height` | integer | 40 | Overview bar height in pixels |
| `zoom_enabled` | boolean | true | Enable timeline zoom |
| `playback_rate_control` | boolean | true | Show playback speed selector |
| `frame_stepping` | boolean | true | Enable frame-by-frame navigation |
| `show_timecode` | boolean | true | Display frame number and time |
| `video_fps` | integer | 30 | Video frame rate for calculations |

## Label Configuration

Labels can be simple strings or detailed objects with colors and keyboard shortcuts:

```yaml
labels:
  - name: "intro"
    color: "#4ECDC4"
    key_value: "1"
  - name: "content"
    color: "#3B82F6"
    key_value: "2"
  - name: "outro"
    color: "#8B5CF6"
    key_value: "3"
```

## Examples by Mode

### Segment Mode (Default)

Mark temporal ranges with start and end times:

```yaml
annotation_schemes:
  - name: "scene_detection"
    description: "Mark each scene in the video"
    annotation_type: "video_annotation"
    mode: "segment"
    labels:
      - name: "indoor"
        color: "#3B82F6"
        key_value: "1"
      - name: "outdoor"
        color: "#22C55E"
        key_value: "2"
      - name: "transition"
        color: "#F59E0B"
        key_value: "3"
    zoom_enabled: true
    playback_rate_control: true
```

### Frame Mode

Classify individual frames:

```yaml
annotation_schemes:
  - name: "frame_quality"
    description: "Classify each frame's quality"
    annotation_type: "video_annotation"
    mode: "frame"
    labels:
      - name: "good"
        color: "#22C55E"
      - name: "blurry"
        color: "#F59E0B"
      - name: "occluded"
        color: "#EF4444"
    frame_stepping: true
    show_timecode: true
```

### Keyframe Mode

Mark important moments:

```yaml
annotation_schemes:
  - name: "highlights"
    description: "Mark key moments in the video"
    annotation_type: "video_annotation"
    mode: "keyframe"
    labels:
      - name: "goal"
        color: "#22C55E"
      - name: "foul"
        color: "#EF4444"
      - name: "highlight"
        color: "#F59E0B"
    frame_stepping: true
```

### Tracking Mode

Track objects across frames with bounding boxes:

```yaml
annotation_schemes:
  - name: "object_tracking"
    description: "Track the ball throughout the video"
    annotation_type: "video_annotation"
    mode: "tracking"
    labels:
      - name: "ball"
        color: "#3B82F6"
      - name: "player"
        color: "#22C55E"
    frame_stepping: true
    video_fps: 30
```

### Combined Mode

Use multiple annotation types in one interface:

```yaml
annotation_schemes:
  - name: "comprehensive"
    description: "Full video annotation"
    annotation_type: "video_annotation"
    mode: "combined"
    labels:
      - name: "action"
        color: "#3B82F6"
      - name: "dialogue"
        color: "#22C55E"
      - name: "transition"
        color: "#F59E0B"
    zoom_enabled: true
    playback_rate_control: true
    frame_stepping: true
    show_timecode: true
```

## Video Display

For simple video playback without annotation (e.g., to accompany other annotation types), use the `video` type:

```yaml
annotation_schemes:
  - name: "video_player"
    description: "Watch the video clip"
    annotation_type: "video"
    video_path: "{{video_url}}"
    controls: true
    autoplay: false
    loop: false
    muted: false
```

## Keyboard Shortcuts

| Key | Action |
|-----|--------|
| `Space` | Play/pause |
| `,` / `.` | Previous/next frame |
| `[` | Mark segment start |
| `]` | Mark segment end |
| `Enter` | Create segment |
| `K` | Mark keyframe |
| `C` | Classify current frame |
| `Delete` | Remove selected annotation |
| `1-9` | Select label |
| `+` / `-` | Zoom in/out |

## Data Format

### Input Data

Your data file should include video paths or URLs:

```json
[
  {
    "id": "video_1",
    "video_url": "https://example.com/videos/sample1.mp4",
    "description": "Meeting recording - identify speakers"
  },
  {
    "id": "video_2",
    "video_url": "/data/videos/sample2.mp4",
    "description": "Activity video - label actions"
  }
]
```

### Output Format

Output varies by mode:

**Segment Mode:**
```json
{
  "id": "video_1",
  "annotations": {
    "scene_detection": [
      {
        "start": 0.0,
        "end": 5.2,
        "start_frame": 0,
        "end_frame": 156,
        "label": "indoor"
      }
    ]
  }
}
```

**Frame Mode:**
```json
{
  "id": "video_1",
  "annotations": {
    "frame_quality": [
      {
        "frame": 120,
        "time": 4.0,
        "label": "good"
      }
    ]
  }
}
```

**Keyframe Mode:**
```json
{
  "id": "video_1",
  "annotations": {
    "highlights": [
      {
        "frame": 450,
        "time": 15.0,
        "label": "goal",
        "note": "First goal of the match"
      }
    ]
  }
}
```

**Tracking Mode:**
```json
{
  "id": "video_1",
  "annotations": {
    "object_tracking": [
      {
        "id": "track_1",
        "label": "ball",
        "frames": [
          {"frame": 0, "bbox": {"x": 100, "y": 200, "w": 30, "h": 30}},
          {"frame": 1, "bbox": {"x": 105, "y": 198, "w": 30, "h": 30}}
        ]
      }
    ]
  }
}
```

## Supported Video Formats

- MP4 (recommended)
- WebM
- MOV
- OGG

## Video Object Tracking with Keyframe Interpolation

*New in v2.2.0*

The tracking mode now supports keyframe interpolation, allowing annotators to mark object positions at key frames and have intermediate frames automatically interpolated. This significantly speeds up object tracking tasks.

```yaml
annotation_schemes:
  - name: "tracking"
    description: "Track objects with keyframe interpolation"
    annotation_type: "video_annotation"
    mode: "tracking"
    labels:
      - name: "person"
        color: "#3B82F6"
      - name: "vehicle"
        color: "#22C55E"
    frame_stepping: true
    video_fps: 30
```

### Workflow

1. Navigate to a keyframe and draw a bounding box around the object
2. Skip ahead several frames and reposition the bounding box
3. The system automatically interpolates positions for intermediate frames
4. Review and adjust interpolated positions as needed

The full set of valid annotation modes is: `segment`, `frame`, `keyframe`, `tracking`, and `combined`.

## Technical Notes

- Uses Peaks.js for timeline visualization
- Standard HTML5 video elements (no additional server dependencies)
- Timeline supports drag-to-select for segment creation

## Best Practices

1. **Use compressed videos** - Large files slow down loading
2. **Set appropriate FPS** - Match `video_fps` to your actual video frame rate
3. **Enable frame stepping** - Essential for precise frame-level annotation
4. **Use playback rate control** - Slow motion helps with detailed work
5. **Provide clear segment definitions** - Define what constitutes segment boundaries
6. **Use distinct colors** - Make labels visually distinguishable
7. **Consider timeline height** - Increase for complex segmentation tasks