Skip to content
intermediatevideo

Scene Boundary Detection

Identify scene boundaries in documentary and narrative videos. Annotators mark transitions between semantically coherent scenes based on visual, audio, and narrative cues.

Get This Design

View on GitHub(opens in a new tab)

Clone or download from the repository

Quick start:

bash
git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/video/boundary-detection/scene-boundary-detection
potato start config.yaml

Configuration File

This Potato config reproduces the annotation task. Save it as config.yaml and run potato start config.yaml to try it.

yaml
# Scene Boundary Detection Configuration
# Based on BBC Planet Earth Scene Dataset (Sidiropoulos et al., 2011)
# Task: Mark scene transitions in documentary videos

annotation_task_name: "Scene Boundary Detection"
task_dir: "."

data_files:
  - data.json
item_properties:
  id_key: "id"
  text_key: "video_url"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

annotation_schemes:
  - name: "scene_boundaries"
    description: |
      Mark the START of each new scene. A scene is a semantically coherent
      segment unified by location, time, characters, or narrative topic.
    annotation_type: "video_annotation"
    mode: "keyframe"
    labels:
      - name: "scene_start"
        color: "#EF4444"
        key_value: "s"
      - name: "gradual_transition"
        color: "#F97316"
        key_value: "g"
      - name: "cut_transition"
        color: "#3B82F6"
        key_value: "c"
    frame_stepping: true
    show_timecode: true
    playback_rate_control: true

  - name: "scene_type"
    description: "What type of scene follows this boundary?"
    annotation_type: radio
    labels:
      - "Establishing shot"
      - "Action/Event"
      - "Dialogue/Interview"
      - "Transition/Montage"
      - "Credits/Title"

allow_all_users: true
instances_per_annotator: 30
annotation_per_instance: 2

annotation_instructions: |
  ## Scene Boundary Detection Task

  Mark where each new SCENE begins in the video.

  ### What defines a scene boundary?
  - Change in location or setting
  - Significant time jump
  - Change in main subject/topic
  - Narrative shift

  ### Transition Types:
  - **Cut**: Instantaneous change between shots
  - **Gradual**: Fade, dissolve, or wipe transition

  ### Tips:
  - A scene is NOT the same as a shot (scenes contain multiple shots)
  - Mark the FIRST frame of the new scene
  - Use frame stepping for precision

Sample Data

json
[
  {
    "id": "scene_001",
    "video_url": "https://example.com/videos/nature_documentary_ep1.mp4",
    "title": "Planet Earth - Mountains Episode",
    "duration_seconds": 600
  },
  {
    "id": "scene_002",
    "video_url": "https://example.com/videos/nature_documentary_ep2.mp4",
    "title": "Planet Earth - Ocean Deep",
    "duration_seconds": 540
  }
]

Dataset & paper

Sidiropoulos et al., IEEE TCSVT 2011

Official dataset(opens in a new tab)Read the paper(opens in a new tab)

Citation (BibTeX)

bibtex
@article{sidiropoulos2011temporal,
  title={Temporal video segmentation to scenes using high-level audiovisual features},
  author={Sidiropoulos, Panagiotis and Mezaris, Vasileios and Kompatsiaris, Ioannis and Meinedo, Hugo and Bugalho, Miguel and Trancoso, Isabel},
  journal={IEEE Transactions on Circuits and Systems for Video Technology},
  volume={21},
  number={8},
  pages={1163--1177},
  year={2011}
}

Details

Annotation Types

radiovideo_annotation

Domain

Computer VisionVideo Understanding

Use Cases

Scene DetectionVideo SegmentationContent Analysis

Tags

videosceneboundarysegmentationdocumentary

Found an issue or want to improve this design?

Open an Issue(opens in a new tab)

Related Designs

DiDeMo Moment Retrieval

Localizing natural language descriptions to specific video moments. Given a text query, annotators identify the corresponding temporal segment in the video.

radiovideo_annotation

FineSports: Fine-grained Basketball Action Recognition

Fine-grained NBA basketball action recognition. Annotators track the ball-handler across a clip and label a two-level hierarchy of 12 coarse ball-handler actions and 52 fine-grained sub-actions (Xu et al., CVPR 2024).

video_annotationradio

VSTAR Video-grounded Dialogue

Video-grounded dialogue annotation. Annotators watch videos and answer questions requiring situated understanding, write dialogue turns grounded in specific video moments, and mark relevant temporal segments.

video_annotationtext