Skip to content
intermediateimage

MovieNet: Holistic Movie Understanding Dataset

MovieNet is a 1,100-movie dataset from Huang et al. (ECCV 2020) with scene, place, action, and cinematic-style annotations. This Potato config reproduces its scene classification labeling task.

About this dataset

MovieNet is a multi-modal dataset for holistic movie understanding, built by Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin from CUHK and SenseTime and published at the European Conference on Computer Vision (ECCV) 2020.

The dataset covers 1,100 movies along with their trailers, photos, and plot descriptions. Its annotation layers include 1.1M character bounding boxes with identities, 42K scene boundaries, 2.5K aligned description sentences, 65K place and action tags, and 92K cinematic style tags.

Scene-level annotation drives the place and action layers: each scene segment receives multiple place tags drawn from 90 place classes, and sub-clips receive multiple action tags from 80 action classes, both filtered to a minimum frequency of 25. Cinematic style tags cover view scale and camera movement. These layers support tasks such as scene segmentation, place recognition, and action analysis.

The Potato config below reproduces the scene classification task: annotators view a scene and assign place tags, time of day, and cinematic attributes using multiselect and radio schemes, mirroring the multi-label place and style annotation in MovieNet.

Movies
1,100
Scene boundaries
42,000
Place + action tags
65,000
Cinematic style tags
92,000
Character boxes + identities
1.1M
Place / action classes
90 / 80
Labels:outdoornatureurbanpeopleanimal+−

Configuration Fileconfig.yaml

This Potato config reproduces the annotation task. Save it as config.yaml and run potato start config.yaml to try it.

yaml
# MovieNet Scene Classification Configuration
# Based on Huang et al., ECCV 2020
# Task: Classify movie scenes by various attributes

annotation_task_name: "MovieNet Scene Classification"
task_dir: "."

data_files:
  - sample-data.json
item_properties:
  id_key: "id"
  text_key: "video_url"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

annotation_schemes:
  - name: "place_type"
    description: "What type of place/location is this scene set in?"
    annotation_type: radio
    labels:
      - "Indoor - Home/Apartment"
      - "Indoor - Office/Workplace"
      - "Indoor - Public (restaurant, store, etc.)"
      - "Indoor - Vehicle"
      - "Outdoor - Urban/City"
      - "Outdoor - Nature/Rural"
      - "Outdoor - Road/Street"
      - "Mixed/Transitional"

  - name: "time_of_day"
    description: "What time of day is depicted?"
    annotation_type: radio
    labels:
      - "Day - Morning"
      - "Day - Afternoon"
      - "Day - Evening/Dusk"
      - "Night"
      - "Unclear/Mixed"

  - name: "weather"
    description: "What is the weather/lighting condition?"
    annotation_type: radio
    labels:
      - "Clear/Sunny"
      - "Cloudy/Overcast"
      - "Rainy"
      - "Snowy"
      - "Foggy/Misty"
      - "Indoor (N/A)"
      - "Unclear"

  - name: "scene_function"
    description: "What narrative function does this scene serve?"
    annotation_type: multiselect
    labels:
      - "Exposition (introduces information)"
      - "Action/Chase"
      - "Dialogue/Conversation"
      - "Emotional/Dramatic"
      - "Comedic"
      - "Romantic"
      - "Suspense/Tension"
      - "Flashback/Dream"
      - "Montage/Transition"

  - name: "shot_scale"
    description: "What is the predominant shot scale?"
    annotation_type: radio
    labels:
      - "Extreme Close-up"
      - "Close-up"
      - "Medium Shot"
      - "Full Shot"
      - "Long Shot"
      - "Extreme Long Shot"
      - "Mixed/Varied"

allow_all_users: true
instances_per_annotator: 50
annotation_per_instance: 2

annotation_instructions: |
  ## Movie Scene Classification Task

  Classify movie scenes by their visual and narrative attributes.

  ### Attributes to Label:

  **Place Type**: Where does the scene take place?

  **Time of Day**: When does the scene occur?

  **Weather**: What's the lighting/weather condition?

  **Scene Function**: What purpose does this scene serve narratively?

  **Shot Scale**: What's the typical camera distance?

  ### Tips:
  - Consider the OVERALL scene, not just one shot
  - Scene function can have multiple labels
  - If unsure, choose the most prominent/frequent option

Sample Datasample-data.json

json
[
  {
    "id": "movienet_001",
    "video_url": "https://example.com/videos/movie_scene_office.mp4",
    "movie": "Sample Movie",
    "scene_number": 1
  },
  {
    "id": "movienet_002",
    "video_url": "https://example.com/videos/movie_scene_outdoor.mp4",
    "movie": "Sample Movie",
    "scene_number": 2
  }
]

Get This Design

View on GitHub

Clone or download from the repository

Quick start:

git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/video/scene-understanding/movienet-scene-classification
potato start config.yaml

Dataset & paper

Huang et al., ECCV 2020

Official dataset ↗Read the paper ↗

Citation (BibTeX)

bibtex
@inproceedings{huang2020movienet,
    title = "MovieNet: A Holistic Dataset for Movie Understanding",
    author = "Huang, Qingqiu and Xiong, Yu and Rao, Anyi and Wang, Jiaze and Lin, Dahua",
    booktitle = "Proceedings of the European Conference on Computer Vision (ECCV)",
    pages = "709--727",
    year = "2020"
}

Details

Annotation Types

multiselectradio

Domain

Computer VisionFilm Studies

Use Cases

Scene ClassificationMovie UnderstandingContent Analysis

Tags

videomoviesceneclassificationmovienetcinematic

Found an issue or want to improve this design?

Open an Issue

Related Designs

ADE20K: Scene Parsing and Semantic Segmentation Dataset

ADE20K is a scene parsing dataset from MIT CSAIL (Zhou et al., CVPR 2017) with dense pixel labels for objects, stuff, and parts. This Potato config adapts it to an image-level category-tagging task over a curated subset of common ADE20K categories.

multiselectradio

BDD100K: Diverse Driving Dataset for Autonomous Driving

BDD100K is a 100,000-video driving dataset from UC Berkeley (Yu et al., CVPR 2020) spanning detection, segmentation, lane marking, and tracking. This Potato config reproduces its driving-scene labeling tasks.

multiselectradio

CUB-200-2011: Fine-Grained Bird Species Dataset

CUB-200-2011 is a fine-grained bird image dataset of 11,788 photos across 200 species, each labeled with parts, attributes, and a bounding box (Wah et al., 2011). This Potato config reproduces the labeling task.

multiselectradio