Skip to content
intermediatevideo

SumMe Video Summarization

Create video summaries by selecting key segments that best represent the content. Annotators identify important moments for automatic video summarization research.

Get This Design

View on GitHub(opens in a new tab)

Clone or download from the repository

Quick start:

bash
git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/video/summarization/summe-summarization
potato start config.yaml

Configuration File

This Potato config reproduces the annotation task. Save it as config.yaml and run potato start config.yaml to try it.

yaml
# SumMe Video Summarization Configuration
# Based on Gygli et al., ECCV 2014
# Task: Select important segments for video summarization

annotation_task_name: "SumMe Video Summarization"
task_dir: "."

data_files:
  - data.json
item_properties:
  id_key: "id"
  text_key: "video_url"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

annotation_schemes:
  - name: "summary_segments"
    description: |
      Mark segments that should be INCLUDED in a summary of this video.
      Select the most important/interesting parts that capture the essence.
    annotation_type: "video_annotation"
    mode: "segment"
    labels:
      - name: "include_in_summary"
        color: "#22C55E"
        key_value: "s"
      - name: "highly_important"
        color: "#EF4444"
        key_value: "h"
    frame_stepping: true
    show_timecode: true
    playback_rate_control: true
    video_fps: 30

  - name: "importance_score"
    description: "Overall, how interesting/important is this video's content?"
    annotation_type: radio
    labels:
      - "5 - Very interesting/important"
      - "4 - Interesting"
      - "3 - Moderately interesting"
      - "2 - Somewhat boring"
      - "1 - Not interesting"

  - name: "video_category"
    description: "What category best describes this video?"
    annotation_type: radio
    labels:
      - "Sports/Action"
      - "Travel/Scenery"
      - "Events/Celebrations"
      - "Animals/Nature"
      - "Tutorial/How-to"
      - "Social/People"
      - "Other"

  - name: "summary_difficulty"
    description: "How difficult was it to select summary segments?"
    annotation_type: radio
    labels:
      - "Easy - clear highlights"
      - "Moderate - some judgment needed"
      - "Difficult - many equally important parts"
      - "Very difficult - no clear structure"

allow_all_users: true
instances_per_annotator: 25
annotation_per_instance: 3

annotation_instructions: |
  ## Video Summarization Task

  Select segments that should be included in a summary of each video.

  ### Goal:
  If the video were shortened to ~15% of its length, what parts should remain?

  ### What makes a good summary segment?
  - Captures key moments or highlights
  - Shows the main subject/action clearly
  - Is visually interesting or informative
  - Would make sense to someone who hasn't seen the full video

  ### Guidelines:
  - Mark all segments you think should be in the summary
  - Use "highly_important" for the absolute best moments
  - Try to select 10-20% of the video
  - Avoid: repetitive content, blurry/unclear footage, transitions

  ### Tips:
  - Watch the whole video first before marking
  - Consider what would be lost if a segment were removed
  - For action videos, focus on peak moments
  - For scenic videos, focus on the best views

Sample Data

json
[
  {
    "id": "summe_001",
    "video_url": "https://example.com/videos/user_video_cooking.mp4",
    "category": "cooking",
    "duration": 180
  },
  {
    "id": "summe_002",
    "video_url": "https://example.com/videos/user_video_travel.mp4",
    "category": "travel",
    "duration": 240
  }
]

Dataset & paper

Gygli et al., ECCV 2014

Official dataset(opens in a new tab)Read the paper(opens in a new tab)

Citation (BibTeX)

bibtex
@inproceedings{gygli2014creating,
  title={Creating summaries from user videos},
  author={Gygli, Michael and Grabner, Helmut and Riemenschneider, Hayko and Van Gool, Luc},
  booktitle={European Conference on Computer Vision},
  pages={505--520},
  year={2014}
}

Details

Annotation Types

radiovideo_annotation

Domain

Computer VisionVideo Summarization

Use Cases

Video SummarizationHighlight DetectionImportance Scoring

Tags

videosummarizationhighlightsimportanceuser-generated

Found an issue or want to improve this design?

Open an Issue(opens in a new tab)

Related Designs

Charades-STA Temporal Grounding

Ground natural language descriptions to video segments. Given a sentence describing an action, identify the exact temporal boundaries where that action occurs.

radiovideo_annotation

DiDeMo Moment Retrieval

Localizing natural language descriptions to specific video moments. Given a text query, annotators identify the corresponding temporal segment in the video.

radiovideo_annotation

FineSports: Fine-grained Basketball Action Recognition

Fine-grained NBA basketball action recognition. Annotators track the ball-handler across a clip and label a two-level hierarchy of 12 coarse ball-handler actions and 52 fine-grained sub-actions (Xu et al., CVPR 2024).

video_annotationradio