Skip to content
Showcase/ActivityNet Captions: Dense Video Captioning Dataset
advancedvideo

ActivityNet Captions: Dense Video Captioning Dataset

ActivityNet Captions pairs 20k untrimmed videos with 100k temporally localized sentence descriptions for dense-captioning research. This Potato config reproduces the segment-and-describe workflow.

About this dataset

ActivityNet Captions is a dense video captioning dataset introduced by Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles at ICCV 2017. It extends the ActivityNet video collection with natural-language descriptions tied to specific moments in each clip.

The dataset covers about 20k untrimmed YouTube videos totaling 849 video hours, paired with 100k sentence descriptions. Each video carries 3.65 sentences on average, and each sentence runs about 13.48 words.

The dense-captioning task asks an annotator to split a video into events and write a description for each one, with a start and end time attached. On average a sentence describes 36 seconds, or 31 percent of its video, and the marked segments overlap, so a single moment can belong to several captions.

The Potato config below reproduces this segment-and-describe workflow: an annotator watches a video, marks temporal start and end times for each event, and writes a free-text caption for the marked span.

Videos
~20,000 untrimmed
Sentence descriptions
~100,000
Total video hours
849
Sentences per video
3.65 average
Words per sentence
13.48 average
Coverage per sentence
~36s / 31% of video
Frame 847 / 3200Running01:12 - 01:28Segments:WalkRunStandActionWalkRunStandWalkSceneOutdoorIndoorDrag to create and label temporal segments

Configuration Fileconfig.yaml

This Potato config reproduces the annotation task. Save it as config.yaml and run potato start config.yaml to try it.

yaml
# ActivityNet Captions Dense Annotation Configuration
# Based on Krishna et al., ICCV 2017
# Task: Segment videos and write captions for each event

annotation_task_name: "ActivityNet Dense Captioning"
task_dir: "."

data_files:
  - data.json
item_properties:
  id_key: "id"
  text_key: "video_url"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

annotation_schemes:
  - name: "event_segments"
    description: |
      Mark temporal segments for each distinct event in the video.
      Events should be semantically meaningful and non-overlapping.
    annotation_type: "video_annotation"
    mode: "segment"
    labels:
      - name: "event"
        color: "#3B82F6"
        key_value: "e"
    zoom_enabled: true
    playback_rate_control: true
    frame_stepping: true
    show_timecode: true
    timeline_height: 80

  - name: "event_caption"
    description: |
      Write a natural language description of the event you just marked.
      Be specific about WHO does WHAT. Start with a verb.
    annotation_type: text
    min_length: 10
    max_length: 200
    placeholder: "e.g., 'A man in a red shirt kicks a soccer ball into the goal'"

allow_all_users: true
instances_per_annotator: 30
annotation_per_instance: 2

annotation_instructions: |
  ## Dense Video Captioning Task

  Your goal is to segment the video into events and describe each one.

  ### Step 1: Identify Events
  - Watch the video and identify distinct events
  - Mark the START and END of each event
  - Events should be meaningful actions/happenings

  ### Step 2: Write Captions
  - Describe WHAT happens in each segment
  - Be specific: mention people, objects, actions
  - Start with a verb (e.g., "A woman picks up...")
  - Keep it concise but complete

  ### Caption Guidelines:
  - Describe visible actions, not intentions
  - Include relevant details (clothing, objects, location)
  - Use present tense
  - Don't describe audio unless relevant

  ### Example Captions:
  - "A chef chops vegetables on a cutting board"
  - "Two children run across a playground"
  - "The camera pans across a mountain landscape"

Sample Datasample-data.json

json
[
  {
    "id": "anetcap_001",
    "video_url": "https://example.com/videos/cooking_video.mp4",
    "duration_seconds": 300,
    "category": "Cooking"
  },
  {
    "id": "anetcap_002",
    "video_url": "https://example.com/videos/sports_clip.mp4",
    "duration_seconds": 180,
    "category": "Sports"
  }
]

Get This Design

View on GitHub

Clone or download from the repository

Quick start:

git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/video/temporal-grounding/activitynet-captions
potato start config.yaml

Dataset & paper

Krishna et al., ICCV 2017

Citation (BibTeX)

bibtex
@inproceedings{krishna2017dense,
  title={Dense-captioning events in videos},
  author={Krishna, Ranjay and Hata, Kenji and Ren, Frederic and Fei-Fei, Li and Niebles, Juan Carlos},
  booktitle={Proceedings of the IEEE International Conference on Computer Vision},
  pages={706--715},
  year={2017}
}

Details

Annotation Types

video_annotationtext

Domain

Computer VisionVideo UnderstandingNLP

Use Cases

Dense CaptioningVideo DescriptionTemporal Grounding

Tags

videocaptionsdensetemporalactivitynetdescription

Found an issue or want to improve this design?

Open an Issue