ActivityNet Captions: Dense Video Captioning Dataset
ActivityNet Captions pairs 20k untrimmed videos with 100k temporally localized sentence descriptions for dense-captioning research. This Potato config reproduces the segment-and-describe workflow.
About this dataset
ActivityNet Captions is a dense video captioning dataset introduced by Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles at ICCV 2017. It extends the ActivityNet video collection with natural-language descriptions tied to specific moments in each clip.
The dataset covers about 20k untrimmed YouTube videos totaling 849 video hours, paired with 100k sentence descriptions. Each video carries 3.65 sentences on average, and each sentence runs about 13.48 words.
The dense-captioning task asks an annotator to split a video into events and write a description for each one, with a start and end time attached. On average a sentence describes 36 seconds, or 31 percent of its video, and the marked segments overlap, so a single moment can belong to several captions.
The Potato config below reproduces this segment-and-describe workflow: an annotator watches a video, marks temporal start and end times for each event, and writes a free-text caption for the marked span.
- Videos
- ~20,000 untrimmed
- Sentence descriptions
- ~100,000
- Total video hours
- 849
- Sentences per video
- 3.65 average
- Words per sentence
- 13.48 average
- Coverage per sentence
- ~36s / 31% of video
Configuration Fileconfig.yaml
This Potato config reproduces the annotation task. Save it as config.yaml and run potato start config.yaml to try it.
# ActivityNet Captions Dense Annotation Configuration
# Based on Krishna et al., ICCV 2017
# Task: Segment videos and write captions for each event
annotation_task_name: "ActivityNet Dense Captioning"
task_dir: "."
data_files:
- data.json
item_properties:
id_key: "id"
text_key: "video_url"
output_annotation_dir: "annotation_output/"
output_annotation_format: "json"
annotation_schemes:
- name: "event_segments"
description: |
Mark temporal segments for each distinct event in the video.
Events should be semantically meaningful and non-overlapping.
annotation_type: "video_annotation"
mode: "segment"
labels:
- name: "event"
color: "#3B82F6"
key_value: "e"
zoom_enabled: true
playback_rate_control: true
frame_stepping: true
show_timecode: true
timeline_height: 80
- name: "event_caption"
description: |
Write a natural language description of the event you just marked.
Be specific about WHO does WHAT. Start with a verb.
annotation_type: text
min_length: 10
max_length: 200
placeholder: "e.g., 'A man in a red shirt kicks a soccer ball into the goal'"
allow_all_users: true
instances_per_annotator: 30
annotation_per_instance: 2
annotation_instructions: |
## Dense Video Captioning Task
Your goal is to segment the video into events and describe each one.
### Step 1: Identify Events
- Watch the video and identify distinct events
- Mark the START and END of each event
- Events should be meaningful actions/happenings
### Step 2: Write Captions
- Describe WHAT happens in each segment
- Be specific: mention people, objects, actions
- Start with a verb (e.g., "A woman picks up...")
- Keep it concise but complete
### Caption Guidelines:
- Describe visible actions, not intentions
- Include relevant details (clothing, objects, location)
- Use present tense
- Don't describe audio unless relevant
### Example Captions:
- "A chef chops vegetables on a cutting board"
- "Two children run across a playground"
- "The camera pans across a mountain landscape"
Sample Datasample-data.json
[
{
"id": "anetcap_001",
"video_url": "https://example.com/videos/cooking_video.mp4",
"duration_seconds": 300,
"category": "Cooking"
},
{
"id": "anetcap_002",
"video_url": "https://example.com/videos/sports_clip.mp4",
"duration_seconds": 180,
"category": "Sports"
}
]Get This Design
Clone or download from the repository
Quick start:
git clone https://github.com/davidjurgens/potato-showcase.git cd potato-showcase/video/temporal-grounding/activitynet-captions potato start config.yaml
Dataset & paper
Krishna et al., ICCV 2017
Citation (BibTeX)
@inproceedings{krishna2017dense,
title={Dense-captioning events in videos},
author={Krishna, Ranjay and Hata, Kenji and Ren, Frederic and Fei-Fei, Li and Niebles, Juan Carlos},
booktitle={Proceedings of the IEEE International Conference on Computer Vision},
pages={706--715},
year={2017}
}Details
Annotation Types
Domain
Use Cases
Tags
Found an issue or want to improve this design?
Open an IssueRelated Designs
FineDiving: Procedure-aware Action Quality Assessment
Fine-grained diving action quality assessment. Annotators mark a dive and its consecutive sub-action steps on the timeline, then record the dive type, difficulty degree, and scores (Xu et al., CVPR 2022).
Fis-V: Figure Skating Video Scoring
Automatic figure skating scoring as video regression. Annotators mark the technical elements and record the Total Element Score (TES) and Total Program Component Score (PCS) for a ladies' short program (Xu et al., IEEE TCSVT 2020).
NExT-QA: Causal and Temporal Video QA Benchmark
NExT-QA (Xiao et al., CVPR 2021) is a video QA benchmark for causal and temporal action reasoning over 5,440 videos and ~52K questions. This Potato config reproduces its multiple-choice answer task.