Showcase/EPIC-KITCHENS Egocentric Action Annotation

advancedimage

EPIC-KITCHENS Egocentric Action Annotation

Annotate fine-grained actions in egocentric kitchen videos with verb-noun pairs. Identify cooking actions from a first-person perspective.

About this dataset

Egocentric action recognition is the task of labeling first-person video, recorded from a head-mounted camera, with the actions the wearer performs, described as verb-noun pairs such as "open fridge" or "cut onion." EPIC-KITCHENS was introduced by Dima Damen and colleagues in "Scaling Egocentric Vision: The EPIC-KITCHENS Dataset" at ECCV 2018. It was created to address the shortage of large first-person datasets capturing how people interact with objects during everyday activity.

The footage was recorded by 32 participants in their own kitchens across four cities in North America and Europe. Recording was unscripted: each person simply started filming whenever they entered the kitchen, capturing natural cooking and cleaning. After recording, the participants narrated their own videos, and those narrations were transcribed and converted into temporally bounded action labels through crowdsourcing.

The dataset provides 55 hours of video totaling 11.5 million frames, densely labeled into 39,596 action segments and 454,255 object bounding boxes. Actions are organized into 125 verb classes and 331 noun classes. Because annotation started from the participants' own narrations, the labels reflect the actor's stated intention rather than an outside observer's guess.

The Potato config below reproduces this task with a video_annotation scheme in segment mode (60 fps, with frame-stepping and timecode display) for marking the temporal boundaries of each action, a radio scheme for the verb, a radio scheme for the noun, free-text fields as fallbacks for out-of-vocabulary verbs and nouns, and a radio scheme rating how visible the action is under hand occlusion. Each clip is shown to two annotators. This is useful for building fine-grained, verb-noun action-recognition datasets from egocentric video.

Participants / kitchens: 32
Video duration: 55 hours (11.5M frames)
Action segments: 39,596
Object bounding boxes: 454,255
Verb / noun classes: 125 verbs / 331 nouns
Venue / Year: ECCV 2018 (EPIC-KITCHENS-55)

Configuration Fileconfig.yaml

This Potato config reproduces the annotation task. Save it as config.yaml and run potato start config.yaml to try it.

yaml

# EPIC-KITCHENS Egocentric Action Annotation Configuration
# Based on Damen et al., ECCV 2018
# Task: Annotate verb-noun action pairs in egocentric kitchen videos

annotation_task_name: "EPIC-KITCHENS Egocentric Action Annotation"
task_dir: "."

data_files:
  - data.json
item_properties:
  id_key: "id"
  text_key: "video_url"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

annotation_schemes:
  - name: "action_segments"
    description: |
      Mark the temporal boundaries of each distinct ACTION.
      An action starts when hands begin moving toward an object
      and ends when the interaction is complete.
    annotation_type: "video_annotation"
    mode: "segment"
    labels:
      - name: "action"
        color: "#22C55E"
        key_value: "a"
    frame_stepping: true
    show_timecode: true
    playback_rate_control: true
    video_fps: 60

  - name: "verb"
    description: "What VERB describes this action?"
    annotation_type: radio
    labels:
      - "take"
      - "put"
      - "open"
      - "close"
      - "wash"
      - "cut"
      - "mix"
      - "pour"
      - "turn-on"
      - "turn-off"
      - "move"
      - "remove"
      - "other"

  - name: "noun"
    description: "What NOUN/OBJECT is being interacted with?"
    annotation_type: radio
    labels:
      - "pan"
      - "plate"
      - "knife"
      - "spoon"
      - "cup"
      - "bowl"
      - "fridge"
      - "tap"
      - "drawer"
      - "cupboard"
      - "food item"
      - "container"
      - "other"

  - name: "verb_free_text"
    description: "If 'other' verb, specify:"
    annotation_type: text

  - name: "noun_free_text"
    description: "If 'other' noun, specify the object:"
    annotation_type: text

  - name: "visibility"
    description: "How visible is the action?"
    annotation_type: radio
    labels:
      - "Fully visible - clear view of hands and object"
      - "Partially visible - some occlusion"
      - "Mostly occluded - hard to see"

allow_all_users: true
instances_per_annotator: 30
annotation_per_instance: 2

annotation_instructions: |
  ## EPIC-KITCHENS Egocentric Action Annotation

  Annotate cooking actions from first-person (egocentric) video.

  ### Task:
  1. Mark the temporal boundaries of each action
  2. Label the VERB (what is being done)
  3. Label the NOUN (what object is involved)

  ### What counts as an action?
  - Any intentional interaction with an object
  - Starts when hands begin reaching/moving
  - Ends when the interaction is complete

  ### Common verb-noun pairs:
  - "take pan", "put plate", "open fridge"
  - "wash spoon", "cut vegetable", "pour water"
  - "turn-on tap", "close drawer", "mix bowl"

  ### Guidelines:
  - One action = one verb + one noun
  - If multiple objects, annotate the PRIMARY one
  - Mark ALL actions, even brief ones
  - Use free text for objects not in the list

  ### Egocentric video tips:
  - Hands often occlude objects - do your best
  - Fast movements may need frame-stepping
  - Camera shake is normal in egocentric video

Sample Datasample-data.json

json

[
  {
    "id": "epic_001",
    "video_url": "https://example.com/videos/kitchen_egocentric_001.mp4",
    "participant": "P01",
    "kitchen": "kitchen_01",
    "duration": 30
  },
  {
    "id": "epic_002",
    "video_url": "https://example.com/videos/kitchen_egocentric_002.mp4",
    "participant": "P01",
    "kitchen": "kitchen_01",
    "duration": 45
  }
]

Get This Design

View on GitHub

Clone or download from the repository

Quick start:

git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/video/action-recognition/epic-kitchens-egocentric
potato start config.yaml

Dataset & paper

Damen et al., ECCV 2018

Official dataset ↗Read the paper ↗

Citation (BibTeX)

bibtex

@inproceedings{damen2018scaling,
    title = {Scaling Egocentric Vision: The EPIC-KITCHENS Dataset},
    author = {Damen, Dima and Doughty, Hazel and Farinella, Giovanni Maria and Fidler, Sanja and Furnari, Antonino and Kazakos, Evangelos and Moltisanti, Davide and Munro, Jonathan and Perrett, Toby and Price, Will and Wray, Michael},
    booktitle = {European Conference on Computer Vision (ECCV)},
    pages = {720--736},
    year = {2018}
}

Details

Annotation Types

radiotextvideo_annotation

Domain

Computer VisionEgocentric VisionActivity Recognition

Use Cases

Action RecognitionVerb-Noun ClassificationKitchen Activities

Related Designs

How2Sign Sign Language Multi-Tier Annotation

Multi-tier ELAN-style annotation of continuous American Sign Language videos. Annotators segment sign glosses, mark mouthing patterns, classify sign handedness, and provide English translations aligned to video timelines. Based on the How2Sign large-scale multimodal ASL dataset.

video_annotationradio

Breakfast Actions Segmentation

Fine-grained temporal action segmentation of breakfast preparation activities. Annotators label sequences of cooking actions like 'take cup', 'pour milk', 'stir'.

textvideo_annotation

Ego4D: Egocentric Video Episodic Memory Annotation

Annotate egocentric (first-person) video for episodic memory tasks including activity segmentation, hand state tracking, natural language query generation, and scene narration. Supports temporal segment annotation with multiple label tiers for the Ego4D benchmark.

video_annotationtext