Robot Episode Annotation
Annotate robot demonstrations with N synchronized video streams and M time-series lanes on one timeline. Phase segmentation, outcome, dense reward, and LeRobot, RLDS and HDF5 import.
A robot demonstration is several synchronized camera streams plus several numeric time series, all indexed by frame, and every question worth asking about one is temporal. When did the grasp start, when did it fail, how well was it going at each moment. So the interface is a timeline: streams above, the arm's own signals as lanes beneath, and every annotation layer on the same time axis.
annotation_schemes:
- annotation_type: episode_annotation
name: episode_review
description: "Mark the phases, judge the outcome, and draw the progress."
source_field: episode
layers: [phases, outcome, reward]
phases:
- {name: reach, color: "#4ECDC4", key_value: "1"}
- {name: grasp, color: "#FFD93D", key_value: "2"}
- {name: transport, color: "#6C8AE4", key_value: "3"}
outcomes: [success, partial, failure]
failure_causes: [missed grasp, object slipped, collision]
reward_range: [0.0, 1.0]
series_shown: [gripper, wrist_force]Runnable example: examples/embodied/lerobot-episode/.
The layers answer different questions
| Layer | Question |
|---|---|
phases | What was the robot doing, and when? |
outcome | Did it work, and if not why? |
reward | How close to done was it at each moment? |
They cost very different amounts of annotator time. Phases are seconds per episode; a dense reward curve is minutes. Most projects enable a subset rather than all three.
Three outcomes, not two
success, partial, failure. Partial is the modal result in real robot data, and forcing it into a binary destroys the signal that makes the dataset worth annotating in the first place.
Instruction relabelling is collected alongside: an episode that failed at the task it was recorded for is often a successful demonstration of some other task, and saying what it actually shows turns a discarded recording into a usable one.
Watch the lanes, not only the video
Lane downsampling is min/max preserving, so a one-frame force spike survives being drawn into a 300-pixel-wide lane instead of being averaged away.
That is not a display nicety. A missed grasp is usually visible in the gripper and wrist-force traces several frames before it becomes obvious on camera, which is why the lanes sit beside the video rather than behind a tab.
Import and export
Imports LeRobot v2 through pyarrow, HDF5 (RoboMimic and ALOHA) through h5py, RLDS/TFDS through tensorflow_datasets, and ROS bags. Exports LeRobot v2 plus a per-frame JSONL sidecar, so a read-only public dataset does not have to be rewritten to carry your annotations.
Agreement
Three measures, because the layers are three different kinds of answer:
- Phase boundaries — temporal IoU
- Outcome labels — Krippendorff's α
- Reward curves — ICC plus Pearson
Each is reported with coverage stated alongside, because a correlation computed over 5% of a timeline says nothing about the other 95%.
Where Potato sits
ATLAS (TU Wien) is a focused desktop tool for long-horizon action segmentation with native ROS bag and RLDS support, and for single-annotator boundary precision it is purpose-built. Its published finding — that showing proprioceptive time series alongside video reduces boundary error against vision-only tools — is why Potato renders those lanes at all. Rerun and Foxglove are the best robotics visualisation tools.
Potato's distinct contribution is being a multi-annotator, web-deployed episode annotator with agreement statistics.