How to Annotate Robot Episodes
Segment a robot demonstration into phases, judge the outcome, and draw a dense reward curve. Why partial is a real outcome and why the time-series lanes matter more than the video.
A robot demonstration is several synchronized camera streams plus several numeric time series, and every question worth asking about one is temporal. When did the grasp start, when did it fail, how close to done was it at each moment. So the interface has to be a timeline, not a form.
Put the signals next to the video
Camera views above, the arm's own signals as lanes beneath, one shared time axis, and every annotation layer drawn onto it.
This is not a layout preference. A missed grasp is usually visible in the gripper and wrist-force traces several frames before it is obvious on camera. Published work supports this: ATLAS (TU Wien) reports that showing proprioceptive time series alongside video reduces boundary error against vision-only tools.
Two practical consequences:
- Do not hide the lanes behind a tab. If the annotator has to go looking, they will not.
- Downsample min/max-preserving. A one-frame force spike drawn into a 300-pixel-wide lane disappears entirely under mean downsampling, and that spike is often the event.
Segment by sub-goal
A phase boundary is where the robot's objective changes — reach, grasp, transport, place, retract — not where the video cuts or where the arm happens to pause.
Phase taxonomies are task-family specific. Manipulation and navigation need entirely different sets, so this belongs in your config rather than in the tool:
phases:
- {name: reach, color: "#4ECDC4", key_value: "1"}
- {name: grasp, color: "#FFD93D", key_value: "2"}
- {name: transport, color: "#6C8AE4", key_value: "3"}
- {name: place, color: "#95E1A3", key_value: "4"}
- {name: retract, color: "#C9A0DC", key_value: "5"}Three outcomes, not two
success, partial, failure.
Partial is the modal result in real robot data. Forcing it into a binary destroys the signal that makes the dataset worth annotating: an episode that got the object off the table but dropped it is not the same as one that never moved, and a policy trained as if they were the same learns the wrong thing.
Pair it with a failure-cause taxonomy — missed grasp, object slipped, collision, wrong object, wrong placement, timeout — so the failures are analysable rather than just counted.
Hindsight relabelling
An episode that failed at the task it was recorded for is frequently a successful demonstration of some other task. Collect what the episode actually shows, and a discarded recording becomes usable training data.
This is cheap to collect at annotation time and impossible to recover later, so ask for it even if you are not sure you will use it.
The reward layer is expensive
A dense progress-reward curve is minutes per episode where phases are seconds. Enable it only when you need it, and expect it to dominate your annotation budget.
Measuring agreement
Three layers, three measures:
| Layer | Measure |
|---|---|
| Phase boundaries | Temporal IoU |
| Outcome labels | Krippendorff's α |
| Reward curves | ICC plus Pearson |
State coverage alongside each. A correlation computed over 5% of a timeline says nothing about the other 95%, and a coefficient without its coverage will be read as though it covered everything.
For boundaries specifically, report a tolerance sweep rather than one threshold.
Formats
LeRobot v2, RLDS/TFDS, HDF5 (RoboMimic and ALOHA), and ROS bags all import. Export a per-frame JSONL sidecar rather than rewriting the dataset, so a read-only public release can carry your annotations without being republished.
Config
annotation_schemes:
- annotation_type: episode_annotation
name: episode_review
description: "Mark the phases, judge the outcome, and draw the progress."
source_field: episode
layers: [phases, outcome, reward]
outcomes: [success, partial, failure]
failure_causes: [missed grasp, object slipped, collision, timeout]
reward_range: [0.0, 1.0]
series_shown: [gripper, wrist_force]Working example: the LeRobot showcase design.
Related tools
ATLAS is purpose-built for single-annotator boundary precision with ROS bags. ELAN and BORIS remain the standards for behavioural coding. Rerun and Foxglove are the best robotics visualisation tools and now carry structured annotations.
Potato's distinct contribution here is being multi-annotator and web-deployed with agreement statistics.