Robot Episode Annotation
Annotate robot demonstrations with N synchronized video streams and M time-series lanes on one timeline. Phase segmentation, outcome, dense reward, and LeRobot, RLDS and HDF5 import.
A robot demonstration is several synchronized camera streams plus several numeric time series, all indexed by frame, and every question worth asking about one is temporal. When did the grasp start, when did it fail, how well was it going at each moment. So the interface is a timeline: streams above, the arm's own signals as lanes beneath, and every annotation layer on the same time axis.
annotation_schemes:
- annotation_type: episode_annotation
name: episode_review
description: "Mark the phases, judge the outcome, and draw the progress."
source_field: episode
layers: [phases, outcome, reward]
phases:
- {name: reach, color: "#4ECDC4", key_value: "1"}
- {name: grasp, color: "#FFD93D", key_value: "2"}
- {name: transport, color: "#6C8AE4", key_value: "3"}
outcomes: [success, partial, failure]
failure_causes: [missed grasp, object slipped, collision]
reward_range: [0.0, 1.0]
series_shown: [gripper, wrist_force]Runnable example: examples/embodied/lerobot-episode/.
The layers answer different questions
| Layer | Question |
|---|---|
phases | What was the robot doing, and when? |
outcome | Did it work, and if not why? |
reward | How close to done was it at each moment? |
They cost very different amounts of annotator time. Phases are seconds per episode; a dense reward curve is minutes. Most projects enable a subset rather than all three.
Three outcomes, not two
success, partial, failure. Partial is the modal result in real robot data, and forcing it into a binary destroys the signal that makes the dataset worth annotating in the first place.
Instruction relabeling is collected alongside: an episode that failed at the task it was recorded for is often a successful demonstration of some other task, and saying what it actually shows turns a discarded recording into a usable one.
Watch the lanes, not only the video
Lane downsampling is min/max preserving, so a one-frame force spike survives being drawn into a 300-pixel-wide lane instead of being averaged away.
That is not a display nicety. A missed grasp is usually visible in the gripper and wrist-force traces several frames before it becomes obvious on camera, which is why the lanes sit beside the video rather than behind a tab.
Import and export
Imports LeRobot v2 through pyarrow, HDF5 (RoboMimic and ALOHA) through h5py, and RLDS/TFDS through tensorflow_datasets. Potato has no ROS bag reader, so a bag has to be converted to one of these formats first. Exports LeRobot v2 plus a per-frame JSONL sidecar, so a read-only public dataset does not have to be rewritten to carry your annotations.
Agreement
Three measures, because the layers are three different kinds of answer:
- Phase boundaries — temporal IoU
- Outcome labels — Krippendorff's α
- Reward curves — ICC plus Pearson
Each is reported with coverage stated alongside, because a correlation computed over 5% of a timeline says nothing about the other 95%.
Pre-filling the timeline from a trained model
New in v2.14.0
An episode-model run learns the phases, the outcome, and the reward curve from episode_annotation answers. The model reads no video. The model's input is the episode's recorded series, such as joint positions, gripper state, and forces. A phase that differs only on camera is therefore out of the model's reach. An episode marked only with an outcome still trains the outcome, because a layer an answer leaves empty is left out of that episode's loss. When model_training.writeback.schemas names the scheme, the model's answer pre-fills the timeline for the layers the scheme shows. The answer is stored once the annotator edits the answer. Science models covers the run.
Where Potato sits
ATLAS (TU Wien) is a focused desktop tool for long-horizon action segmentation with native ROS bag and RLDS support, and for single-annotator boundary precision it is purpose-built. Its published finding — that showing proprioceptive time series alongside video reduces boundary error against vision-only tools — is why Potato renders those lanes at all. Rerun and Foxglove are the best robotics visualisation tools.
Potato's distinct contribution is being a multi-annotator, web-deployed episode annotator with agreement statistics.
Related
Designs de exemplo
Cada design é uma configuração do Potato funcional que você pode baixar e executar.
- RoboMIND Robot Episode Failure ReviewMark phases, judge success or failure, and name the cause from nine failure categories on robot episodes. The episodes are synthetic and illustrative.
- LeRobot - Robot Episode AnnotationAnnotate teleoperated robot demonstrations: synchronized camera streams and per-frame arm signals on one timeline, with phase segmentation, outcome and failure cause, and a dense progress-reward curve.