# Robot Episode Annotation

Source: https://www.potatoannotator.com/docs/vision-spatial/robot-episodes

**A robot demonstration is several synchronized camera streams plus several numeric time series, all indexed by frame, and every question worth asking about one is temporal.** When did the grasp start, when did it fail, how well was it going at each moment. So the interface is a timeline: streams above, the arm's own signals as lanes beneath, and every annotation layer on the same time axis.

```yaml
annotation_schemes:
  - annotation_type: episode_annotation
    name: episode_review
    description: "Mark the phases, judge the outcome, and draw the progress."
    source_field: episode
    layers: [phases, outcome, reward]
    phases:
      - {name: reach,     color: "#4ECDC4", key_value: "1"}
      - {name: grasp,     color: "#FFD93D", key_value: "2"}
      - {name: transport, color: "#6C8AE4", key_value: "3"}
    outcomes: [success, partial, failure]
    failure_causes: [missed grasp, object slipped, collision]
    reward_range: [0.0, 1.0]
    series_shown: [gripper, wrist_force]
```

Runnable example: [`examples/embodied/lerobot-episode/`](https://github.com/davidjurgens/potato/tree/main/examples/embodied/lerobot-episode).

## The layers answer different questions

| Layer | Question |
|---|---|
| `phases` | What was the robot doing, and when? |
| `outcome` | Did it work, and if not why? |
| `reward` | How close to done was it at each moment? |

They cost very different amounts of annotator time. Phases are seconds per episode; a dense reward curve is minutes. Most projects enable a subset rather than all three.

## Three outcomes, not two

`success`, `partial`, `failure`. **Partial is the modal result in real robot data**, and forcing it into a binary destroys the signal that makes the dataset worth annotating in the first place.

Instruction relabelling is collected alongside: an episode that failed at the task it was recorded for is often a successful demonstration of some other task, and saying what it actually shows turns a discarded recording into a usable one.

## Watch the lanes, not only the video

Lane downsampling is **min/max preserving**, so a one-frame force spike survives being drawn into a 300-pixel-wide lane instead of being averaged away.

That is not a display nicety. A missed grasp is usually visible in the gripper and wrist-force traces several frames before it becomes obvious on camera, which is why the lanes sit beside the video rather than behind a tab.

## Import and export

Imports [LeRobot](https://github.com/huggingface/lerobot) v2 through pyarrow, HDF5 (RoboMimic and ALOHA) through h5py, [RLDS/TFDS](https://github.com/google-research/rlds) through tensorflow_datasets, and ROS bags. Exports LeRobot v2 plus a per-frame JSONL sidecar, so a read-only public dataset does not have to be rewritten to carry your annotations.

## Agreement

Three measures, because the layers are three different kinds of answer:

- **Phase boundaries** — temporal IoU
- **Outcome labels** — Krippendorff's α
- **Reward curves** — ICC plus Pearson

Each is reported **with coverage stated alongside**, because a correlation computed over 5% of a timeline says nothing about the other 95%.

## Where Potato sits

[ATLAS](https://arxiv.org/abs/2503.03539) (TU Wien) is a focused desktop tool for long-horizon action segmentation with native ROS bag and RLDS support, and for single-annotator boundary precision it is purpose-built. Its published finding — that showing proprioceptive time series alongside video reduces boundary error against vision-only tools — is why Potato renders those lanes at all. [Rerun](https://rerun.io/) and [Foxglove](https://foxglove.dev/) are the best robotics visualisation tools.

Potato's distinct contribution is being a multi-annotator, web-deployed episode annotator with agreement statistics.

## Related

- [World-model evaluation](/docs/vision-spatial/world-model-evaluation)
- [Temporal agreement](/docs/measurement/temporal-agreement)
- [Guide: robot episode annotation](/docs/guides/robot-episode-annotation)
- [Source documentation](https://github.com/davidjurgens/potato/blob/main/docs/annotation-types/embodied/episodes.md)