# Partial Is a Real Outcome: Annotating Robot Episodes

Source: https://www.potatoannotator.com/blog/partial-is-a-real-outcome

A robot demonstration is several synchronized camera streams plus several numeric time series — joint positions, gripper state, force-torque, reward — all indexed by frame. Every question worth asking about one is temporal: when did the grasp start, when did it fail, how close to done was it at each moment.

So the interface has to be a timeline rather than a form. Here is what that turned out to require.

## Partial is the modal result

The first design decision was the outcome vocabulary, and the obvious answer is wrong.

Success or failure is a natural binary and it destroys the signal. In real robot data, **partial success is the modal outcome**: the arm got the object off the table and dropped it; it placed the block in the right bin but the wrong compartment; it completed the task two seconds after the timeout.

Forcing those into "failure" makes them indistinguishable from an arm that never moved. Forcing them into "success" is worse. A policy trained on either mapping learns something untrue about what happened.

Three outcomes, with a failure-cause taxonomy attached — missed grasp, object slipped, collision, wrong object, wrong placement, timeout — makes the failures analysable instead of merely counted.

## Put the signals next to the video, not behind a tab

This one has published support. [ATLAS](https://arxiv.org/abs/2503.03539) (TU Wien, 2026) reports that showing proprioceptive time series alongside video **reduces boundary error** against vision-only tools. That finding is why Potato renders those lanes at all.

It matches what you see immediately on real episodes: a missed grasp is visible in the gripper and wrist-force traces several frames before it becomes obvious on camera. The gripper closes, the force never rises, and only later does the video show an empty hand.

If the lanes are behind a tab, annotators do not open them.

## Downsample min/max, not mean

A 30-second episode at 30 Hz is 900 samples per series, drawn into a lane maybe 300 pixels wide. Something has to be dropped.

Mean downsampling drops exactly the wrong thing. A **one-frame force spike is the event**, and averaging it with its neighbours removes it entirely. The lane looks smooth and the moment of contact is gone.

Min/max-preserving downsampling keeps both extremes in each bucket, so a single-frame spike survives to the pixel. This is a small implementation detail that determines whether the lane is useful at all.

## Layers cost very different amounts

Three annotation layers, and their costs are not comparable:

| Layer | Cost per episode |
|---|---|
| Phase segmentation | Seconds |
| Outcome plus cause | Seconds |
| Dense progress reward | Minutes |

So they are individually configurable, and most projects enable a subset. Shipping all three as one non-negotiable form would have made the reward curve the reason nobody finished a study.

## Hindsight relabelling turns failures into data

An episode that failed at the task it was recorded for is frequently a **successful demonstration of some other task**. The arm meant to place the block in the left bin and put it in the right one: that is a failure of the recorded instruction and a clean success of a different one.

Collecting what the episode *actually* demonstrates costs one text field at annotation time and is impossible to recover later. Ask for it even if you are not sure you will use it.

## Three questions, three agreement measures

The layers are three different kinds of answer, so they need three measures:

- **Phase boundaries** — temporal IoU
- **Outcome labels** — Krippendorff's α
- **Reward curves** — ICC plus Pearson

Each is reported **with coverage stated alongside**. A correlation computed over 5% of a timeline says nothing about the other 95%, and a coefficient without its coverage will be read as though it covered everything.

For boundaries specifically, the tolerance matters: agreement at 0.25 s and agreement at 2 s are different claims, so the report sweeps the tolerance rather than picking one.

## Formats

LeRobot v2, RLDS/TFDS, HDF5 (RoboMimic and ALOHA), and ROS bags all import through the same field. Export is LeRobot v2 plus a per-frame JSONL sidecar, so a read-only public dataset can carry your annotations without being republished.

## Where Potato is not the answer

[ATLAS](https://arxiv.org/abs/2503.03539) is purpose-built for single-annotator boundary precision with ROS bags, and is better at it. [ELAN](https://archive.mpi.nl/tla/elan) and BORIS remain the standards for behavioural coding. [Rerun](https://rerun.io/) and [Foxglove](https://foxglove.dev/) are the best robotics visualisation tools.

What Potato adds is being multi-annotator and web-deployed with agreement statistics over the result. If one person is annotating, use ATLAS.

## Further reading

- [Robot episode annotation guide](/docs/guides/robot-episode-annotation)
- [Robot episodes reference](/docs/vision-spatial/robot-episodes)
- [LeRobot showcase design](/showcase/lerobot-episode-annotation)