Potato opens a robot demonstration episode as one timeline, with the camera streams above and the robot's own signals beneath, so annotators can mark task phases, judge the outcome and its cause, and draw a progress curve. Potato is free and open source, runs on your own server, and also runs blind evaluations of world-model rollouts with agreement between annotators.
The success flags in large robot datasets are not always right. A discussion on the DROID sample reports an episode marked successful in which the gripper dropped the object, and checking those flags is human work. A reviewer needs to see the same moment in every camera and in the gripper and force signals, because a failed grasp often shows in those traces before the failure is obvious on video. Potato's episode_annotation scheme puts the cameras and the signals on one time axis. Several annotators can review the same episode, and Potato reports agreement for each kind of answer.
Robotics tasks in Potato
Each row is a layer or scheme that ships with Potato, and the linked page has its configuration.
| Task | How Potato does it | Docs |
|---|---|---|
| Segment an episode into phases | A phases layer with your own phase names and keyboard shortcuts | Robot episodes |
| Label success and failure | An outcome layer with success, partial and failure, plus failure causes you define | Robot episodes |
| Draw a progress or reward curve | A reward layer drawn over the timeline within a range you set | Robot episodes |
| Relabel the instruction | Instruction relabeling collected with the outcome, for an episode that failed at its recorded task but shows another | Robot episodes |
| Evaluate world-model rollouts | rollout_evaluation: several rollouts frame-locked on one timeline, with the break point marked and a winner picked | World-model evaluation |
| 3D labels on sensor data | Cuboids on point clouds and readouts on depth maps | Point clouds |
An episode review scheme
The scheme below enables all three layers. Phases take seconds per episode and a dense reward curve takes minutes, so most projects turn on only the layers the study needs.
annotation_schemes:
- annotation_type: episode_annotation
name: episode_review
description: "Mark the phases, judge the outcome, and draw the progress."
source_field: episode
layers: [phases, outcome, reward]
phases:
- {name: reach, color: "#4ECDC4", key_value: "1"}
- {name: grasp, color: "#FFD93D", key_value: "2"}
- {name: transport, color: "#6C8AE4", key_value: "3"}
outcomes: [success, partial, failure]
failure_causes: [missed grasp, object slipped, collision]
reward_range: [0.0, 1.0]
series_shown: [gripper, wrist_force]series_shown picks which recorded signals appear as lanes under the video. The lanes are downsampled in a way that keeps the minimum and maximum, so a force spike one frame long stays visible.
Data in and out
Potato imports LeRobot v2 datasets, HDF5 files in the RoboMimic and ALOHA layouts, and RLDS datasets. Potato has no ROS bag reader, so convert a bag to one of these formats first. The export writes LeRobot v2 and a per-frame JSONL sidecar (episode_jsonl), so a read-only public dataset does not have to be rewritten to hold your labels.
Working alongside ATLAS, Rerun, and Foxglove
ATLAS is a desktop tool for long-horizon action segmentation with native ROS bag and RLDS support, built for one annotator placing boundaries precisely. Rerun and Foxglove are built for visualizing and debugging robot data. Potato is a web-based episode annotator for several people at once, with agreement measured on each layer. Phase boundaries are scored by temporal IoU, outcomes by Krippendorff's alpha, and reward curves by ICC with Pearson correlation.
Frequently asked questions
How do I label robot episodes as success or failure and mark where they failed?
Use an episode_annotation scheme with the outcome layer, which records success, partial, or failure and a failure cause from a list you define, and the phases layer, which marks when each phase starts and ends. Annotators see the camera streams and the robot's signals on the same timeline while deciding.
How do I run a blind human evaluation of world-model or video-generation outputs?
Load several rollouts of the same scenario as one item, and a rollout_evaluation scheme plays the rollouts frame-locked side by side. The panels can be blinded and shuffled per annotator, with a stable order seeded on the annotator and the item. The annotator marks the frame where each rollout stops making sense, says why, and picks a winner. Potato reports agreement on whether a break was found, where the break was, and what kind of break.
Further reading
- Robot episode annotation, for planning an episode review study
- Evaluating world models, for evaluation protocols on generated video
- Computer vision and AI agent evaluation, for related pages on this site