RoboMIND Robot Episode Failure Review

# yaml-language-server: $schema=https://potatoannotator.readthedocs.io/en/latest/schemas/potato-config.schema.json
#
# RoboMIND - Robot Episode Failure Review
#
# Dataset: RoboMIND (Multi-embodiment Intelligence Normative Data for Robot
#          Manipulation), https://huggingface.co/datasets/x-humanoid-robomind/RoboMIND
#          Apache-2.0 on the dataset card, but access-gated (12.3 TB of HDF5).
# Paper:   Wu et al., "RoboMIND: Benchmark on Multi-embodiment Intelligence
#          Normative Data for Robot Manipulation", Robotics: Science and Systems
#          XXI (RSS 2025).
# DOI:     10.15607/RSS.2025.XXI.152
# BibTeX:  @inproceedings{wu2025robomind,
#            title     = {RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulation},
#            author    = {Wu, Kun and Hou, Chengkai and Liu, Jiaming and Che, Zhengping and others},
#            (the paper's byline spells one author "He, Jingyang"; Crossref's BibTeX has "Jingyang, He")
#            booktitle = {Robotics: Science and Systems XXI},
#            publisher = {Robotics: Science and Systems Foundation},
#            year      = {2025},
#            doi       = {10.15607/rss.2025.xxi.152}
#          }
#          (37 authors; the full list is in metadata.json.)
#
# THE EPISODES IN THIS FOLDER ARE SYNTHETIC. The dataset is gated, so none of
# its trajectories or video is redistributed here. The eleven episodes under
# media/ were generated for this design (see media/LICENSE-media.txt). They use
# Potato's episode manifest format, mimic four of RoboMIND's embodiments only by
# label and joint count, and show one failure signature each so the nine
# categories below can be tried. They say nothing about RoboMIND's actual data.
#
# The paper (Sec. V-F) says "we recorded not only whether the model's task execution
# was successful but also the reasons for any failures" (ACT policy test rollouts),
# and "We predefined nine failure categories":
#   (1) Inaccurate Positioning; (2) Cannot Close Gripper; (3) Cannot Approach
#   Object; (4) Early Gripper Release; (5) Object Drop; (6) Cannot Release
#   Gripper; (7) Cannot Return to Home Pose; (8) Collision Recovery Failure;
#   (9) Excessive Speed.
# The Introduction adds that the dataset includes "5k real-world failure
# demonstrations, each accompanied by detailed causes". Section III-C says
# language annotation for 10k successful trajectories starts with Gemini, which
# "segment[s] each video based on the sequence of operations", and is then
# refined by hand. The nine categories below are quoted from Sec. V-F of the
# arXiv version 3 (2412.13877v3, 27 May 2025).
#
# Adaptations
# -----------
# - Synthetic episodes (above), at 20 Hz, six seconds, with two camera streams
#   and joint, gripper and wrist-force series. Real RoboMIND episodes are HDF5.
#   Potato also reads RoboMimic and ALOHA HDF5 through the same `source_field`.
# - Outcome is success or failure, as in the paper's tally of successful and
#   failed rollouts. The sibling lerobot-episode-annotation design adds "partial".
# - Phases (reach, grasp, transport, place, retract) are this design's choice.
#   The paper segments by operation sequence with free-text descriptions per
#   segment, which is closer to the `instruction` layer than to a fixed list.
# - The `instruction` layer stands in for that free-text description. It is
#   per episode here, not per segment, unless the annotator aligns it.
# - The paper does not name a reward layer, so none is collected.
# - Each item carries an `embodiment` label. The paper covers the Franka Emika
#   Panda, the X-Humanoid Tien Kung humanoid, the AgileX dual-arm robot and the
#   UR5e. Lane names are shared (joint_1 ... joint_N, gripper, wrist_force) so
#   one `series_shown` list fits every embodiment.
#
# Training loop
# -------------
# `model_training` is on. The `episode-model` trainer (pick it on
# /admin/training, admin key below) fits a small convolution plus bidirectional
# GRU over each episode's recorded series, then predicts phase segments and the
# success or failure outcome for episodes nobody has labelled, written back as
# pre-filled answers to confirm or correct. It reads the series only, not the
# video, and it needs at least 5 labeled episodes. It does not predict the
# failure cause (the cause field is left empty), the instruction text, or any
# of RoboMIND's nine categories.
#
# Run: potato start config.yaml -p 8000

port: 8000
annotation_task_name: "RoboMIND - Robot Episode Failure Review"
task_dir: "."
media_directory: media
output_annotation_dir: "annotation_output/"
export_annotation_format: "jsonl"

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "instruction"

user_config:
  allow_all_users: true
  users: []

admin_api_key: "demo-admin-key"

annotation_schemes:
  - annotation_type: episode_annotation
    name: episode_review
    description: "Mark the phases, judge the outcome, and name the failure cause."
    source_field: episode
    layers:
      - phases
      - outcome
      - instruction
    phases:
      - name: reach
        color: "#4ECDC4"
        key_value: "1"
      - name: grasp
        color: "#FFD93D"
        key_value: "2"
      - name: transport
        color: "#6C8AE4"
        key_value: "3"
      - name: place
        color: "#95E1A3"
        key_value: "4"
      - name: retract
        color: "#C9A0DC"
        key_value: "5"
    outcomes:
      - success
      - failure
    # The nine categories predefined in the RoboMIND paper, Sec. V-F, in the
    # paper's order and wording.
    failure_causes:
      - Inaccurate Positioning
      - Cannot Close Gripper
      - Cannot Approach Object
      - Early Gripper Release
      - Object Drop
      - Cannot Release Gripper
      - Cannot Return to Home Pose
      - Collision Recovery Failure
      - Excessive Speed
    series_shown:
      - gripper
      - wrist_force
      - joint_2
      - joint_3

  - annotation_type: text
    name: notes
    description: "Anything the phases, outcome and cause do not capture? (optional)"

model_training:
  enabled: true
  writeback:
    schemas: ["episode_review"]
    include_unlabeled: true
    min_confidence: 0.6

annotation_instructions: |
  Each item is one robot rollout: two camera views above, the arm's own signals
  as lanes below, and one timeline through all of them. The episodes are
  synthetic and each was built to show one failure signature.

  1. Segment the episode into phases. Drag a boundary where the robot's
     sub-goal changes. Number keys 1-5 select phases directly.
  2. Judge the outcome, success or failure.
  3. If it failed, pick the cause from the nine categories. Choose the first
     thing that went wrong, not the last thing visible in the video.
  4. Use the instruction box to write what the robot actually did when that
     differs from the instruction it was given.

  Read the lanes as well as the video. A gripper that never closes, a force
  trace that stays flat at the grasp, or a sudden spike followed by frozen
  joints each point to a different category.