# World Model and Robot Episode Evaluation Tools Compared

Source: https://www.potatoannotator.com/docs/guides/world-model-evaluation-tools-compared

**World models and video generators are evaluated with benchmarks such as VBench, WorldModelBench and Physics-IQ, which define what to measure and provide automatic scorers. Those scorers are checked against human judgements. Collecting human judgements you can defend, with raters blinded to the model and their agreement reported, is annotation work. Potato has schemas for judging generated rollouts and for segmenting robot episodes.**

A world model predicts how a scene will change, often given an action, and a video generation model can be evaluated as one by asking whether its output obeys physics and follows the instruction it was given. A rollout is one generated continuation. A robot episode is a recording of one attempt at a task, usually several camera streams plus joint positions, gripper state and other signals.

This page covers world models, generated video and robotics. [Annotation Tools Compared](/docs/guides/annotation-tools-compared) covers every data type on one page.

## What each tool does

| Tool | What it is | Where people come in |
|---|---|---|
| **[VBench](https://github.com/Vchitect/VBench)** | Benchmark suite for video generation, Apache-2.0 | Human preference annotations for each dimension, used to check that the automatic scores match people |
| **[WorldModelBench](https://arxiv.org/abs/2502.20694)** | Benchmark of video generators as world models | 67,000 crowd-sourced human labels, also used to train an automatic judge |
| **[Physics-IQ](https://github.com/google-deepmind/physics-IQ-benchmark)** | Benchmark of physical understanding in generative video, from Google DeepMind | A score and a leaderboard |
| **[ATLAS](https://arxiv.org/abs/2503.03539)** | Desktop tool for long-horizon action segmentation, with ROS bag and RLDS support | One annotator per episode |
| **[Rerun](https://rerun.io/)**, **[Foxglove](https://foxglove.dev/)** | Robotics visualization | Viewing, not labelling studies |
| **ELAN**, **BORIS** | Behavioural coding of video | ELAN reports inter-annotator reliability |
| **CVAT**, **Label Studio** | General video annotation | Tracks and timeline labels, with no world-model scheme |
| **[Potato](/)** | Annotation tool with `rollout_evaluation` and `episode_annotation` schemes | Several annotators per item, blinded panels, agreement reported |

## Benchmarks define the protocol

**VBench** scores text-to-video models on 16 dimensions, including subject consistency, motion smoothness and spatial relationships. VBench-2.0 adds 18 dimensions on commonsense reasoning, physics, human motion and composition. For each dimension the authors collected human preference annotations and showed that the automatic scores track them.

**WorldModelBench** evaluates video generators on instruction following and physics adherence, catching violations such as an object changing size in breach of mass conservation. The authors crowd-sourced 67,000 human labels to evaluate 14 frontier models, then fine-tuned a 2-billion-parameter judge that predicts world-modelling violations with 8.6% higher accuracy than GPT-4o ([arXiv 2502.20694](https://arxiv.org/abs/2502.20694)).

**Physics-IQ** covers solid mechanics, fluid dynamics, optics, thermodynamics and magnetism. Testing Sora, Runway, Pika, Lumiere, Stable Video Diffusion and VideoPoet, its authors found physical understanding severely limited and unrelated to visual realism ([arXiv 2501.09038](https://arxiv.org/abs/2501.09038)).

Use these for the metrics and the reference protocols. They are what a result will be compared against.

## Where human judgement comes in

VBench uses human judgements to validate its dimensions, and WorldModelBench uses them to train its judge. When you evaluate your own model, the human part has to be repeated for each new checkpoint. It then has the problems of any annotation study: raters who know which model made which video, raters who disagree on what counts as a violation, and no agreement figure to show the labels can be trusted.

## Judging generated rollouts in Potato

The `rollout_evaluation` scheme shows two or more rollouts on a shared clock. The annotator marks the frame where the world stops making sense, tags which physical or causal property broke, states a preference against a rubric, and rates whether a counterfactual divergence is plausible given the intervention. Panels can be blinded and shuffled in a stable order per annotator, so a reload does not reveal which model is which.

Agreement on the break point is reported three ways: whether annotators detected a break at all, where they placed it, and which category they gave it. The matching tolerance is shown as a sweep across values instead of at one threshold.

```yaml
annotation_schemes:
  - annotation_type: rollout_evaluation
    name: rollout_review
    description: "Where does each rollout stop making sense?"
    streams:
      - {field: real,  name: "Recording", role: real}
      - {field: gen_a, name: "Model A"}
      - {field: gen_b, name: "Model B"}
    fps: 25
    layers: [violations, preference, counterfactual]
    blind: true
    shuffle: true
    require_clean: true
```

`require_clean` makes "no break" an explicit answer, so a rollout nobody flagged is distinguishable from one nobody finished. See [How to Evaluate Generated Video and World Models](/docs/guides/world-model-evaluation).

## Annotating robot episodes

The `episode_annotation` scheme puts several synchronized video streams and robot time-series lanes (joint positions, gripper state, force-torque, reward) on one timeline. Annotators segment phases, mark the outcome of each phase, draw a dense progress reward, and relabel the instruction in hindsight. Potato imports LeRobot v2, HDF5 (RoboMimic and ALOHA) and RLDS/TFDS, and writes a per-frame JSONL sidecar so a read-only public dataset does not have to be rewritten.

Agreement is reported as temporal IoU for phase boundaries, α for outcome labels, and ICC with Pearson correlation for reward curves, with coverage stated alongside, because a correlation over 5% of a timeline says nothing about the rest.

**ATLAS**, from TU Wien, is a desktop tool for long-horizon action segmentation that reads ROS bags and RLDS natively. For one annotator placing boundaries precisely, it is purpose-built. Its published result, that showing proprioceptive time series next to the video reduces boundary error compared with video alone, is why Potato draws those lanes. **Rerun** and **Foxglove** are the best robotics visualization tools, and **ELAN** and **BORIS** remain the standards for behavioural coding. See [How to Annotate Robot Episodes](/docs/guides/robot-episode-annotation).

For grounding and pointing evaluation of vision-language models, see [VLM Grounding Evaluation](/docs/guides/vlm-grounding-evaluation).

## What Potato does not do here

- **No automatic physics scorer.** Use a benchmark's scorer for that, and Potato for the human labels it is validated against.
- **No model training or inference.** Potato shows the rollouts you generated.
- **No point-cloud sequences.** 3D annotation is per frame.

Checked against each project's repository and paper in August and September 2026.

## Frequently asked questions

### How are world models evaluated?

With benchmarks that score generated video for physics adherence, instruction following and visual quality, such as VBench, WorldModelBench and Physics-IQ, and with human judgements that validate or train those scorers. For a new model, the human part means people watching rollouts, marking where they stop making physical sense and comparing them, with the rater blinded to the model.

### Which world-model benchmarks use human judgements?

VBench collected human preference annotations for each of its dimensions to show that its automatic scores match people. WorldModelBench crowd-sourced 67,000 human labels across 14 models and used them to fine-tune an automatic judge.

### What tool can I use to annotate robot episodes?

ATLAS is a desktop tool for one annotator segmenting long episodes from ROS bags or RLDS. Potato annotates episodes in a browser for several annotators, imports LeRobot v2, HDF5 and RLDS/TFDS, and reports agreement on phase boundaries, outcomes and reward curves.

### How do I measure agreement between people judging generated videos?

Have each rollout judged by at least two raters who cannot see which model produced it. Report whether they agree that it breaks at all, how close their break points are across a range of tolerances, and whether they give the same category. Potato's `rollout_evaluation` reports these three separately.

## Further reading

- [Annotation Tools Compared](/docs/guides/annotation-tools-compared), for every data type on one page
- [Video Annotation Tools Compared](/docs/guides/video-annotation-tools-compared)
- [Image Annotation Tools Compared](/docs/guides/image-annotation-tools-compared)
- [AI Agent Evaluation Tools Compared](/docs/guides/agent-evaluation-tools-compared)
- [Text Annotation Tools Compared](/docs/guides/text-annotation-tools-compared)
- [Audio Annotation Tools Compared](/docs/guides/audio-annotation-tools-compared)
- [Robot Episodes](/docs/vision-spatial/robot-episodes)