# World-Model and Generative-Video Evaluation

Source: https://www.potatoannotator.com/docs/vision-spatial/world-model-evaluation

**Several videos of the same scenario, frame-locked on one timeline. The annotator finds the frame at which each rollout stops making sense, says why, picks a winner, and where the scenario carries an intervention, judges whether the divergence follows from it.** Potato then reports whether the annotators agreed on *where* the break was.

Runnable example: [`examples/agent-traces/world-model-rollouts/`](https://github.com/davidjurgens/potato/tree/main/examples/agent-traces/world-model-rollouts).

## Break-points beat ratings

Rating a generated video 3 out of 5 for "physical plausibility" produces a number that cannot be checked, cannot be localised, and cannot be used to fix anything.

A point in time plus a category can be all three:

- **Checkable.** A researcher can open frame 47 and look.
- **Localised.** The failure has a place in the tensor rather than only a score.
- **Comparable.** Two annotators' answers are two points on a line, so a real chance-corrected agreement statistic applies.

Video-generation benchmarks report FVD and win rates. None reports whether the humans producing those win rates agree with each other, which means none can separate a model difference from annotator noise.

## Configuration

```yaml
annotation_schemes:
  - annotation_type: rollout_evaluation
    name: rollout_review
    description: "Where does each rollout stop making sense?"
    streams:
      - {field: real,  name: "Recording", role: real}
      - {field: gen_a, name: "Model A"}
      - {field: gen_b, name: "Model B"}
    fps: 25
    layers: [violations, preference, counterfactual]
    blind: true
    shuffle: true
    require_clean: true
    violation_types:
      - {name: object_permanence, description: "An object vanishes, or appears, with nothing causing it."}
      - {name: interpenetration,  description: "Two solid objects pass through each other."}
      - {name: gravity_violation, description: "Something floats, falls upward, or stands unsupported."}
```

Declare `fps`, or frame numbers are omitted from the output rather than guessed.

## Stable blinding and shuffling

Panels can be blinded and shuffled per annotator, and the shuffle is **stable**, seeded on the annotator and the item. An annotator's second look at the same item agrees with their first, which a fresh random order each time would break.

`require_clean: true` makes "no breaks" an explicit act rather than an empty answer. Without it, a rollout with no marks is indistinguishable from one that was never reviewed.

## Break-point agreement is a sweep

Agreement is reported three ways: **detection** (did they find a break at all), **localization** (did they put it in the same place), and **category** (did they call it the same thing). The matching tolerance is shown as a **sweep** rather than a single number.

That is not extra detail for its own sake. Agreement at 0.25 s and agreement at 2 s are different claims, and reporting one threshold lets a reader assume whichever supports their conclusion.

## Where this sits in the ecosystem

The comparable work is benchmarks — [VBench](https://arxiv.org/abs/2311.17982), WorldModelBench, Physics-IQ — and crowd panels. Those provide the metrics and the reference protocols; Potato provides an instrument for running the human side of such a protocol repeatedly, with reliability measured. They are complementary, and importing a benchmark's rollouts into Potato is a reasonable way to collect the human data those benchmarks validate against.

## Related

- [Robot episodes](/docs/vision-spatial/robot-episodes)
- [Temporal agreement](/docs/measurement/temporal-agreement)
- [Guide: world-model evaluation](/docs/guides/world-model-evaluation)
- [Source documentation](https://github.com/davidjurgens/potato/blob/main/docs/agent-evaluation/world_model_eval.md)