# How to Evaluate Generated Video and World Models

Source: https://www.potatoannotator.com/docs/guides/world-model-evaluation

**Rating a generated video 3 out of 5 for "physical plausibility" produces a number that cannot be checked, cannot be localised, and cannot be used to fix anything.** Asking instead for *the frame at which the world stops making sense*, plus why, produces an annotation that can be all three.

## Why a break-point

A point in time plus a category is:

- **Checkable.** A researcher can open frame 47 and look.
- **Localised.** The failure has a place in the tensor rather than only a score.
- **Comparable.** Two annotators' answers are two points on a line, so a real chance-corrected agreement statistic applies.

Video-generation benchmarks report FVD and win rates. None of them reports whether the humans producing those win rates agreed with each other, which means none of them can separate a model difference from annotator noise.

## The taxonomy

Give annotators named categories rather than asking them to describe the failure. A workable physics and causality set:

| Category | What it means |
|---|---|
| Object permanence | An object vanishes, or appears, with nothing causing it |
| Rigid-body violation | A solid object bends, stretches or changes size |
| Interpenetration | Two solid objects pass through each other |
| Gravity violation | Something floats, falls upward, or stands unsupported |
| Causality violation | An effect happens before, or without, its cause |
| Identity flicker | An object swaps identity or category between frames |
| Appearance drift | Texture, colour or shape drifts with no event to explain it |

Add a severity scale. "Visible on a second look" and "nothing after this point is worth judging" are different findings.

## Blinding must be stable

Shuffle panel order per annotator so model identity is not visible, and **seed the shuffle on the annotator and the item** so it is reproducible.

Randomising afresh on each view breaks re-annotation: the annotator's second look at an item disagrees with their first for reasons that have nothing to do with the videos.

## Make "no breaks" explicit

Require a positive "nothing wrong here" answer. Without it, a rollout with no marks is indistinguishable from one that was never reviewed, and your denominator is wrong in a direction that flatters the model.

## Report the sweep

Break-point agreement decomposes three ways:

- **Detection** — did they find a break at all?
- **Localization** — did they put it in the same place?
- **Category** — did they call it the same thing?

And localization depends on a matching tolerance. Report the **sweep** across tolerances rather than one number, because agreement at 0.25 s and agreement at 2 s are different claims, and picking one lets the reader assume whichever supports their conclusion.

## Config

```yaml
annotation_schemes:
  - annotation_type: rollout_evaluation
    name: rollout_review
    description: "Where does each rollout stop making sense?"
    streams:
      - {field: real,  name: "Recording", role: real}
      - {field: gen_a, name: "Model A"}
      - {field: gen_b, name: "Model B"}
    fps: 25
    layers: [violations, preference, counterfactual]
    blind: true
    shuffle: true
    require_clean: true
```

Declare `fps`. Without it, frame numbers are omitted rather than guessed.

## Where this fits

[VBench](https://arxiv.org/abs/2311.17982), WorldModelBench and Physics-IQ provide the metrics and the reference protocols. Crowd panels provide scale. What none of them provides is an instrument for running the human side repeatedly with reliability measured — which is the gap this fills, and importing a benchmark's rollouts is a reasonable way to collect the human data those benchmarks validate against.

## Further reading

- [World-model evaluation reference](/docs/vision-spatial/world-model-evaluation)
- [Temporal agreement](/docs/measurement/temporal-agreement)
- [Robot episode annotation](/docs/guides/robot-episode-annotation)