How to Evaluate Generated Video and World Models

Ask annotators to mark the frame where the world stops making sense rather than to rate plausibility. Break-point annotation is checkable, localised, and produces a real agreement statistic.

Rating a generated video 3 out of 5 for "physical plausibility" produces a number that cannot be checked, cannot be localised, and cannot be used to fix anything. Asking instead for the frame at which the world stops making sense, plus why, produces an annotation that can be all three.

Why a break-point

A point in time plus a category is:

Checkable. A researcher can open frame 47 and look.
Localised. The failure has a place in the tensor rather than only a score.
Comparable. Two annotators' answers are two points on a line, so a real chance-corrected agreement statistic applies.

Video-generation benchmarks report FVD and win rates. None of them reports whether the humans producing those win rates agreed with each other, which means none of them can separate a model difference from annotator noise.

The taxonomy

Give annotators named categories rather than asking them to describe the failure. A workable physics and causality set:

Category	What it means
Object permanence	An object vanishes, or appears, with nothing causing it
Rigid-body violation	A solid object bends, stretches or changes size
Interpenetration	Two solid objects pass through each other
Gravity violation	Something floats, falls upward, or stands unsupported
Causality violation	An effect happens before, or without, its cause
Identity flicker	An object swaps identity or category between frames
Appearance drift	Texture, colour or shape drifts with no event to explain it

Add a severity scale. "Visible on a second look" and "nothing after this point is worth judging" are different findings.

Blinding must be stable

Shuffle panel order per annotator so model identity is not visible, and seed the shuffle on the annotator and the item so it is reproducible.

Randomising afresh on each view breaks re-annotation: the annotator's second look at an item disagrees with their first for reasons that have nothing to do with the videos.

Make "no breaks" explicit

Require a positive "nothing wrong here" answer. Without it, a rollout with no marks is indistinguishable from one that was never reviewed, and your denominator is wrong in a direction that flatters the model.

Report the sweep

Break-point agreement decomposes three ways:

Detection — did they find a break at all?
Localization — did they put it in the same place?
Category — did they call it the same thing?

And localization depends on a matching tolerance. Report the sweep across tolerances rather than one number, because agreement at 0.25 s and agreement at 2 s are different claims, and picking one lets the reader assume whichever supports their conclusion.

Config

yaml

annotation_schemes:
  - annotation_type: rollout_evaluation
    name: rollout_review
    description: "Where does each rollout stop making sense?"
    streams:
      - {field: real,  name: "Recording", role: real}
      - {field: gen_a, name: "Model A"}
      - {field: gen_b, name: "Model B"}
    fps: 25
    layers: [violations, preference, counterfactual]
    blind: true
    shuffle: true
    require_clean: true

Declare fps. Without it, frame numbers are omitted rather than guessed.

Where this fits

VBench, WorldModelBench and Physics-IQ provide the metrics and the reference protocols. Crowd panels provide scale. What none of them provides is an instrument for running the human side repeatedly with reliability measured — which is the gap this fills, and importing a benchmark's rollouts is a reasonable way to collect the human data those benchmarks validate against.