World-Model and Generative-Video Evaluation
Show 2 to N generated videos frame-locked on one clock and ask annotators to mark the frame at which the world stops making sense, then tag which physical property broke.
Several videos of the same scenario, frame-locked on one timeline. The annotator finds the frame at which each rollout stops making sense, says why, picks a winner, and where the scenario carries an intervention, judges whether the divergence follows from it. Potato then reports whether the annotators agreed on where the break was.
Runnable example: examples/agent-traces/world-model-rollouts/.
Break-points beat ratings
Rating a generated video 3 out of 5 for "physical plausibility" produces a number that cannot be checked, cannot be localised, and cannot be used to fix anything.
A point in time plus a category can be all three:
- Checkable. A researcher can open frame 47 and look.
- Localised. The failure has a place in the tensor rather than only a score.
- Comparable. Two annotators' answers are two points on a line, so a real chance-corrected agreement statistic applies.
Video-generation benchmarks report FVD and win rates. None reports whether the humans producing those win rates agree with each other, which means none can separate a model difference from annotator noise.
Configuration
annotation_schemes:
- annotation_type: rollout_evaluation
name: rollout_review
description: "Where does each rollout stop making sense?"
streams:
- {field: real, name: "Recording", role: real}
- {field: gen_a, name: "Model A"}
- {field: gen_b, name: "Model B"}
fps: 25
layers: [violations, preference, counterfactual]
blind: true
shuffle: true
require_clean: true
violation_types:
- {name: object_permanence, description: "An object vanishes, or appears, with nothing causing it."}
- {name: interpenetration, description: "Two solid objects pass through each other."}
- {name: gravity_violation, description: "Something floats, falls upward, or stands unsupported."}Declare fps, or frame numbers are omitted from the output rather than guessed.
Stable blinding and shuffling
Panels can be blinded and shuffled per annotator, and the shuffle is stable, seeded on the annotator and the item. An annotator's second look at the same item agrees with their first, which a fresh random order each time would break.
require_clean: true makes "no breaks" an explicit act rather than an empty answer. Without it, a rollout with no marks is indistinguishable from one that was never reviewed.
Break-point agreement is a sweep
Agreement is reported three ways: detection (did they find a break at all), localization (did they put it in the same place), and category (did they call it the same thing). The matching tolerance is shown as a sweep rather than a single number.
That is not extra detail for its own sake. Agreement at 0.25 s and agreement at 2 s are different claims, and reporting one threshold lets a reader assume whichever supports their conclusion.
Where this sits in the ecosystem
The comparable work is benchmarks — VBench, WorldModelBench, Physics-IQ — and crowd panels. Those provide the metrics and the reference protocols; Potato provides an instrument for running the human side of such a protocol repeatedly, with reliability measured. They are complementary, and importing a benchmark's rollouts into Potato is a reasonable way to collect the human data those benchmarks validate against.