VideoPhy-2 Generated Video Physics Evaluation

# yaml-language-server: $schema=https://potatoannotator.readthedocs.io/en/latest/schemas/potato-config.schema.json
#
# VideoPhy-2 - Generated Video Physics Evaluation
#
# Dataset: VideoPhy-2, https://huggingface.co/datasets/videophysics/videophy2_test
#          (train split: https://huggingface.co/datasets/videophysics/videophy2_train).
#          MIT on the dataset card. The card gives no redistribution terms for the
#          generated videos it links (a `video_url` column), so none are used here.
# Paper:   Bansal et al., "VideoPhy-2: A Challenging Action-Centric Physical
#          Commonsense Evaluation in Video Generation", arXiv 2025.
# DOI:     10.48550/arXiv.2503.06800 (arXiv only; arXiv lists no published DOI)
# BibTeX:  @misc{bansal2025videophy2,
#            title     = {VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation},
#            author    = {Bansal, Hritik and Peng, Clark and Bitton, Yonatan and Goldenberg, Roman and Grover, Aditya and Chang, Kai-Wei},
#            publisher = {arXiv},
#            year      = {2025},
#            doi       = {10.48550/ARXIV.2503.06800},
#            url       = {https://arxiv.org/abs/2503.06800}
#          }
#
# THE CLIPS IN THIS FOLDER ARE SYNTHETIC. Generated videos from the dataset are
# not redistributable under any licence the card states, so the ten clips under
# media/ were rendered for this design (see media/LICENSE-media.txt). Each is
# either coherent or carries one deliberate defect. The prompts and candidate
# rules in sample-data.json are written for this design in the style of the
# dataset's, and are not copied from it.
#
# Rating scales, quoted from the paper (Sec. 3.1):
#   Both semantic adherence (SA) and physical commonsense (PC) use
#   "Very Unlikely (1), Unlikely (2), Neutral (3), Likely (4), Very Likely (5)".
#   SA aims "to assess whether the input text prompt is semantically grounded in
#   the generated video". "very unlikely indicates that the video does not match
#   the prompt at all" and "very likely highlights the video fully adheres to the
#   prompt with no inconsistencies."
#   PC: "Our goal is to assess whether the generated video follows the physical
#   laws of the real-world intuitively". "very unlikely that the video contains
#   numerous violations of fundamental physical laws" and "very likely indicates
#   that the video demonstrates a strong understanding of physical commonsense
#   with no violations."
#   Per rule: "we determine whether a candidate physical rule is violated (0),
#   followed (1), or cannot be determined (2)." The paper has an LLM
#   (Gemini-2.0-Flash-Exp) generate three candidate rules per video, and the
#   "cannot be determined" option exists because "LLM-generated rules may not
#   always be visually grounded in the video." Annotators may also add violated
#   rules the list missed (Sec. 2, Stage 3; Sec. 3.1, footnote 4).
#   The paper's footnote 6 says the overall PC judgment "should extend beyond the
#   predefined physical rules", so the PC rating is separate from the rule labels.
#
# Adaptations
# -----------
# - Break-point marking is Potato's addition. VideoPhy-2 collects per-video
#   ratings and per-rule labels, not the time at which a video stops making
#   sense. The `rollout_evaluation` scheme (violations layer only) adds that,
#   with one stream per item. Its violation types, severities and keyboard
#   shortcuts come from Potato's defaults, not from the paper. The preference and
#   counterfactual layers are off because the benchmark shows one video per prompt.
# - Per-rule labels are three radio schemes (rule_1 to rule_3) whose wording is
#   fixed, and the item's `rules` text names rule 1, 2 and 3. Potato has no scheme
#   that takes a per-item list of statements to label, so the labels are tied to
#   positions in the text. "Violated (0), followed (1), cannot be determined (2)"
#   keep the paper's wording, and the stored value is the label name.
# - SA is rated while the prompt is visible and PC is rated alongside it. The
#   paper collects them as separate tasks "to avoid bias from annotators seeing
#   the prompt during PC judgments", and shows the original prompt, not the
#   upsampled one. One page cannot hide the prompt for one scheme only, so use two
#   passes (drop the sa scheme in a second config) when that separation matters.
# - The paper used 12 annotators on Amazon Mechanical Turk, three per video, with
#   scores averaged and rule labels decided by majority vote. This config sets
#   num_annotators_per_item: 3 and leaves aggregation to analysis. Recruit
#   through another channel, because MTurk has closed.
# - Annotator-added violated rules are not collected as structured data. The
#   notes box takes them as text.
#
# Training loop
# -------------
# There is no model_training block. Potato has no trainer that consumes
# rollout_evaluation, and the likert and radio schemes here score a video the
# annotator watched, not text or an embedding a trainer could fit. VideoPhy-2's
# own auto-rater (VideoPhy-2-AutoEval) is a separate model that Potato does not
# train or serve.
#
# Run: potato start config.yaml -p 8000

port: 8000
annotation_task_name: "VideoPhy-2 - Generated Video Physics Evaluation"
task_dir: "."
media_directory: media
output_annotation_dir: "annotation_output/"
export_annotation_format: "jsonl"

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "rules"

user_config:
  allow_all_users: true
  users: []

num_annotators_per_item: 3

annotation_schemes:
  - annotation_type: likert
    name: semantic_adherence
    description: "Semantic adherence: how likely is it that the video matches the prompt? (Very Unlikely means the video does not match the prompt at all. Very Likely means it fully adheres to the prompt with no inconsistencies.)"
    displaying_score: true
    labels:
      - "Very Unlikely"
      - "Unlikely"
      - "Neutral"
      - "Likely"
      - "Very Likely"

  - annotation_type: likert
    name: physical_commonsense
    description: "Physical commonsense: how likely is it that the video follows the physical laws of the real world intuitively? (Very Unlikely means the video contains numerous violations of fundamental physical laws. Very Likely means it shows a strong understanding of physical commonsense with no violations.)"
    displaying_score: true
    labels:
      - "Very Unlikely"
      - "Unlikely"
      - "Neutral"
      - "Likely"
      - "Very Likely"

  - annotation_type: radio
    name: rule_1
    description: "Candidate rule (1) in the text above: is it violated, followed, or can it not be determined from the video?"
    labels:
      - "violated"
      - "followed"
      - "cannot be determined"

  - annotation_type: radio
    name: rule_2
    description: "Candidate rule (2) in the text above: is it violated, followed, or can it not be determined from the video?"
    labels:
      - "violated"
      - "followed"
      - "cannot be determined"

  - annotation_type: radio
    name: rule_3
    description: "Candidate rule (3) in the text above: is it violated, followed, or can it not be determined from the video?"
    labels:
      - "violated"
      - "followed"
      - "cannot be determined"

  - annotation_type: rollout_evaluation
    name: break_points
    description: "Where does the video stop making sense? Mark each frame at which the physics breaks, or mark the video as having no breaks."
    streams:
      - field: video
        name: "Generated video"
    fps: 25
    prompt_field: prompt
    layers:
      - violations
    blind: false
    shuffle: false
    require_clean: true
    violation_types:
      - name: object_permanence
        description: "An object vanishes, or appears, with nothing causing it."
      - name: rigid_body_violation
        description: "A solid object bends, stretches or changes size."
      - name: interpenetration
        description: "Two solid objects pass through each other."
      - name: gravity_violation
        description: "Something floats, falls upward, or stands unsupported."
      - name: identity_flicker
        description: "An object swaps identity or category between frames."
      - name: appearance_drift
        description: "Texture, colour or shape drifts with no event to explain it."
    severities:
      - value: 1
        name: subtle
        description: "Visible on a second look."
      - value: 2
        name: clear
        description: "Obvious on first viewing."
      - value: 3
        name: "breaks the scene"
        description: "Nothing after this point is worth judging."

  - annotation_type: text
    name: notes
    description: "Any violated rule the list missed, or anything else the ratings do not capture? (optional)"

annotation_instructions: |
  Each item is one generated video with the prompt that produced it. The ten
  clips are synthetic and illustrative, and each is either coherent or built
  with one defect.

  1. Watch the video. Rate semantic adherence, meaning how well it matches the
     prompt, on the five-point scale.
  2. Rate physical commonsense, meaning whether it follows the physical laws of
     the real world intuitively. Judge the whole video, not only the three
     candidate rules.
  3. For each of the three candidate rules in the text, choose violated,
     followed, or cannot be determined. Choose "cannot be determined" when the
     rule cannot be judged from what the video shows.
  4. Step through the video with the comma and period keys, and mark the frame
     at which the physics first breaks. Pick a type and a severity. If it never
     breaks, mark the video as having no breaks.
  5. Add any violated rule the list missed in the notes box.