MultiChallenge Multi-Turn Context Use Review

# MultiChallenge: Multi-Turn Context Use Review
#
# Dataset: MultiChallenge (Scale AI), https://huggingface.co/datasets/ScaleAI/MultiChallenge
# Licence: CC BY 4.0 (Hugging Face dataset card). The ten conversations in
#   sample-data.json are real MultiChallenge test items, used with attribution.
#   The final "candidate response" turn on each item is illustrative. It was
#   written for this demo and is not a model output from the benchmark.
# Paper: Deshpande et al., Findings of ACL 2025, pages 18632-18702.
#   @inproceedings{deshpande-etal-2025-multichallenge,
#     title = "MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs",
#     author = "Deshpande, Kaustubh and Sirdeshmukh, Ved and Mols, Johannes Baptist and Jin, Lifeng and
#       Hernandez-Cardona, Ed-Yeremai and Lee, Dean and Kritz, Jeremy and Primack, Willow E. and
#       Yue, Summer and Xing, Chen",
#     booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
#     year = "2025", address = "Vienna, Austria",
#     url = "https://aclanthology.org/2025.findings-acl.958/" }
# DOI: 10.18653/v1/2025.findings-acl.958  (the arXiv preprint, 2501.17399, lists Sirdeshmukh first)
#
# Challenge categories (Section 3.1; the Hugging Face card spells them
# INSTRUCTION_RETENTION, INFERENCE_MEMORY, RELIABLE_VERSION_EDITING, SELF_COHERENCE):
# - Instruction Retention: whether the response keeps following an instruction the user gave in the first turn.
# - Inference Memory (of user information): whether the response recalls and connects details scattered
#   in earlier user turns that the final request needs only implicitly.
# - Reliable Versioned Editing: whether the response resolves version references, retrieves the correct
#   version without hallucination, and edits it as instructed.
# - Self-Coherence: whether the response stays consistent with the model's own earlier turns.
#
# Rubric protocol (Section 3.2), quoted: "For each test example, human raters provide a binary
# rubric question answerable with only "yes" or "no". This question requires only the final model
# response as context ... A "yes" answer indicates that the model response has passed this test example."
# Review protocol (Section 4.2), quoted: "The human review process mainly assesses 3 aspects of data
# quality, a) alignment with challenge category definition; b) naturalness and realism of
# conversation; c) reasonable failure of frontier LLMs."
#
# Adaptations:
# - The benchmark ends each conversation on a user turn and scores a model's reply. The released data
#   has no replies, so each item carries one illustrative candidate response as a last Assistant turn.
# - The paper's raters judge the rubric question on the final response. This design adds
#   context_attribution (a Potato scheme) so raters also mark which earlier turns the response used
#   correctly, ignored, or invented context from. The paper has no such step.
# - The paper's review aspects (a) and (b) become the category and naturalness items below.
#   Aspect (c), frontier-model failure, needs the six model responses and is left out.
# - The paper's two reviewer layers are approximated with num_annotators_per_item, not a phase chain.
# - Rubric questions are shown verbatim from the dataset's target_question field. The expected answer
#   (pass_criteria) stays in the data file and is not displayed.
#
# Trainer: none. No trainer in Potato's registry consumes context_attribution, so this config has no
# model_training block and nothing learns from these labels.

annotation_task_name: "MultiChallenge Multi-Turn Context Use Review"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: id
  text_key: rubric_question
  private_keys:
    - pass_criteria

output_annotation_dir: "annotation_output/"
export_annotation_format: "jsonl"

port: 8000

instance_display:
  layout:
    direction: vertical
    gap: 16px
  fields:
    - key: rubric_question
      type: text
      label: "Rubric question (answer it about the final Assistant turn only)"
    - key: conversation
      type: dialogue
      label: "Conversation"
      display_options:
        show_turn_numbers: true
        alternating_shading: true
        max_height: 480

annotation_schemes:
  - annotation_type: radio
    name: rubric_answer
    description: "Answer the rubric question above about the final Assistant turn. Yes means the response passed."
    labels:
      - name: "Yes"
        tooltip: "The final response satisfies the rubric question (pass)"
        key_value: "1"
      - name: "No"
        tooltip: "The final response does not satisfy the rubric question (fail)"
        key_value: "2"

  - annotation_type: context_attribution
    name: context_use
    description: "Tag each turn by how it uses earlier context, and click the source turn for used_context_correctly and hallucinated_context."
    turns_key: conversation

  - annotation_type: radio
    name: challenge_category
    description: "Which challenge category does this conversation test?"
    labels:
      - name: "Instruction Retention"
        tooltip: "A first-turn instruction should still be followed in the final response"
      - name: "Inference Memory"
        tooltip: "Earlier user details are implicitly needed in the final response"
      - name: "Reliable Versioned Editing"
        tooltip: "The user refers to earlier versions of material being edited"
      - name: "Self-Coherence"
        tooltip: "The response should stay consistent with the model's own earlier turns"
      - name: "Unclear or more than one"
        tooltip: "The conversation does not unambiguously target a single category"

  - annotation_type: likert
    name: naturalness
    description: "How natural and realistic is the conversation?"
    min_label: "Contrived"
    max_label: "Realistic"
    size: 5

annotation_instructions: |
  You review conversations from the MultiChallenge benchmark, which tests whether a model keeps
  track of an earlier part of a multi-turn conversation.

  For each item:
  1. Read the conversation and the final Assistant turn. That turn is an illustrative candidate response.
  2. Answer the rubric question about the final response only. Yes means it passed.
  3. Tag turns with how they use earlier context. For used_context_correctly and hallucinated_context,
     click the earlier turn that the tagged turn draws on. Use ignored_context where a turn should have
     used earlier information and did not.
  4. Pick the challenge category the conversation tests, or mark it unclear.
  5. Rate how natural and realistic the conversation is.

user_config:
  allow_all_users: true
max_annotations_per_user: 10
num_annotators_per_item: 2