ChaosNLI: NLI Label Distributions

# ChaosNLI — Label Distributions for Natural Language Inference
# Based on the ChaosNLI dataset:
#   Yixin Nie, Xiang Zhou, and Mohit Bansal (2020), "What Can We Learn from
#   Collective Human Opinions on Natural Language Inference Data?" Proceedings of
#   the 2020 Conference on Empirical Methods in Natural Language Processing
#   (EMNLP), pp. 9131-9143.
#   DOI: 10.18653/v1/2020.emnlp-main.734
#   Paper: https://aclanthology.org/2020.emnlp-main.734/
#   Dataset: https://github.com/easonnie/ChaosNLI
#
# Task: given a context (premise) and a statement (hypothesis), judge whether the
# statement is definitely correct (Entailment), definitely incorrect
# (Contradiction), or neither (Neutral).
#
# FRAMING CAVEAT: ChaosNLI's label distributions were collected as 100
# independent single-choice categorical judgments per item from different
# crowd-workers; no individual annotator ever produced a distribution. This
# showcase config demonstrates Potato's soft_label scheme as a direct-elicitation
# alternative: each annotator first gives the original 3-way categorical judgment
# (radio), then distributes 100 points across the three labels — capturing the
# same distributional signal ChaosNLI aggregates over a crowd, but per annotator.
#
# Annotation instructions reproduced verbatim from Figure 1 (Section 3.2) of the
# paper — the compressed Mechanical Turker instructions for the NLI collections
# (the Abductive-NLI variant is omitted); the trailing note on the point
# allocation is a showcase addition.

annotation_task_name: "ChaosNLI: NLI Label Distributions"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "statement"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

port: 8000
server_name: localhost

annotation_schemes:
  - annotation_type: radio
    name: nli_label
    description: "Choose the correct category for the given pair of context and statement."
    labels:
      - "Definitely correct (Entailment)"
      - "Neither (Neutral)"
      - "Definitely incorrect (Contradiction)"
    keyboard_shortcuts:
      "Definitely correct (Entailment)": "1"
      "Neither (Neutral)": "2"
      "Definitely incorrect (Contradiction)": "3"
    tooltips:
      "Definitely correct (Entailment)": "Given the context, the statement is definitely correct"
      "Neither (Neutral)": "The statement is neither definitely correct nor definitely incorrect"
      "Definitely incorrect (Contradiction)": "Given the context, the statement is definitely incorrect"

  - annotation_type: soft_label
    name: nli_label_distribution
    description: "Now distribute 100 points across the three categories to reflect how a large group of careful readers would judge this pair."
    total: 100
    min_per_label: 0
    show_distribution_chart: true
    labels:
      - "Entailment"
      - "Neutral"
      - "Contradiction"
    label_requirement:
      required: true

annotation_instructions: |
  Given a context, a statement can be either:

  - **Definitely correct (Entailment)**; or
  - **Definitely incorrect (Contradiction)**; or
  - **Neither (Neutral)**.

  Your goal is to choose the correct category for a given pair of context and
  statement.

  An automatic detector will estimate your annotation accuracy on this task. If
  your estimated accuracy is too low, you might be **disqualified**.

  If you feel **uncertain** about some examples, just choose the best category you
  believe the statement should be in.

  Examples:

  > **Context:** *A guitarist is playing in a band.*
  > **Statement:** *Some people are performing.*
  > **Answer:** The statement is **definitely correct**.

  [Showcase addition: after choosing your single best category, distribute 100
  points across Entailment / Neutral / Contradiction to express how plausible each
  category is. ChaosNLI collected this distribution by aggregating 100 independent
  single-choice judgments per item; here you report your belief distribution
  directly.]

html_layout: |
  <div style="padding: 15px; max-width: 820px; margin: auto;">
    <div style="background: #f1f5f9; border: 1px solid #cbd5e1; border-radius: 8px; padding: 6px 12px; margin-bottom: 8px; font-size: 13px; color: #475569;">
      <strong>Source:</strong> {{source}}
    </div>
    <div style="background: #eff6ff; border: 1px solid #bfdbfe; border-radius: 8px; padding: 16px; margin-bottom: 8px;">
      <strong style="color: #1e40af;">Context:</strong>
      <p style="font-size: 16px; line-height: 1.6; margin: 8px 0 0 0;">{{context}}</p>
    </div>
    <div style="background: #fffbeb; border: 1px solid #fde68a; border-radius: 8px; padding: 16px;">
      <strong style="color: #92400e;">Statement:</strong>
      <p style="font-size: 16px; line-height: 1.6; margin: 8px 0 0 0;">{{statement}}</p>
    </div>
  </div>

allow_all_users: true
instances_per_annotator: 50
annotation_per_instance: 3
allow_skip: true