Showcase/Fact Verification
intermediatetext

Fact Verification

Verify claims as supported, refuted, or not enough information based on provided evidence.

📝

text annotation

Configuration Fileconfig.yaml

# Fact Verification Configuration
# Verify claims against provided evidence

annotation_task_name: "Fact Verification"

data_files:
  - "data/claims.json"

item_properties:
  id_key: "id"
  text_display_key: "display"

user_config:
  allow_all_users: true

annotation_schemes:
  - annotation_type: "radio"
    name: "verdict"
    description: "Based on the evidence, is the claim..."
    labels:
      - name: "Supported"
        tooltip: "Evidence confirms the claim is true"
        key_value: "s"
        color: "#22c55e"
      - name: "Refuted"
        tooltip: "Evidence contradicts the claim"
        key_value: "r"
        color: "#ef4444"
      - name: "Not Enough Info"
        tooltip: "Evidence doesn't clearly support or refute"
        key_value: "n"
        color: "#eab308"

  - annotation_type: "text"
    name: "evidence_quote"
    description: "Quote the most relevant part of the evidence"

  - annotation_type: "likert"
    name: "confidence"
    description: "How confident are you in your verdict?"
    size: 5
    min_label: "Not confident"
    max_label: "Very confident"

  - annotation_type: "multiselect"
    name: "reasoning_type"
    description: "What type of reasoning did you use?"
    labels:
      - name: "Direct statement"
      - name: "Numerical/Statistical"
      - name: "Logical inference"
      - name: "Temporal reasoning"
      - name: "Multiple sources"

  - annotation_type: "text"
    name: "explanation"
    description: "Explain your reasoning"
    required: false

output: "annotation_output/"

Sample Datasample-data.json

[
  {
    "id": "fact_001",
    "claim": "The Eiffel Tower is the tallest structure in France.",
    "evidence": "The Eiffel Tower stands at 330 meters including antennas. The Millau Viaduct, also in France, reaches 343 meters at its highest point, making it the tallest structure in France.",
    "display": "**CLAIM:** The Eiffel Tower is the tallest structure in France.\n\n**EVIDENCE:** The Eiffel Tower stands at 330 meters including antennas. The Millau Viaduct, also in France, reaches 343 meters at its highest point, making it the tallest structure in France."
  },
  {
    "id": "fact_002",
    "claim": "Python is the most popular programming language in 2024.",
    "evidence": "According to the TIOBE Index from January 2024, Python holds the top position with a rating of 13.97%, followed by C at 11.44% and C++ at 10.66%. The PYPL index also ranks Python first based on Google search trends for language tutorials.",
    "display": "**CLAIM:** Python is the most popular programming language in 2024.\n\n**EVIDENCE:** According to the TIOBE Index from January 2024, Python holds the top position with a rating of 13.97%, followed by C at 11.44% and C++ at 10.66%. The PYPL index also ranks Python first based on Google search trends for language tutorials."
  }
]

// ... and 1 more items

Get This Design

View on GitHub

Clone or download from the repository

Quick start:

git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/fact-verification
potato start config.yaml

Details

Annotation Types

radiotext

Domain

nlpfact-checking

Use Cases

fact-checkingmisinformation-detection

Tags

fact-checkverificationclaimsevidencenlp

Found an issue or want to improve this design?

Open an Issue

Related Designs

Reading Comprehension QA

Evaluate question-answer pairs for reading comprehension by verifying answers and rating quality.

radiospan

Sarcasm Detection

Identify sarcastic statements and label their type and target in social media and conversational text.

radiotext

Commonsense Inference (ATOMIC 2020)

Annotate commonsense inferences about events, mental states, and social interactions. Based on ATOMIC 2020 (Hwang et al., AAAI 2021). Generate if-then knowledge about causes, effects, intents, and reactions.

radiotext