Skip to content
intermediatetext

Fact Verification

Verify claims as supported, refuted, or not enough information based on provided evidence.

Try it live — no install

Boot the real Potato server in your browser (WebAssembly) and annotate with this exact config. Nothing leaves your machine.

Run live in your browser

Get This Design

View on GitHub(opens in a new tab)

Clone or download from the repository

Quick start:

bash
git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/templates/text/fact-verification
potato start config.yaml

Configuration File

yaml
# Fact Verification Configuration
# Verify claims against provided evidence

task_dir: "."
annotation_task_name: "Fact Verification"

data_files:
  - "data/claims.json"

item_properties:
  id_key: "id"
  text_key: "display"
  text_display_key: "display"

user_config:
  allow_all_users: true

annotation_schemes:
  - annotation_type: "radio"
    name: "verdict"
    description: "Based on the evidence, is the claim..."
    labels:
      - name: "Supported"
        tooltip: "Evidence confirms the claim is true"
        key_value: "s"
        color: "#22c55e"
      - name: "Refuted"
        tooltip: "Evidence contradicts the claim"
        key_value: "r"
        color: "#ef4444"
      - name: "Not Enough Info"
        tooltip: "Evidence doesn't clearly support or refute"
        key_value: "n"
        color: "#eab308"

  - annotation_type: "text"
    name: "evidence_quote"
    description: "Quote the most relevant part of the evidence"

  - annotation_type: "likert"
    name: "confidence"
    description: "How confident are you in your verdict?"
    size: 5
    min_label: "Not confident"
    max_label: "Very confident"

  - annotation_type: "multiselect"
    name: "reasoning_type"
    description: "What type of reasoning did you use?"
    labels:
      - name: "Direct statement"
      - name: "Numerical/Statistical"
      - name: "Logical inference"
      - name: "Temporal reasoning"
      - name: "Multiple sources"

  - annotation_type: "text"
    name: "explanation"
    description: "Explain your reasoning"
    required: false

output: "annotation_output/"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

Sample Data

json
[
  {
    "id": "fact_001",
    "claim": "The Eiffel Tower is the tallest structure in France.",
    "evidence": "The Eiffel Tower stands at 330 meters including antennas. The Millau Viaduct, also in France, reaches 343 meters at its highest point, making it the tallest structure in France.",
    "display": "**CLAIM:** The Eiffel Tower is the tallest structure in France.\n\n**EVIDENCE:** The Eiffel Tower stands at 330 meters including antennas. The Millau Viaduct, also in France, reaches 343 meters at its highest point, making it the tallest structure in France."
  },
  {
    "id": "fact_002",
    "claim": "Python is the most popular programming language in 2024.",
    "evidence": "According to the TIOBE Index from January 2024, Python holds the top position with a rating of 13.97%, followed by C at 11.44% and C++ at 10.66%. The PYPL index also ranks Python first based on Google search trends for language tutorials.",
    "display": "**CLAIM:** Python is the most popular programming language in 2024.\n\n**EVIDENCE:** According to the TIOBE Index from January 2024, Python holds the top position with a rating of 13.97%, followed by C at 11.44% and C++ at 10.66%. The PYPL index also ranks Python first based on Google search trends for language tutorials."
  }
]

// ... and 1 more items

Details

Annotation Types

likertmultiselectradiotext

Domain

nlpfact-checking

Use Cases

fact-checkingmisinformation-detection

Tags

fact-checkverificationclaimsevidencenlp

Found an issue or want to improve this design?

Open an Issue(opens in a new tab)

Related Designs

Intent Classification

Classify user utterances into intents for chatbot and virtual assistant training.

likertmultiselect

AgentBoard Progress Scoring

Assess multi-turn LLM agent progress by identifying achieved milestones, scoring overall progress, categorizing the agent environment, and noting partial progress observations.

multiselectlikert

Audio Transcription Review

Review and correct automatic speech recognition transcriptions with waveform visualization.

likertmultiselect