Skip to content
beginnertext

SNLI: Stanford Natural Language Inference Corpus

SNLI is a corpus of 570k human-written English sentence pairs labeled entailment, contradiction, or neutral (Bowman et al., EMNLP 2015). This Potato config reproduces the premise-hypothesis judgment task.

About this dataset

The Stanford Natural Language Inference (SNLI) corpus was released by Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning at EMNLP 2015. It became a standard benchmark for training and evaluating sentence-pair reasoning models.

The corpus holds 570,000 human-written English sentence pairs. Each premise is an image caption drawn from the Flickr30k corpus, and crowd workers wrote a hypothesis for each of the three target labels given that premise.

Annotators read a premise and a hypothesis, then pick one of three relations: entailment (the hypothesis follows from the premise), contradiction (the hypothesis conflicts with it), or neutral (neither follows nor conflicts). About 10% of the corpus (~56,900 pairs) was relabeled by four additional workers to create validated gold labels by majority vote; the remaining pairs carry the single label from their hypothesis author.

The Potato config below reproduces this judgment task: it shows a premise and hypothesis, asks the annotator to select entailment, contradiction, or neutral, and collects an optional free-text justification.

Sentence pairs
570,000
Labels
entailment, contradiction, neutral
Premise source
Flickr30k image captions
Validation
~10% subset relabeled by 4 extra workers
Published
Bowman et al., EMNLP 2015
Language
English
Submit

Configuration Fileconfig.yaml

This Potato config reproduces the annotation task. Save it as config.yaml and run potato start config.yaml to try it.

yaml
# SNLI - Textual Entailment
# Based on Bowman et al., EMNLP 2015
# Paper: https://aclanthology.org/D15-1075/
# Dataset: https://nlp.stanford.edu/projects/snli/
#
# This task presents a premise sentence and a hypothesis sentence.
# Annotators determine the inferential relationship between them:
# entailment, contradiction, or neutral.
#
# Label Definitions:
# - Entailment: The hypothesis is definitely true given the premise
# - Contradiction: The hypothesis is definitely false given the premise
# - Neutral: The hypothesis may or may not be true given the premise
#
# Annotation Guidelines:
# 1. Read the premise sentence carefully
# 2. Read the hypothesis sentence
# 3. Determine the relationship between them
# 4. Provide a brief reasoning for your choice

annotation_task_name: "SNLI - Textual Entailment"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "text"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

port: 8000
server_name: localhost

annotation_schemes:
  # Step 1: Classify the relationship
  - annotation_type: radio
    name: nli_label
    description: "What is the inferential relationship between the premise and hypothesis?"
    labels:
      - "Entailment"
      - "Contradiction"
      - "Neutral"
    keyboard_shortcuts:
      "Entailment": "1"
      "Contradiction": "2"
      "Neutral": "3"
    tooltips:
      "Entailment": "The hypothesis is definitely true given the premise"
      "Contradiction": "The hypothesis is definitely false given the premise"
      "Neutral": "The hypothesis may or may not be true given the premise"

  # Step 2: Provide reasoning
  - annotation_type: text
    name: reasoning
    description: "Briefly explain your reasoning for the chosen label"

annotation_instructions: |
  You will be shown two sentences: a premise and a hypothesis. Your task is to:
  1. Determine the relationship between the premise and the hypothesis.
  2. Select one of: Entailment, Contradiction, or Neutral.
  3. Provide a brief explanation for your choice.

  - Entailment: If the premise is true, the hypothesis must also be true.
  - Contradiction: If the premise is true, the hypothesis must be false.
  - Neutral: The premise does not give enough information to determine if the hypothesis is true or false.

html_layout: |
  <div style="padding: 15px; max-width: 800px; margin: auto;">
    <div style="background: #f0f9ff; border: 1px solid #bae6fd; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
      <strong style="color: #0369a1; font-size: 14px; text-transform: uppercase; letter-spacing: 0.5px;">Premise:</strong>
      <p style="font-size: 16px; line-height: 1.7; margin: 8px 0 0 0;">{{text}}</p>
    </div>
    <div style="background: #fefce8; border: 1px solid #fde68a; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
      <strong style="color: #a16207; font-size: 14px; text-transform: uppercase; letter-spacing: 0.5px;">Hypothesis:</strong>
      <p style="font-size: 16px; line-height: 1.7; margin: 8px 0 0 0;">{{hypothesis}}</p>
    </div>
  </div>

allow_all_users: true
instances_per_annotator: 50
annotation_per_instance: 2
allow_skip: true
skip_reason_required: false

Sample Datasample-data.json

json
[
  {
    "id": "snli_001",
    "text": "A man wearing a hard hat is dancing on a street corner.",
    "hypothesis": "A man is outdoors."
  },
  {
    "id": "snli_002",
    "text": "Two women are embracing while holding to-go packages.",
    "hypothesis": "The women are fighting each other."
  }
]

// ... and 8 more items

Get This Design

View on GitHub

Clone or download from the repository

Quick start:

git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/text/natural-language-inference/snli-textual-entailment
potato start config.yaml

Dataset & paper

Bowman et al., EMNLP 2015

Official dataset ↗Read the paper ↗

Citation (BibTeX)

bibtex
@inproceedings{bowman-etal-2015-large,
    title = "A large annotated corpus for learning natural language inference",
    author = "Bowman, Samuel R.  and Angeli, Gabor  and Potts, Christopher  and Manning, Christopher D.",
    booktitle = "Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing",
    month = sep,
    year = "2015",
    address = "Lisbon, Portugal",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/D15-1075",
    pages = "632--642"
}

Details

Annotation Types

radiotext

Domain

NLPNatural Language Inference

Use Cases

Textual EntailmentSemantic ReasoningSentence Pair Classification

Tags

snlinlientailmentcontradictionsentence-pairsemnlp2015

Found an issue or want to improve this design?

Open an Issue

Related Designs

MultiNLI - Multi-Genre Natural Language Inference

Natural language inference across multiple genres of text, based on the Multi-Genre NLI corpus (Williams et al., NAACL 2018). Annotators classify premise-hypothesis relationships with genre-diverse examples from fiction, government, travel, and more.

radiotext

Safe Biomedical NLI

Safe biomedical natural language inference task requiring annotators to determine entailment or contradiction between statements and clinical trial report sections and provide reasoning. Based on SemEval-2024 Task 2 (NLI4CT: Safe Biomedical Natural Language Inference for Clinical Trials).

radiotext

BRAINTEASER - Commonsense-Defying QA

Lateral thinking and commonsense-defying question answering task requiring annotators to select answers to brain teasers that defy default commonsense assumptions and provide explanations. Based on SemEval-2024 Task 9 (BRAINTEASER).

radiotext