ATOMIC: If-Then Commonsense Reasoning (PersonX / PersonY)
ATOMIC is the AAAI 2019 knowledge graph of 877k if-then commonsense inferences over 24,313 PersonX/PersonY base events (Sap et al.). This Potato config reproduces its 9-dimension free-text annotation task: xIntent, xNeed, xAttr, xEffect, xReact, xWant, oReact, oWant, oEffect.
About this dataset
ATOMIC (An Atlas of Machine Commonsense) is a knowledge graph of everyday if-then reasoning, introduced by Maarten Sap, Ronan LeBras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi at AAAI 2019. Rather than encoding taxonomic facts like a traditional knowledge base, ATOMIC organizes inferential knowledge: given an event, what is likely to happen before and after, and what are the mental states of the people involved.
The graph holds 877,108 if-then triples over 24,313 base events. Each base event is written with PersonX, PersonY, and PersonZ placeholders (for example, 'PersonX pays PersonY a compliment'), so the inferences generalize across actors. Events are connected to inferences through nine typed relations grouped into three families: If-Event-Then-Mental-State (xIntent, xReact, oReact), If-Event-Then-Event (xNeed, xWant, xEffect, oWant, oEffect), and If-Event-Then-Persona (xAttr).
ATOMIC was built by crowdsourcing free-text inferences on Amazon Mechanical Turk. For each base event and dimension, workers wrote short natural-language phrases (up to four answers per dimension) or marked the dimension 'none' when no inference applied. The 'x' relations describe PersonX, the subject of the event, while the 'o' relations describe others such as PersonY. This free-text design is what makes ATOMIC a generation target: COMET and later commonsense models are trained to produce these inferences.
The Potato config below reproduces the ATOMIC annotation task. Each base event is shown with nine free-text boxes, one per dimension, each prompting the annotator to complete a templated phrase (for example, xIntent -> 'PersonX wanted ...', oReact -> 'Others feel ...'). Pooling several annotators per event with annotation_per_instance recovers the many-answers-per-event structure of the original dataset.
- Base events
- 24,313 (PersonX/PersonY templates)
- If-then triples
- 877,108
- Inference dimensions
- 9 (xIntent, xNeed, xAttr, xEffect, xReact, xWant, oReact, oWant, oEffect)
- Relation families
- 3 (mental-state, event, persona)
- Annotation
- Free-text on Amazon Mechanical Turk
- Venue
- AAAI 2019
Configuration Fileconfig.yaml
This Potato config reproduces the annotation task. Save it as config.yaml and run potato start config.yaml to try it.
# ATOMIC: An Atlas of Machine Commonsense for If-Then Reasoning
# Based on Sap et al., AAAI 2019
# Paper: https://arxiv.org/abs/1811.00146
# Data: https://homes.cs.washington.edu/~msap/atomic/
#
# ATOMIC is a knowledge graph of 877k if-then commonsense inferences over
# 24,313 everyday base events written with PersonX / PersonY / PersonZ
# placeholders (e.g. "PersonX pays PersonY a compliment"). Crowdworkers on
# Amazon Mechanical Turk wrote free-text inferences for each event across
# nine dimensions, grouped into three families:
#
# If-Event-Then-Mental-State
# xIntent - why PersonX does this (intent)
# xReact - how PersonX feels afterward
# oReact - how others (PersonY) feel afterward
#
# If-Event-Then-Event
# xNeed - what PersonX needs to do before
# xWant - what PersonX wants to do afterward (voluntary)
# xEffect - what happens to PersonX afterward (involuntary)
# oWant - what others want to do afterward (voluntary)
# oEffect - what happens to others afterward (involuntary)
#
# If-Event-Then-Persona
# xAttr - how PersonX is described / perceived
#
# Reproduction notes:
# - Each box accepts a short free-text phrase that completes the prompt.
# In the original HIT, workers gave up to four answers per dimension and
# could mark a dimension "none" when no inference applied. Here we elicit
# one phrase per dimension per annotator; pool multiple annotators with
# annotation_per_instance to recover the original many-answers-per-event set.
# - Keep PersonX/PersonY/PersonZ verbatim in answers (do not substitute names),
# matching ATOMIC's templated convention.
# - Write "none" if a dimension does not apply to the event.
annotation_task_name: "ATOMIC - If-Then Commonsense Inference"
task_dir: "."
data_files:
- sample-data.json
item_properties:
id_key: "id"
text_key: "event"
output_annotation_dir: "annotation_output/"
output_annotation_format: "json"
annotation_schemes:
# ---- If-Event-Then-Mental-State -------------------------------------
- name: "xIntent"
description: |
Why does PersonX do this? Complete: "PersonX wanted ..."
(the likely intent behind the event)
annotation_type: text
textarea: false
- name: "xReact"
description: |
How does PersonX feel afterward? Complete: "PersonX feels ..."
(the subject's emotional reaction)
annotation_type: text
textarea: false
- name: "oReact"
description: |
How do others (PersonY) feel afterward? Complete: "Others feel ..."
(others' emotional reaction; write "none" if no one else is involved)
annotation_type: text
textarea: false
# ---- If-Event-Then-Event --------------------------------------------
- name: "xNeed"
description: |
What does PersonX need to do before this event?
Complete: "Before, PersonX needed ..."
annotation_type: text
textarea: false
- name: "xWant"
description: |
What does PersonX want to do afterward (voluntary)?
Complete: "Afterward, PersonX wants ..."
annotation_type: text
textarea: false
- name: "xEffect"
description: |
What happens to PersonX afterward (involuntary)?
Complete: "PersonX then ..."
annotation_type: text
textarea: false
- name: "oWant"
description: |
What do others (PersonY) want to do afterward (voluntary)?
Complete: "Afterward, others want ..." (write "none" if not applicable)
annotation_type: text
textarea: false
- name: "oEffect"
description: |
What happens to others (PersonY) afterward (involuntary)?
Complete: "Others then ..." (write "none" if not applicable)
annotation_type: text
textarea: false
# ---- If-Event-Then-Persona ------------------------------------------
- name: "xAttr"
description: |
How is PersonX described or perceived because of this event?
Complete: "PersonX is seen as ..."
annotation_type: text
textarea: false
# Pool several annotators per event to recover ATOMIC's many-answers-per-event set.
per_annotator_quota:
default: 25
min_annotators_per_instance: 4
Sample Datasample-data.json
[
{
"id": "atomic_001",
"event": "PersonX pays PersonY a compliment"
},
{
"id": "atomic_002",
"event": "PersonX makes PersonY's coffee"
}
]
// ... and 10 more itemsGet This Design
Clone or download from the repository
Quick start:
git clone https://github.com/davidjurgens/potato-showcase.git cd potato-showcase/text/commonsense-ethics/atomic-if-then-reasoning potato start config.yaml
Dataset & paper
Sap et al., AAAI 2019
Citation (BibTeX)
@inproceedings{sap2019atomic,
title={ATOMIC: An Atlas of Machine Commonsense for If-Then Reasoning},
author={Sap, Maarten and Le Bras, Ronan and Allaway, Emily and Bhagavatula, Chandra and Lourie, Nicholas and Rashkin, Hannah and Roof, Brendan and Smith, Noah A. and Choi, Yejin},
booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
volume={33},
number={01},
pages={3027--3035},
year={2019},
url={https://arxiv.org/abs/1811.00146}
}Details
Annotation Types
Domain
Use Cases
Tags
Found an issue or want to improve this design?
Open an IssueRelated Designs
Argument Reasoning in Civil Procedure
Legal argument reasoning task requiring annotators to answer multiple-choice questions about civil procedure by selecting the best answer and providing legal reasoning. Based on SemEval-2024 Task 5.
BRAINTEASER - Commonsense-Defying QA
Lateral thinking and commonsense-defying question answering task requiring annotators to select answers to brain teasers that defy default commonsense assumptions and provide explanations. Based on SemEval-2024 Task 9 (BRAINTEASER).
Check-COVID: Fact-Checking COVID-19 News Claims
Fact-checking COVID-19 news claims. Annotators verify claims against evidence, identify supporting/refuting spans, and provide verdicts with explanations. Based on the Check-COVID dataset targeting misinformation during the pandemic.