Skip to content
Showcase/Social Chemistry 101 (Social Norms)
advancedsurvey

Social Chemistry 101 (Social Norms)

Annotate rules-of-thumb for social and moral norms. Based on Forbes et al., EMNLP 2020. Capture 12 dimensions of social judgment including cultural pressure, moral foundations, and legality.

About this dataset

Social Chemistry 101 is a large resource for reasoning about social and moral norms, assembled by Maxwell Forbes, Jena D. Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi and published at EMNLP 2020. The goal was to catalog the everyday rules of thumb people use to judge behavior and to make those judgments learnable by models.

The core unit is a rule of thumb, a short descriptive norm such as "It is rude to run a blender at 5am," attached to a real-life situation. Situations were drawn from four sources: Reddit's Am I the Asshole forum, the Dear Abby advice column, ROCStories, and a Confessions dataset. Crowd workers wrote rules of thumb for each situation and then annotated each one along 12 dimensions of judgment covering social acceptability, moral foundations, expected cultural pressure, and assumed legality.

The released corpus, SOCIAL-CHEM-101, contains 292,000 rules of thumb over 104,000 situations, with 365,000 sets of structural annotations that break each rule into its 12 dimensions. Altogether it holds more than 4.5 million categorical and free-text annotations.

The Potato config below reproduces a slice of this scheme: a radio scale for the social judgment of an action, a radio scheme for its primary moral foundation, a Likert scale for cultural pressure, a radio scheme for legality, and a Likert scale for how much others would agree with the judgment. It fits projects that need structured human norm judgments over described situations.

Rules-of-thumb
292,000
Situations
104,000 real-life situations
Dimensions per rule-of-thumb
12
Structural annotation sets
365,000
Total annotations
4.5 million+ (categorical + free-text)
Situation sources
AITA (Reddit), Dear Abby, ROCStories, Confessions
Q1: Rate your experience12345Q2: Primary use case?ResearchIndustryEducationQ3: Additional feedback

Configuration Fileconfig.yaml

This Potato config reproduces the annotation task. Save it as config.yaml and run potato start config.yaml to try it.

yaml
# Social Chemistry 101 (Social Norms)
# Based on Forbes et al., EMNLP 2020
# Paper: https://aclanthology.org/2020.emnlp-main.48/
#
# Social Chemistry catalogs rules-of-thumb (RoTs) for social/moral norms.
# Each RoT is annotated across 12 dimensions capturing different aspects
# of social judgment.
#
# Key Concepts:
# - Rule-of-Thumb (RoT): A descriptive norm like "It's rude to be late"
# - Situation: The context that evokes the RoT
# - Action: The specific behavior being judged
#
# 12 Annotation Dimensions:
# 1. Social judgment: Is the action good/bad/expected/unexpected?
# 2. Moral foundations: Care, fairness, loyalty, authority, purity
# 3. Cultural pressure: How strongly does society expect this?
# 4. Legality: Is this legal/illegal?
# 5. Agency: Who should do/avoid the action?
# 6. Anticipated agreement: Would others agree with this judgment?
#
# Annotation Guidelines:
# 1. Consider the SPECIFIC situation, not abstract principles
# 2. Think about what MOST people in your culture would think
# 3. RoTs can conflict - the same action may have multiple valid judgments
# 4. Consider both the action and its consequences

annotation_task_name: "Social Chemistry - Social Norms"
task_dir: "."

data_files:
  - sample-data.json
item_properties:
  id_key: "id"
  text_key: "situation"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

annotation_schemes:
  # Step 1: Social judgment
  - annotation_type: radio
    name: judgment
    description: "How would most people judge this action?"
    labels:
      - "Very bad"
      - "Bad"
      - "Expected/okay"
      - "Good"
      - "Very good"
    tooltips:
      "Very bad": "Strongly condemned by society"
      "Bad": "Generally disapproved of"
      "Expected/okay": "Neutral, normal, or expected behavior"
      "Good": "Generally approved of"
      "Very good": "Strongly praised by society"

  # Step 2: Moral foundation
  - annotation_type: radio
    name: moral_foundation
    description: "Which moral foundation does this primarily relate to?"
    labels:
      - "Care/Harm"
      - "Fairness/Cheating"
      - "Loyalty/Betrayal"
      - "Authority/Subversion"
      - "Purity/Degradation"
      - "Liberty/Oppression"
      - "None/Other"
    tooltips:
      "Care/Harm": "About protecting others from harm, showing compassion"
      "Fairness/Cheating": "About justice, rights, reciprocity"
      "Loyalty/Betrayal": "About group membership, patriotism, self-sacrifice"
      "Authority/Subversion": "About respect for tradition, hierarchy, duty"
      "Purity/Degradation": "About cleanliness, sanctity, disgust"
      "Liberty/Oppression": "About freedom, autonomy, avoiding domination"
      "None/Other": "Doesn't clearly map to moral foundations"

  # Step 3: Cultural pressure
  - annotation_type: likert
    name: cultural_pressure
    description: "How strongly does society expect/pressure this behavior?"
    min_value: 1
    max_value: 5
    labels:
      1: "Very weak pressure"
      2: "Weak pressure"
      3: "Moderate pressure"
      4: "Strong pressure"
      5: "Very strong pressure"
    tooltips:
      1: "Society rarely enforces this norm"
      2: "Mild social expectation"
      3: "Moderate social expectation"
      4: "Strong social expectation with consequences"
      5: "Violating this would be severely condemned"

  # Step 4: Legality
  - annotation_type: radio
    name: legality
    description: "Is this action legal or illegal?"
    labels:
      - "Clearly legal"
      - "Legal but regulated"
      - "Gray area"
      - "Illegal (minor)"
      - "Illegal (serious)"
    tooltips:
      "Clearly legal": "No legal issues"
      "Legal but regulated": "Legal with some restrictions"
      "Gray area": "Legal status unclear or varies"
      "Illegal (minor)": "Minor infraction or misdemeanor"
      "Illegal (serious)": "Serious crime or felony"

  # Step 5: Anticipated agreement
  - annotation_type: likert
    name: agreement
    description: "How much would others agree with your judgment?"
    min_value: 1
    max_value: 5
    labels:
      1: "Very controversial"
      2: "Somewhat controversial"
      3: "Mixed opinions"
      4: "Mostly agree"
      5: "Nearly everyone agrees"

allow_all_users: true
instances_per_annotator: 50
annotation_per_instance: 3
allow_skip: true
skip_reason_required: false

Sample Datasample-data.json

json
[
  {
    "id": "sc_001",
    "situation": "Someone cuts in line at a busy coffee shop.",
    "action": "Cutting in line"
  },
  {
    "id": "sc_002",
    "situation": "A person helps an elderly stranger carry their groceries to the car.",
    "action": "Helping a stranger"
  }
]

// ... and 8 more items

Try it live — no install

Boot the real Potato server in your browser (WebAssembly) and annotate with this exact config. Nothing leaves your machine.

▶ Run live in your browser

Get This Design

View on GitHub

Clone or download from the repository

Quick start:

git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/text/commonsense-ethics/social-chemistry
potato start config.yaml

Dataset & paper

Forbes et al., EMNLP 2020

Citation (BibTeX)

bibtex
@inproceedings{forbes-etal-2020-social,
    title = "Social Chemistry 101: Learning to Reason about Social and Moral Norms",
    author = "Forbes, Maxwell and Hwang, Jena D. and Shwartz, Vered and Sap, Maarten and Choi, Yejin",
    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2020.emnlp-main.48",
    pages = "653--670"
}

Details

Annotation Types

radiolikert

Domain

NLPEthicsCommonsense

Use Cases

Social NormsMoral ReasoningCommonsense AI

Tags

social-normsethicsrules-of-thumbemnlp2020moral-foundations

Found an issue or want to improve this design?

Open an Issue