Skip to content
intermediatetext

GoEmotions - Fine-Grained Emotion Classification

GoEmotions is a fine-grained emotion classification dataset built by Dorottya Demszky and colleagues at Google Research and published at ACL 2020.

Multi-label emotion classification with 27 emotion categories plus neutral, based on the Google Research GoEmotions dataset (Demszky et al., ACL 2020). Taxonomy covers 12 positive, 11 negative, and 4 ambiguous emotions designed for Reddit comment analysis.

Try it live — no install

Boot the real Potato server in your browser (WebAssembly) and annotate with this exact config. Nothing leaves your machine.

Run live in your browser

Get This Design

View on GitHub(opens in a new tab)

Clone or download from the repository

Quick start:

bash
git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/text/emotion-sentiment/goemotions
potato start config.yaml

About this dataset

GoEmotions is a fine-grained emotion classification dataset built by Dorottya Demszky and colleagues at Google Research and published at ACL 2020. The authors wanted a large, manually annotated resource with a richer set of emotion labels than the handful used in most sentiment work, so that models could be trained and evaluated on subtle emotional distinctions.

The data is 58,009 English comments sampled from popular Reddit subreddits. Each comment was labeled for the emotions the author expresses, drawn from a taxonomy of 27 emotions plus a Neutral option. Annotators could assign more than one emotion to a comment, since a single message often carries several feelings at once.

The 27 emotions are grouped by sentiment into 12 positive, 11 negative, and 4 ambiguous categories. Each comment was rated by several annotators, and the authors report that 94% of examples have at least two raters agreeing on at least one emotion label; a filtered training split of 43,410 examples keeps only comments with rater agreement. A BERT baseline reaches an average F1 of .46 across the taxonomy.

The Potato config below reproduces this task with a multiselect scheme carrying all 27 emotion labels so annotators can mark every emotion present, plus a radio scheme that flags whether a comment is emotionally neutral. It suits anyone building an emotion or affect classifier who needs multi-label judgments on short, informal text.

Comments
58,009 English Reddit comments
Emotion categories
27 emotions + Neutral
Sentiment grouping
12 positive, 11 negative, 4 ambiguous
Inter-rater agreement
94% of examples have 2+ raters agreeing on a label
Filtered training set
43,410 examples
BERT baseline
Average F1 = .46

Configuration File

This Potato config reproduces the annotation task. Save it as config.yaml and run potato start config.yaml to try it.

yaml
# GoEmotions - Fine-Grained Emotion Classification
# Based on Demszky et al., ACL 2020
# Paper: https://aclanthology.org/2020.acl-main.372/
# Dataset: https://github.com/google-research/google-research/tree/master/goemotions
#
# Taxonomy: 27 emotions organized by sentiment:
# - Positive (12): admiration, amusement, approval, caring, desire, excitement,
#                  gratitude, joy, love, optimism, pride, relief
# - Negative (11): anger, annoyance, disappointment, disapproval, disgust,
#                  embarrassment, fear, grief, nervousness, remorse, sadness
# - Ambiguous (4): confusion, curiosity, realization, surprise
# - Plus: neutral
#
# Guidelines:
# - Select ALL emotions expressed in the text (multi-label)
# - Focus on emotion expressed by author, not evoked in reader
# - Related emotions differ by intensity (annoyance -> anger)

annotation_task_name: "GoEmotions: Fine-Grained Emotion Classification"
task_dir: "."

data_files:
  - sample-data.json
item_properties:
  id_key: "id"
  text_key: "text"

output_annotation_dir: "annotation_output/"
export_annotation_format: "jsonl"

annotation_schemes:
  - annotation_type: multiselect
    name: emotions
    description: "Select ALL emotions expressed in this text (multiple selections allowed)"
    labels:
      # Positive emotions (12)
      - name: Admiration
        tooltip: "Finding something impressive or worthy of respect"
        key_value: "1"
      - name: Amusement
        tooltip: "Finding something funny or being entertained"
        key_value: "2"
      - name: Approval
        tooltip: "Having or expressing a favorable opinion"
        key_value: "3"
      - name: Caring
        tooltip: "Displaying kindness and concern for others"
        key_value: "4"
      - name: Desire
        tooltip: "A strong feeling of wanting something"
      - name: Excitement
        tooltip: "Feeling very enthusiastic and eager"
      - name: Gratitude
        tooltip: "Being thankful, ready to show appreciation"
      - name: Joy
        tooltip: "A feeling of great pleasure and happiness"
        key_value: "5"
      - name: Love
        tooltip: "A strong positive feeling of affection"
        key_value: "6"
      - name: Optimism
        tooltip: "Hopefulness and confidence about the future"
      - name: Pride
        tooltip: "Pleasure or satisfaction from achievements"
      - name: Relief
        tooltip: "Reassurance and relaxation after anxiety"
      # Negative emotions (11)
      - name: Anger
        tooltip: "A strong feeling of displeasure or hostility"
        key_value: "7"
      - name: Annoyance
        tooltip: "Slight irritation or frustration"
      - name: Disappointment
        tooltip: "Sadness caused by unfulfilled expectations"
      - name: Disapproval
        tooltip: "Possession or expression of an unfavorable opinion"
      - name: Disgust
        tooltip: "A strong feeling of aversion or revulsion"
      - name: Embarrassment
        tooltip: "Self-conscious awkwardness or shame"
      - name: Fear
        tooltip: "Being afraid or worried about something"
        key_value: "9"
      - name: Grief
        tooltip: "Deep sorrow, especially caused by loss"
      - name: Nervousness
        tooltip: "Easily agitated or worried"
      - name: Remorse
        tooltip: "Deep regret or guilt for wrongdoing"
      - name: Sadness
        tooltip: "Feeling or showing sorrow; unhappiness"
        key_value: "8"
      # Ambiguous emotions (4)
      - name: Confusion
        tooltip: "Lack of understanding; uncertainty"
      - name: Curiosity
        tooltip: "A strong desire to know or learn something"
      - name: Realization
        tooltip: "Becoming fully aware of something"
      - name: Surprise
        tooltip: "Feeling of astonishment or shock"
        key_value: "0"

  - annotation_type: radio
    name: is_neutral
    description: "Is this text emotionally neutral?"
    labels:
      - name: "Contains emotion(s)"
        key_value: "e"
      - name: "Neutral (no emotion)"
        key_value: "n"

user_config:
  allow_all_users: true
max_annotations_per_user: 200
num_annotators_per_item: 3

Sample Data

json
[
  {
    "id": "goemotions_001",
    "text": "This is absolutely incredible! I can't believe how talented you are. Keep up the amazing work!"
  },
  {
    "id": "goemotions_002",
    "text": "Ugh, I've been waiting for 3 hours and still nothing. This is so frustrating."
  }
]

// ... and 10 more items

Dataset & paper

Demszky et al., ACL 2020

Citation (BibTeX)

bibtex
@inproceedings{demszky-etal-2020-goemotions,
    title = "{G}o{E}motions: A Dataset of Fine-Grained Emotions",
    author = "Demszky, Dorottya and Movshovitz-Attias, Dana and Ko, Jeongwoo and Cowen, Alan and Nemade, Gaurav and Ravi, Sujith",
    booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics",
    month = jul,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2020.acl-main.372",
    pages = "4040--4054"
}

Details

Annotation Types

multiselectradio

Domain

NLPSocial Media

Use Cases

Emotion DetectionSentiment AnalysisSocial Media Analysis

Tags

emotionmulti-labelredditfine-grainedgoemotionsacl2020

Found an issue or want to improve this design?

Open an Issue(opens in a new tab)