Showcase/SemEval-2021 Task 7: HaHackathon Humor Detection

beginnersurvey

SemEval-2021 Task 7: HaHackathon Humor Detection

SemEval-2021 Task 7 (HaHackathon) detects whether a text is humorous and rates how offensive it is, using 10,000 tweets and short jokes. This Potato config reproduces the humor and offense annotation interface.

About this dataset

HaHackathon was Task 7 at SemEval-2021, the 15th International Workshop on Semantic Evaluation. It was organized by J. A. Meaney, Steven Wilson, Luis Chiruzzo, Adam Lopez, and Walid Magdy, and was the first shared task to combine humor detection and offense detection in one dataset.

The dataset contains 10,000 texts drawn from Twitter and the Kaggle Short Jokes dataset. Each text was annotated by 20 people aged 18 to 70, and the data was split into 8,000 training, 1,000 development, and 1,000 test instances.

The task had four subtasks. Subtask 1a is binary humor detection (funny or not). Subtask 1b is humor rating as a regression value. Subtask 1c predicts whether a text is controversial based on the variance in its humor ratings. Subtask 2a rates how offensive a text is on a scale from 1 to 5. Across the subtasks the organizers received 36 to 58 submissions.

The Potato config below reproduces the core annotation flow: a binary radio choice for whether the text is funny, followed by a 5-point Likert scale rating how offensive it is, treating humor and offense as independent dimensions.

Total texts: 10,000
Train / Dev / Test: 8,000 / 1,000 / 1,000
Annotators per text: 20 (aged 18-70)
Offense rating scale: 1 to 5
Source: Twitter + Kaggle Short Jokes
Venue: SemEval-2021, Task 7

Configuration Fileconfig.yaml

This Potato config reproduces the annotation task. Save it as config.yaml and run potato start config.yaml to try it.

yaml

# HaHackathon - Detecting and Rating Humor and Offense
# Based on Meaney et al., SemEval 2021
# Paper: https://aclanthology.org/2021.semeval-1.9/
# Dataset: https://competitions.codalab.org/competitions/27446
#
# Annotators first decide whether a text is funny, then rate how
# offensive it is on a 5-point scale from "Not Offensive" to "Very Offensive".

annotation_task_name: "HaHackathon - Detecting and Rating Humor and Offense"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "text"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

port: 8000
server_name: localhost

annotation_schemes:
  - annotation_type: radio
    name: humor_judgment
    description: "Is this text funny?"
    labels:
      - "Funny"
      - "Not Funny"
    keyboard_shortcuts:
      "Funny": "1"
      "Not Funny": "2"
    tooltips:
      "Funny": "The text is intended to be humorous and succeeds in being funny"
      "Not Funny": "The text is not funny or the humor attempt fails"

  - annotation_type: likert
    name: offense_rating
    description: "Rate how offensive this text is."
    min_label: "Not Offensive"
    max_label: "Very Offensive"
    size: 5

annotation_instructions: |
  You will see a short text that may or may not be humorous. Your task is to:
  1. Read the text carefully.
  2. Decide whether the text is funny or not funny.
  3. Rate how offensive the text is on a scale from "Not Offensive" to "Very Offensive".

  Note that humor and offense are independent dimensions - a text can be funny
  and offensive, funny and inoffensive, or not funny but still offensive.

html_layout: |
  <div style="padding: 15px; max-width: 800px; margin: auto;">
    <div style="background: #f0f9ff; border: 1px solid #bae6fd; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
      <strong style="color: #0369a1;">Text:</strong>
      <p style="font-size: 16px; line-height: 1.7; margin: 8px 0 0 0;">{{text}}</p>
    </div>
  </div>

allow_all_users: true
instances_per_annotator: 50
annotation_per_instance: 2
allow_skip: true
skip_reason_required: false

Sample Datasample-data.json

json

[
  {
    "id": "humor_001",
    "text": "I told my wife she was drawing her eyebrows too high. She looked surprised."
  },
  {
    "id": "humor_002",
    "text": "The weather forecast calls for rain tomorrow and continued disappointment throughout the week."
  }
]

// ... and 8 more items

Get This Design

View on GitHub

Clone or download from the repository

Quick start:

git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/semeval/2021/task07-hahackathon-humor
potato start config.yaml

Dataset & paper

Meaney et al., SemEval 2021

Official dataset ↗Read the paper ↗

Citation (BibTeX)

bibtex

@inproceedings{meaney-etal-2021-semeval,
    title = "{S}em{E}val-2021 {T}ask 7: {H}a{H}ackathon, Detecting and Rating Humor and Offense",
    author = "Meaney, J. A. and Wilson, Steven and Chiruzzo, Luis and Lopez, Adam and Magdy, Walid",
    booktitle = "Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)",
    year = "2021",
    pages = "105--119",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.semeval-1.9"
}

Details

Annotation Types

radiolikert

Domain

NLPSemEval

Use Cases

Humor DetectionOffense DetectionContent Moderation

Related Designs

Assessing Humor in Edited News Headlines

Rate the funniness of edited news headlines on a 0-3 scale and judge which of two edited versions is funnier, based on SemEval-2020 Task 7 (Hossain et al.). Headlines are minimally edited by replacing a single word to create humorous effect.

likertradio

Fine-Grained Sentiment Analysis on Financial Microblogs and News

Graded sentiment analysis of financial text with topic classification, rating market sentiment from very bearish to very bullish on a 7-point scale. Based on SemEval-2017 Task 5.