Skip to content
beginnerevaluation

#HashtagWars - Learning a Sense of Humor

Humor ranking of tweets submitted to Comedy Central's @midnight #HashtagWars, classifying comedic quality. Based on SemEval-2017 Task 6.

About this dataset

SemEval-2017 Task 6, #HashtagWars: Learning a Sense of Humor, was organized by Peter Potash, Alexey Romanov, and Anna Rumshisky of the University of Massachusetts Lowell. The task reframes humor detection as comparative ranking instead of the usual binary humor/not-humor classification. Because every tweet in a group is written for the same hashtag prompt, systems are asked to judge which tweets the show found funnier rather than whether a tweet is humorous at all.

The data comes from the Hashtag Wars segment of Comedy Central's late-night show @midnight, where the host proposes a topic as a hashtag and viewers reply with tweets. The organizers pulled tweets through the public Twitter search API and used the show's official Tumblr top-10 posts and website to label each tweet on a three-point scale: 2 for the episode's single winning tweet, 1 for a tweet that made the top ten, and 0 for all others.

Collection ran for roughly eight months and produced 12,734 tweets across 112 hashtags. The organizers released 101 files (11,325 tweets) as training data, 5 files (660 tweets) as trial data, and 6 files (749 tweets) for evaluation. Two subtasks were scored: pairwise comparison (subtask A, 7 teams, best accuracy 0.675) and full ranking (subtask B, 5 teams, best rank edit distance 0.872).

The Potato config below reproduces this task with a single radio scheme that rates each tweet against its hashtag prompt as Funny (Winner), Somewhat Funny, or Not Funny. It works for collecting quick per-tweet humor judgments. The original dataset uses ordinal 0/1/2 labels derived from the show's top-10 and winner; the config adapts these into a three-way funniness rating.

Total tweets
12,734
Hashtags
112
Labels
0, 1, 2 (2 = winner, 1 = top-10, 0 = other)
Training set
101 files, 11,325 tweets
Trial set
5 files, 660 tweets
Evaluation set
6 files, 749 tweets
Submit

Configuration Fileconfig.yaml

This Potato config reproduces the annotation task. Save it as config.yaml and run potato start config.yaml to try it.

yaml
# #HashtagWars - Learning a Sense of Humor
# Based on Potash et al., SemEval 2017
# Paper: https://aclanthology.org/S17-2004/
# Dataset: https://github.com/pppotash/SemEval-2017-Task-6
#
# This task asks annotators to rate the funniness of tweets submitted
# to Comedy Central's @midnight #HashtagWars segments.
#
# Humor Labels:
# - Funny (Winner): The tweet is genuinely funny and could win the competition
# - Somewhat Funny: The tweet has some humor but is not top-tier
# - Not Funny: The tweet is not funny or misses the mark

annotation_task_name: "#HashtagWars - Humor Rating"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "text"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

port: 8000
server_name: localhost

annotation_schemes:
  - annotation_type: radio
    name: humor_rating
    description: "How funny is this tweet in response to the hashtag prompt?"
    labels:
      - "Funny (Winner)"
      - "Somewhat Funny"
      - "Not Funny"
    keyboard_shortcuts:
      "Funny (Winner)": "1"
      "Somewhat Funny": "2"
      "Not Funny": "3"
    tooltips:
      "Funny (Winner)": "The tweet is genuinely funny and stands out as a potential winner"
      "Somewhat Funny": "The tweet has some humor but is not exceptional"
      "Not Funny": "The tweet is not funny, too obvious, or misses the mark"

annotation_instructions: |
  You will be shown a tweet that was submitted in response to a #HashtagWars prompt
  from Comedy Central's @midnight show. Your task is to rate how funny the tweet is.
  Consider creativity, wit, and comedic timing in your judgment.

html_layout: |
  <div style="padding: 15px; max-width: 800px; margin: auto;">
    <div style="background: #fefce8; border: 1px solid #fde68a; border-radius: 8px; padding: 12px; margin-bottom: 12px;">
      <strong style="color: #a16207;">Hashtag Prompt:</strong>
      <span style="font-size: 15px; font-weight: bold;">{{hashtag_prompt}}</span>
    </div>
    <div style="background: #f0f9ff; border: 1px solid #bae6fd; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
      <strong style="color: #0369a1;">Tweet:</strong>
      <p style="font-size: 16px; line-height: 1.7; margin: 8px 0 0 0;">{{text}}</p>
    </div>
  </div>

allow_all_users: true
instances_per_annotator: 50
annotation_per_instance: 2
allow_skip: true
skip_reason_required: false

Sample Datasample-data.json

json
[
  {
    "id": "humor_001",
    "text": "My diet starts Monday. Just like it has every Monday since 2003.",
    "hashtag_prompt": "#MyDietIn5Words"
  },
  {
    "id": "humor_002",
    "text": "Please stop talking about crossfit.",
    "hashtag_prompt": "#ThingsYouSayAtThanksgiving"
  }
]

// ... and 8 more items

Get This Design

View on GitHub

Clone or download from the repository

Quick start:

git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/semeval/2017/task06-hashtagwars-humor
potato start config.yaml

Dataset & paper

Potash et al., SemEval 2017

Official dataset ↗Read the paper ↗

Citation (BibTeX)

bibtex
@inproceedings{potash-etal-2017-semeval,
    title = "{S}em{E}val-2017 Task 6: {\#}{H}ashtag{W}ars: Learning a Sense of Humor",
    author = "Potash, Peter and Romanov, Alexey and Rumshisky, Anna",
    booktitle = "Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)",
    month = aug,
    year = "2017",
    address = "Vancouver, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/S17-2004/",
    doi = "10.18653/v1/S17-2004",
    pages = "49--57"
}

Details

Annotation Types

radio

Domain

SemEvalNLPHumor DetectionSocial Media

Use Cases

Humor DetectionComedy EvaluationSocial Media Analysis

Tags

semevalsemeval-2017shared-taskhumorhashtagwarscomedytwitter

Found an issue or want to improve this design?

Open an Issue

Related Designs

AfriSenti - African Language Sentiment

Sentiment analysis for tweets in African languages, classifying text as positive, negative, or neutral. Covers 14 African languages including Amharic, Hausa, Igbo, Yoruba, and Swahili. Based on SemEval-2023 Task 12 (Muhammad et al.).

radio

Detecting Stance in Tweets

Classification of stance expressed in tweets toward specific targets as favor, against, or neither. Based on SemEval-2016 Task 6 (Stance Detection).

radio

Detection and Interpretation of English Puns

Detection of puns in text with identification of the specific pun word, combining binary classification with span annotation. Based on SemEval-2017 Task 7.

radiospan