iSarcasmEval: Intended Sarcasm Detection

# iSarcasmEval: Intended Sarcasm Detection
# Based on Abu Farha, Oprea, Wilson, and Magdy, SemEval-2022 Task 6
# Paper: https://aclanthology.org/2022.semeval-1.111/
# Dataset: https://github.com/iabufarha/iSarcasmEval
#
# iSarcasmEval is the first shared task to focus on INTENDED sarcasm: each
# text is labeled by its own author, not by third-party annotators. The
# shared task defines three subtasks:
#   - SubTask A: is the text sarcastic? (binary)
#   - SubTask B: which category of ironic speech does it express?
#   - SubTask C: given a sarcastic text and its non-sarcastic rephrase,
#                tell which one is the sarcastic one (pairwise)
# This showcase models SubTask A (binary) and SubTask B (category), and
# collects an optional non-sarcastic rephrase in the spirit of SubTask C.
#
# SubTask B ironic-speech categories (Leggitt & Gibbs):
# - Sarcasm: contradicts the state of affairs and is critical toward someone
# - Irony: contradicts the state of affairs but is not necessarily critical
# - Satire: ridicules a subject through humor, irony, or exaggeration
# - Understatement: undermines the magnitude of a state of affairs
# - Overstatement: exaggerates the magnitude of a state of affairs
# - Rhetorical Question: a question whose answer is not expected

annotation_task_name: "iSarcasmEval - Intended Sarcasm Detection"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "text"

output_annotation_dir: "annotation_output/"
export_annotation_format: "jsonl"

port: 8000

annotation_schemes:
  - annotation_type: radio
    name: is_sarcastic
    description: "Is this text sarcastic (as intended by its author)?"
    labels:
      - name: "Sarcastic"
        tooltip: "The author intended the text to be sarcastic or ironic"
        key_value: "1"
      - name: "Not Sarcastic"
        tooltip: "The text is meant literally"
        key_value: "2"

  - annotation_type: multiselect
    name: sarcasm_category
    description: "If sarcastic, which category of ironic speech applies? (Select all that apply)"
    labels:
      - name: "Sarcasm"
        tooltip: "Contradicts the state of affairs and is critical toward someone/something"
      - name: "Irony"
        tooltip: "Contradicts the state of affairs but is not necessarily critical"
      - name: "Satire"
        tooltip: "Ridicules a subject through humor, irony, or exaggeration"
      - name: "Understatement"
        tooltip: "Undermines the magnitude of a state of affairs"
      - name: "Overstatement"
        tooltip: "Exaggerates the magnitude of a state of affairs"
      - name: "Rhetorical Question"
        tooltip: "A question posed without expecting an answer"

  - annotation_type: text
    name: rephrase
    description: "If sarcastic, provide a non-sarcastic rephrase that conveys the intended meaning (optional)"
    multiline: true
    required: false
    placeholder: "Non-sarcastic rephrase..."

annotation_instructions: |
  You will see a short social-media text. First decide whether it is
  sarcastic as intended by its author. If it is, select the category or
  categories of ironic speech it expresses, and optionally write a
  non-sarcastic rephrase that conveys the same intended meaning.

user_config:
  allow_all_users: true
max_annotations_per_user: 50
num_annotators_per_item: 3