CREMA-D Crowd Ratings of Emotion in Acted Speech

# CREMA-D: Crowd-Sourced Perceptual Ratings of Acted Speech Emotion
#
# Dataset: CREMA-D (Crowd-sourced Emotional Multimodal Actors Dataset)
# Paper: Cao, H., Cooper, D. G., Keutmann, M. K., Gur, R. C., Nenkova, A., and
#   Verma, R. (2014). CREMA-D: Crowd-Sourced Emotional Multimodal Actors Dataset.
#   IEEE Transactions on Affective Computing 5(4):377-390.
#   Open copy: https://pmc.ncbi.nlm.nih.gov/articles/PMC4313618/
# BibTeX (Crossref):
#   @article{Cao_2014, title={CREMA-D: Crowd-Sourced Emotional Multimodal Actors Dataset},
#     volume={5}, DOI={10.1109/taffc.2014.2336244}, number={4},
#     journal={IEEE Transactions on Affective Computing},
#     author={Cao, Houwei and Cooper, David G. and Keutmann, Michael K. and Gur, Ruben C. and Nenkova, Ani and Verma, Ragini},
#     year={2014}, month=Oct, pages={377--390}}
# DOI: 10.1109/TAFFC.2014.2336244
# Dataset: https://github.com/CheyneyComputerScience/CREMA-D
# Licence: the database is made available under the Open Database License
#   (ODbL 1.0), and any rights in the individual contents of the database
#   (the clips) are licensed under the Database Contents License (DbCL 1.0),
#   per the repository README. This design ships no audio. Its ten sample items
#   hotlink the MP3 files of the official repository through
#   media.githubusercontent.com (raw.githubusercontent.com serves only Git LFS
#   pointer files for them), so the clips stay on the authors' host and are not
#   part of the download. Attribution: CREMA-D, Cao et al. 2014. The ODbL has
#   its own conditions for a derived database that is publicly used, which this
#   design does not create; read the licence before building one.
# Access: the README asks anyone who accesses the GitHub repository to fill out
#   the form it links, "so we can keep a record of the community of CREMA-D
#   users." Please do so before using the clips.
#
# Protocol (Cao et al. 2014, Section 2.2, "Perceptual Ratings From Crowd Sourcing"):
#   "Upon completion of the sound test, raters were given a description of the
#   rating task, which is to specify two ratings of emotion expressed in each
#   clip presented to them. The first rating is the emotional category. Raters
#   can select an emotion from the target emotions (anger, disgust, fear,
#   happy, sad) or select 'No Emotion' if they do not perceive any specific
#   emotions. The second rating is a continuous value, which corresponds to
#   the intensity of the emotional state if an emotion is recognized or the
#   confidence level that there is no perceivable emotion if 'No Emotion' is
#   selected."
#   "The intensity rating is provided on a continuous sliding scale that is
#   scored from 0 to 100."
#   Raters saw three sections in a fixed order (audio-only, visual-only,
#   audio-visual) of 35 items each (2 practice, 30 dataset items, and 3 repeats
#   to test consistency), 105 items per list.
#   The 12 acted sentences were rated as emotionally neutral in a prior study
#   and were spoken in six intended emotions (the dataset README codes them
#   ANG, DIS, FEA, HAP, NEU, SAD).
#
# Adaptations:
#   - Only the audio-only section is reproduced. The visual-only and
#     audio-visual sections need video, which this design does not ship.
#   - The paper's "No Emotion" choice is kept as "No Emotion" (the released
#     tables code it as Neutral, N).
#   - The paper shows the intensity control only after an emotion is chosen.
#     Here the slider is always visible, and it starts at 50, so an untouched
#     slider and a rating of 50 look the same in the output.
#   - The paper's sound calibration, sound test, practice items, and repeated
#     consistency items are not reproduced. The headphone check would be a
#     `phases` page and the repeats would be `attention_checks`.
#   - Ten sample clips only. The clip name, actor, intended emotion, and
#     intended level are stored in the data file for reference and are not
#     displayed to the rater.
#   - No model_training block. Potato's audio trainers consume temporal
#     segments (the hf-audio-tagger trainer is registered for kinds
#     ('temporal',)), and no registered trainer consumes a per-clip radio or
#     slider label on audio. The labels here are exported for analysis only.
#
# Agreement across crowd raters can be read from /admin/iaa once several
# annotators label the same clips (num_annotators_per_item below).

annotation_task_name: "CREMA-D: Rate the Emotion in Acted Speech"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "audio_url"
  # The intended emotion and level are the dataset's reference labels. The clip
  # file names in audio_url also carry them, so a real study should serve the
  # clips under neutral names.
  private_keys: [clip_name, intended_emotion, intended_level]

output_annotation_dir: "annotation_output/"
export_annotation_format: "jsonl"

instance_display:
  fields:
    - key: audio_url
      type: audio
      label: "Audio clip (audio-only presentation)"
      display_options:
        controls: true
        preload: "auto"
  layout:
    direction: vertical
    gap: 16px

annotation_schemes:
  - annotation_type: radio
    name: perceived_emotion
    description: "Which emotion does the speaker express in this clip? Choose 'No Emotion' if you do not perceive any specific emotion."
    labels:
      - name: "Anger"
        key_value: "a"
      - name: "Disgust"
        key_value: "d"
      - name: "Fear"
        key_value: "f"
      - name: "Happy"
        key_value: "h"
      - name: "Sad"
        key_value: "s"
      - name: "No Emotion"
        key_value: "n"

  - annotation_type: slider
    name: emotion_intensity
    description: "How intense is the emotion (0 = none, 100 = extreme)? If you chose 'No Emotion', rate how confident you are that no emotion is present."
    min_value: 0
    max_value: 100
    starting_value: 50
    step: 1

user_config:
  allow_all_users: true

max_annotations_per_user: 10
num_annotators_per_item: 3

annotation_instructions: |
  You will hear short recordings of actors reading everyday sentences.
  Listen to the whole clip, choose the emotion you perceive in the speaker's
  voice, then use the slider to rate its intensity from 0 to 100. Rate the
  voice, not the words, because every sentence was written to be emotionally
  neutral. Use headphones in a quiet place.