Speech Emotion Annotation with Categories and Dimensions

Speech emotion annotation asks a listener to label the emotion heard in a recording, either as a category such as anger or sadness or as ratings of valence, arousal, and dominance. The major corpora collected two to ten judgments per clip and reported chance-corrected agreement between 0.27 and 0.52 for categories. Plan for at least five raters per clip, and keep every rater's label.

Speech emotion recognition models learn from speech emotion labels. A categorical label assigns one emotion from a fixed set, as in the discrete models of emotion classification. A dimensional rating places the utterance on continuous scales, usually valence (negative to positive), arousal or activation (calm to excited), and dominance (weak to strong). The three scales are the axes of the PAD model. Dimensional ratings are often collected with the Self-Assessment Manikin. The manikin is a row of cartoon figures for each scale.

Perceived emotion is what a listener hears, and expressed or intended emotion is what the speaker or actor meant to convey. Agreement between raters is reported as inter-rater reliability. The most common coefficients are Fleiss' kappa or Krippendorff's alpha.

This guide covers the design decisions for a speech emotion study and uses Potato as the worked example. Audio Annotation maps every audio task to a Potato scheme, and the tutorial Classifying Emotions in Speech builds several emotion interfaces step by step. Audio Annotation Tools Compared compares the tools, and Rating Scales and Likert Design covers scale design for any task.

Labels and raters in four speech emotion corpora

Four widely used English corpora show the range of label sets, rater counts, and consensus rules. Each row comes from the corpus paper listed under References:

IEMOCAP's raters reached a majority on 74.6% of turns, and MSP-Podcast reports a "no agreement" class for the turns where no plurality emerged. CREMA-D sets clips without a binomial majority aside as ambiguous. A study that needs one label per clip should plan for a share of clips that end without a majority label.

Published agreement for speech emotion labels

The four corpora report chance-corrected agreement on emotion categories between 0.27 and 0.52. IEMOCAP reports a Fleiss' kappa of 0.27 over all turns. The kappa rose to 0.35 after the authors merged happiness with excited and folded fear, disgust, and surprise into other. MSP-Podcast reports a kappa of 0.411 for primary emotions, and CREMA-D reports a Krippendorff's alpha of 0.42 for emotion labels. RAVDESS reports a Fleiss' kappa for audio-only speech of 0.41 at normal intensity and 0.52 at strong intensity.

Valence ratings agree best in both corpora that report all three dimensions. MSP-Podcast reports Krippendorff's alpha of 0.508 for valence, 0.441 for arousal, and 0.386 for dominance. IEMOCAP reports Cronbach's alpha between its two raters of 0.809 for valence, 0.607 for activation, and 0.608 for dominance. Because Cronbach's alpha and Krippendorff's alpha measure different things, compare the two corpora by the order of the dimensions rather than by the values. A pilot with a categorical kappa near 0.4 is in line with published corpora, and a pilot kappa well below IEMOCAP's 0.27 is a reason to revise the guidelines or check the raters before scaling up.

Perceived and expressed emotion

Actors and listeners often disagree about which emotion an utterance expresses. In IEMOCAP, six actors labeled the improvised turns each actor had performed, and the actors' labels matched the listeners' majority label 60% of the time, against 79% for the other listeners. In CREMA-D, individual raters matched the actor's intended emotion on 40.9% of audio-only clips, 58.2% of visual-only clips, and 63.6% of audio-visual clips. RAVDESS raters identified the intended emotion in 58% of audio-only speech at normal intensity and 67% at strong intensity.

Decide which construct the labels should measure before writing the question. For a perceived-emotion study, ask "Which emotion do you hear?" and treat the raters' distribution as the answer. For an expressed-emotion study, keep the speaker's intended label or self-report, and use listener ratings to check that the recording conveys the emotion. Acted corpora such as CREMA-D and RAVDESS record both the intended and the perceived emotion, and naturalistic corpora such as MSP-Podcast record only the perceived emotion.

Whole dialogue or isolated utterance

The IEMOCAP authors had raters judge turns in context because the authors considered the context of an utterance important for recognizing emotion. IEMOCAP raters watched each session and assessed the turns in order. The earlier turns were available to the rater for every judgment. MSP-Podcast cuts podcasts into single-speaker turns and has each turn rated as a separate item. MSP-Podcast sets a lower bound of 2.75 seconds to give raters enough speech to judge. The MSP-Podcast team split turns longer than 11 seconds because emotion can change within a turn.

In Potato, a per-clip label is a radio or likert scheme beside an audio display, and earlier turns can sit in a text field above the clip. For long recordings where emotion shifts, the audio_annotation scheme lets raters mark segments on the waveform and label each one. In questions mode, the same scheme asks a set of questions about every segment the rater marks.

Listening setup and rater screening

RAVDESS, CREMA-D, and MSP-Podcast each document controls on what raters heard or who rated. RAVDESS raters sat in sound-attenuated booths with Sennheiser headphones at a fixed volume, and a rating screen appeared only after the clip finished. CREMA-D raters first had to recognize three words played at low, medium, and high volume, with up to three tries, before the task began. MSP-Podcast removed 430 crowd workers and the workers' 44,968 annotations after agreement checks, then moved to trained student raters who passed a screening test and received weekly agreement reports.

Potato covers the online versions of those controls. A headphone_check phase screens out participants listening on loudspeakers. A training phase gives practice clips with feedback, and gold items and attention checks score accuracy during the real task. With min_response_time set, Potato fails any attention-check answer given faster than the clip could have been heard. Raters can come from Prolific or a university SONA pool, as Running a Crowdsourced Study on Prolific and SONA describes. Potato is the software only, and the Potato team does not provide raters.

Majority vote or label distributions

A majority vote turns several judgments into one training label and discards the disagreement. IEMOCAP and MSP-Podcast both use a vote for categories, and MSP-Podcast averages the dimensional ratings. MSP-Podcast also distributes the individual annotations for each turn, and the authors list training with soft labels among the uses. A soft label is the share of raters who chose each emotion. A clip heard as angry by three of five raters and as frustrated by two becomes 0.6 anger and 0.4 frustration.

Potato stores each rater's answers separately, and a vote, a mean, or a distribution can be computed after collection. The soft_label scheme goes further and asks each rater to split 100 points across the emotions. Aggregating Crowd Labels compares vote-based and model-based aggregation in more detail.

The agreement report at /admin/iaa picks measures by scheme type:

• radio: percent agreement, Cohen's kappa, Fleiss' kappa, and nominal Krippendorff's alpha.
• likert: linear and quadratic weighted kappa, Spearman's rho, and ordinal Krippendorff's alpha.
• slider: Pearson's r, mean absolute error, root mean square error, interval Krippendorff's alpha, and ICC(2,k).
• soft_label: Krippendorff's alpha pooled over all emotions and for each emotion separately.

Raters per clip in published corpora

The published corpora used between two and ten raters per clip. IEMOCAP's three categorical raters and two dimensional raters are the low end, and CREMA-D and RAVDESS collected 10 judgments for each clip. MSP-Podcast's minimum of five raters is enough for a plurality vote and gives a five-vote distribution for soft labels. Pay for more raters on the ambiguous clips rather than on every clip, and see How Many Annotators Do You Need? for the power calculation.

A speech emotion task in Potato

The IEMOCAP showcase entry summarizes the corpus. The config below adapts IEMOCAP's label set to audio-only clips, rates the three dimensions on IEMOCAP's five-point scale, and shows earlier turns as text. The context field copies IEMOCAP's in-context design. Each clip gets five raters to match the MSP-Podcast minimum, and the headphone check and practice clips run before the first real item:

annotation_task_name: "Speech emotion, one utterance at a time"
task_dir: "."
data_files:
  - "data/utterances.jsonl"
item_properties:
  id_key: id
  text_key: audio_url
output_annotation_dir: "annotation_output/"
 
instance_display:
  fields:
    - key: context
      type: text
      label: "Earlier in the conversation"
    - key: audio_url
      type: audio
      label: "Utterance to rate"
      display_options:
        show_waveform: true
 
annotation_schemes:
  - annotation_type: radio
    name: emotion
    description: "Which emotion do you hear in this utterance?"
    labels: [neutral, happiness, excitement, sadness, anger, frustration, fear, surprise, disgust, other]
    label_requirement: {required: true}
  - annotation_type: likert
    name: valence
    description: "How negative or positive does the speaker sound?"
    size: 5
    min_label: "Very negative"
    max_label: "Very positive"
  - annotation_type: likert
    name: arousal
    description: "How calm or excited does the speaker sound?"
    size: 5
    min_label: "Very calm"
    max_label: "Very excited"
  - annotation_type: likert
    name: dominance
    description: "How weak or strong does the speaker sound?"
    size: 5
    min_label: "Very weak"
    max_label: "Very strong"
 
num_annotators_per_item: 5
 
training:
  enabled: true
  data_file: "data/training.json"
  allow_retry: true
 
phases:
  order: [headphones, training, annotation]
  headphones:
    type: headphone_check
    test: huggins
  training: {type: training, title: "Practice clips"}
  annotation: {type: annotation}

Each line of data/utterances.jsonl contains an id, an audio_url, and a context string with the preceding turns. Remove the context field from instance_display to rate each turn as a separate item, as MSP-Podcast does. data/training.json lists the practice clips with a correct_answers entry and an explanation for each clip. Start the server with potato start config.yaml and check /admin/iaa after a pilot batch to see whether agreement is near the published range.

Frequently asked questions

What is speech emotion annotation?

Speech emotion annotation is the labeling of recorded speech with the emotion a listener hears, as a category, as ratings of valence, arousal, and dominance, or as both. IEMOCAP and MSP-Podcast collect both kinds of label. The labels are used to train and evaluate speech emotion recognition models. Published corpora collect two to ten judgments per clip.

What emotion labels are used in audio and speech annotation?

IEMOCAP, MSP-Podcast, CREMA-D, and RAVDESS each use a short list of basic emotions plus a neutral or no-emotion option. IEMOCAP adds frustration and excited to anger, sadness, happiness, disgust, fear, and surprise, and MSP-Podcast adds contempt to the primary list and offers 16 secondary emotions. Dimensional labels for valence, arousal, and dominance usually sit beside the categories. Raters score each dimension on a five-point or seven-point manikin scale.

How do you set up a task where raters listen to voice clips and label emotions?

Show each clip in an audio player next to one categorical question and an optional rating scale for each dimension, then assign at least five raters per clip. Screen raters with a headphone check and practice clips before the real task. Keep every rater's answer, and compute a majority label, a label distribution, or both after collection.

Further reading

• Audio Annotation, for every audio task set up in Potato
• Classifying Emotions in Speech, a step-by-step tutorial
• Potato for audio and speech annotation
• Rating Scales and Likert Design
• Inter-Annotator Agreement Explained
• Aggregating Crowd Labels
• Audio Annotation Tools Compared

References

Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., and Narayanan, S. S. (2008). IEMOCAP: Interactive Emotional Dyadic Motion Capture Database. Language Resources and Evaluation, 42(4), 335-359. https://doi.org/10.1007/s10579-008-9076-6

Busso, C., Lotfian, R., Sridhar, K., Salman, A. N., Lin, W.-C., Goncalves, L., Parthasarathy, S., Naini, A. R., Leem, S.-G., Martinez-Lucas, L., Chou, H.-C., and Mote, P. (2025). The MSP-Podcast Corpus. arXiv 2509.09791. https://doi.org/10.48550/arXiv.2509.09791

Cao, H., Cooper, D. G., Keutmann, M. K., Gur, R. C., Nenkova, A., and Verma, R. (2014). CREMA-D: Crowd-Sourced Emotional Multimodal Actors Dataset. IEEE Transactions on Affective Computing, 5(4), 377-390. https://doi.org/10.1109/TAFFC.2014.2336244

Livingstone, S. R., and Russo, F. A. (2018). The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A Dynamic, Multimodal Set of Facial and Vocal Expressions in North American English. PLOS ONE, 13(5), e0196391. https://doi.org/10.1371/journal.pone.0196391