Skip to content
Tutorials7 min read

Pronunciation Assessment Annotation

Build a pronunciation rating task in Potato with audio playback, waveform display, Likert scales for comprehensibility and accentedness, and free-text feedback.

Potato Team

Pronunciation assessment shows up in language learning apps, speech therapy, and accent coaching. This tutorial builds the interfaces for rating pronunciation quality, flagging specific errors, and writing feedback. For the underlying audio features, see the audio annotation documentation.

Playing the audio and showing the target

Content and questions are separate in Potato. instance_display says what the rater sees and hears, and annotation_schemes says what they are asked. A rating task needs an audio field for the recording and a text field for the sentence the learner was asked to say:

yaml
annotation_task_name: "Pronunciation Assessment"
task_dir: "."
 
data_files:
  - data/learner_recordings.json
 
item_properties:
  id_key: id
  text_key: target_text
 
instance_display:
  fields:
    - key: target_text
      type: text
      label: "Target sentence"
 
    - key: audio_path
      type: audio
      label: "Learner recording"
      display_options:
        controls: true
        show_waveform: true
 
annotation_schemes:
  - annotation_type: likert
    name: overall_quality
    description: "Overall pronunciation quality"
    size: 5
    min_label: "Very poor"
    max_label: "Excellent"
    label_requirement:
      required: true
 
output_annotation_dir: annotation_output/
export_annotation_format: jsonl
 
user_config:
  allow_all_users: true

item_properties takes id_key and text_key, and nothing about audio. The recording is named in instance_display instead, by the key that holds its path. Writing audio_path: under item_properties is ignored, which reads as a player that never appears.

A rating is a likert, not an audio_annotation. The audio_annotation type is for marking regions of the waveform, so using it to collect a 1-to-5 score gives the rater a segmenting interface and no scale.

Rating several dimensions

Pronunciation research usually separates comprehensibility from accentedness, because a heavily accented speaker can be perfectly easy to understand and the two move independently. Ask them as separate scales:

yaml
annotation_task_name: "Detailed Pronunciation Assessment"
task_dir: "."
 
data_files:
  - data/recordings.json
 
item_properties:
  id_key: id
  text_key: sentence
 
instance_display:
  fields:
    - key: sentence
      type: text
      label: "Target text"
 
    - key: native_language
      type: text
      label: "First language"
 
    - key: learner_audio
      type: audio
      label: "Learner recording"
      display_options:
        controls: true
        show_waveform: true
 
    - key: native_audio
      type: audio
      label: "Native reference"
      display_options:
        controls: true
 
annotation_schemes:
  - annotation_type: likert
    name: overall
    description: "Overall pronunciation quality"
    size: 5
    min_label: "Very poor"
    max_label: "Excellent"
    label_requirement:
      required: true
 
  - annotation_type: likert
    name: intelligibility
    description: "How understandable is the speaker?"
    size: 5
    min_label: "Unintelligible"
    max_label: "Perfectly clear"
    label_requirement:
      required: true
 
  - annotation_type: likert
    name: accent_strength
    description: "How strong is the foreign accent?"
    size: 5
    min_label: "No accent"
    max_label: "Very strong"
 
  - annotation_type: likert
    name: intonation
    description: "Intonation and rhythm"
    size: 5
    min_label: "Unnatural"
    max_label: "Native-like"
 
  - annotation_type: likert
    name: fluency
    description: "Fluency, meaning smoothness and pausing"
    size: 5
    min_label: "Very disfluent"
    max_label: "Very fluent"
 
  - annotation_type: multiselect
    name: error_types
    description: "What errors are present? Select all that apply."
    labels:
      - Vowel errors
      - Consonant errors
      - Word stress errors
      - Sentence intonation errors
      - Rhythm or timing errors
      - Added sounds
      - Deleted sounds
      - Substituted sounds
      - No significant errors
 
  - annotation_type: text
    name: problem_words
    description: "List any mispronounced words"
    placeholder: "e.g. 'thought' pronounced as 'taught'"
    label_requirement:
      required: false
 
  - annotation_type: radio
    name: native_comparison
    description: "Compared to the native reference"
    labels:
      - Very different
      - Somewhat different
      - Fairly similar
      - Very similar
      - Indistinguishable
 
  - annotation_type: likert
    name: confidence
    description: "Your confidence in this assessment"
    size: 5
    min_label: "Low"
    max_label: "High"
 
output_annotation_dir: annotation_output/
export_annotation_format: jsonl

Any field in the data can be shown by adding it to instance_display, which is how the learner's first language reaches the rater here. There is no separate metadata block.

Rating each word

multirate puts one row per item against a shared scale, which suits word-by-word judgments. It requires both the rows (options) and the columns (labels):

yaml
annotation_schemes:
  - annotation_type: multirate
    name: word_ratings
    description: "Rate the pronunciation of each word"
    options:
      - "the"
      - "weather"
      - "is"
      - "really"
      - "nice"
      - "today"
    labels:
      - Poor
      - Fair
      - Good
    arrangement: vertical

The rows are fixed in the config, so a per-item word list means either one config per sentence or options_from_data, which reads the rows from the item instead. Fixed rows suit a study where every learner reads the same prompts.

Phoneme-level transcription

speech_transcript shows the expected segments and lets the rater correct them, which is what phonetic error coding needs:

yaml
data_files:
  - data/phoneme_data.json
 
item_properties:
  id_key: id
  text_key: expected_phonemes
 
annotation_schemes:
  - annotation_type: speech_transcript
    name: phonemes
    description: "Mark substitutions, deletions, and insertions against the expected phonemes"
    segments_key: expected_phonemes
    error_types:
      - substitution
      - deletion
      - insertion
    allow_correction: true

Complete configuration

This runs an L2 English assessment on the 9-point scales used in the second-language pronunciation literature, with a training phase in front of it:

yaml
annotation_task_name: "L2 English Pronunciation Assessment"
task_dir: "."
 
data_files:
  - data/esl_recordings.json
 
item_properties:
  id_key: id
  text_key: target_sentence
 
instance_display:
  fields:
    - key: target_sentence
      type: text
      label: "The learner was asked to say"
 
    - key: native_language
      type: text
      label: "First language"
 
    - key: proficiency_level
      type: text
      label: "Level"
 
    - key: audio_url
      type: audio
      label: "Recording"
      display_options:
        controls: true
        show_waveform: true
 
annotation_instructions: |
  ## Pronunciation Assessment Guidelines
 
  ### Rating scales
 
  **Overall score (1-9)**
  - 1-3: significant difficulty, hard to understand
  - 4-6: noticeable accent, still understandable
  - 7-9: minor issues through to native-like
 
  **Comprehensibility** is how much effort understanding takes. Ignore the
  accent where the message is clear.
 
  **Accentedness** is the degree of foreign accent, judged independently of
  whether you understood it.
 
  ### What to listen for
  Individual sounds, word stress, sentence intonation, and speaking rate.
 
  ### Tips
  Listen at least twice. Use slow playback for the details. Be consistent
  across recordings rather than across raters.
 
annotation_schemes:
  - annotation_type: likert
    name: overall_score
    description: "Overall pronunciation score"
    size: 9
    min_label: "Very poor"
    max_label: "Native-like"
    label_requirement:
      required: true
 
  - annotation_type: likert
    name: comprehensibility
    description: "How easy is it to understand?"
    size: 9
    min_label: "Very hard"
    max_label: "Very easy"
    label_requirement:
      required: true
 
  - annotation_type: likert
    name: accentedness
    description: "How strong is the foreign accent?"
    size: 9
    min_label: "No accent"
    max_label: "Very strong"
 
  - annotation_type: likert
    name: vowels
    description: "Vowel pronunciation accuracy"
    size: 5
    min_label: "Poor"
    max_label: "Excellent"
 
  - annotation_type: likert
    name: consonants
    description: "Consonant pronunciation accuracy"
    size: 5
    min_label: "Poor"
    max_label: "Excellent"
 
  - annotation_type: likert
    name: word_stress
    description: "Word stress patterns"
    size: 5
    min_label: "Incorrect"
    max_label: "Correct"
 
  - annotation_type: likert
    name: sentence_intonation
    description: "Sentence intonation"
    size: 5
    min_label: "Unnatural"
    max_label: "Natural"
 
  - annotation_type: likert
    name: fluency
    description: "Speaking fluency"
    size: 5
    min_label: "Disfluent"
    max_label: "Fluent"
 
  - annotation_type: multiselect
    name: common_errors
    description: "Check all that apply"
    labels:
      - Th-sounds (think, this)
      - R-sounds
      - L-sounds
      - Vowel length
      - Final consonants dropped
      - Consonant clusters simplified
      - Word stress on wrong syllable
      - Flat intonation
      - No errors detected
    label_requirement:
      required: false
 
  - annotation_type: text
    name: feedback
    description: "Specific words or sounds that need work"
    rows: 4
    placeholder: "e.g. 'world' - /r/ needs work; 'through' - th sound"
    label_requirement:
      required: false
 
  - annotation_type: radio
    name: confidence
    description: "How confident are you in your ratings?"
    labels:
      - Not confident
      - Somewhat confident
      - Very confident
 
training:
  enabled: true
  data_file: data/calibration_recordings.json
  annotation_schemes: ["overall_score", "comprehensibility"]
  passing_criteria:
    min_correct: 4
  allow_retry: true
 
phases:
  order: [training, annotation]
  training:
    type: training
    file: data/calibration_recordings.json
 
output_annotation_dir: annotation_output/
export_annotation_format: jsonl
 
user_config:
  allow_all_users: true

The training phase is configured by the top-level training block, and phases only places it in the order. Calibration matters more here than on most tasks, because accentedness ratings drift between raters who never hear the same recording.

What the output contains

Each scheme stores its value under its own name:

json
{
  "overall_score": "6",
  "comprehensibility": "7",
  "accentedness": "5",
  "vowels": "4",
  "consonants": "3",
  "word_stress": "4",
  "sentence_intonation": "3",
  "fluency": "4",
  "common_errors": ["Th-sounds (think, this)", "R-sounds"],
  "feedback": "'weather' - th sound; 'really' - r sound",
  "confidence": "Very confident"
}

Likert values come back as strings. Convert them before averaging, since "9" sorts below "10" as text and any scale past nine points will quietly rank wrong.

Tips for pronunciation assessment

Rate a few recordings together as a group before starting, so everyone calibrates to the same bar. Keep reference recordings on hand to anchor the scale. Listen at least twice, once for the overall impression and again for detail. Use headphones, because phone speakers hide exactly the consonant detail being rated. Raters with phonetics training pick this up faster, though calibration matters more than background.

Next steps

Full audio documentation at Audio Annotation.