Audio Annotation

Configure waveform-based audio annotation in Potato. Display interactive playback controls, set time markers, and combine with radio, likert, or text schemas.

Potato 2.0 supports audio annotation with waveform visualization powered by Peaks.js, segment labeling, and keyboard shortcuts.

Use Cases

Speech transcription and review
Speaker diarization
Music analysis
Audio event detection
Emotion recognition in speech
Call center quality assurance

Enabling Audio Support

An audio task is an audio_annotation scheme in annotation_schemes. The scheme reads the audio URL from the field named by item_properties.text_key, so no separate media key is needed:

yaml

annotation_schemes:
  - annotation_type: audio_annotation
    name: audio_segments
    description: "Segment and label the audio"
    labels:
      - Speech
      - Music
      - Silence
      - Noise

Operational Modes

Potato supports three audio annotation modes:

Label Mode

Segment audio and assign category labels to each segment:

yaml

annotation_schemes:
  - annotation_type: audio_annotation
    name: speaker_diarization
    mode: label
    description: "Identify speakers in the audio"
    labels:
      - Speaker A
      - Speaker B
      - Overlap

Questions Mode

Each segment gets its own question set instead of a single label. The annotator marks a region, selects it, and answers the questions in the Segment Details panel. segment_schemes is required in this mode, and each entry is an ordinary annotation scheme rendered per segment:

yaml

annotation_schemes:
  - annotation_type: audio_annotation
    name: speech_quality
    mode: questions
    description: "Evaluate speech segments"
    segment_schemes:
      - annotation_type: radio
        name: speaker_type
        description: "Who is speaking?"
        labels: ["host", "guest", "unknown"]
      - annotation_type: likert
        name: clarity
        description: "How clear is this segment?"
        size: 5
        min_label: "Unintelligible"
        max_label: "Perfectly clear"

Both Mode

Combine labeling with per-segment questions. This mode requires labels and segment_schemes together:

yaml

annotation_schemes:
  - annotation_type: audio_annotation
    name: full_analysis
    mode: both
    description: "Label and analyze audio segments"
    labels:
      - Speech
      - Music
      - Noise
    segment_schemes:
      - annotation_type: radio
        name: foreground
        description: "Is this the foreground audio?"
        labels: ["Yes", "No"]

Answers land on the segment rather than the item, keyed by the sub-scheme's name. Two limits are worth knowing before designing around segment questions: keybindings are not bound inside a segment, because a segment's fields only exist while it is selected, and label_requirement is not enforced on a sub-scheme. Use min_segments to require segments, and check completeness during analysis.

Configuration Options

Basic Setup

yaml

annotation_schemes:
  - annotation_type: audio_annotation
    name: segments
    description: "Create audio segments"
    labels:
      - Label A
      - Label B
 
    # Optional constraints
    min_segments: 1
    max_segments: 50

Keyboard Shortcuts

Labels can be assigned using number keys 1-9:

yaml

annotation_schemes:
  - annotation_type: audio_annotation
    name: speakers
    description: "Mark each speaker turn"
    labels:
      - Speaker A  # Press 1
      - Speaker B  # Press 2
      - Overlap    # Press 3

Label Colors

Customize segment colors:

yaml

annotation_schemes:
  - annotation_type: audio_annotation
    name: segments
    description: "Segment the recording and label each region"
    labels:
      - name: Speech
        color: "#22C55E"
      - name: Music
        color: "#3B82F6"
      - name: Silence
        color: "#9CA3AF"

Waveform Performance

For optimal performance with long audio files, install the BBC audiowaveform tool:

bash

# macOS
brew install audiowaveform
 
# Ubuntu/Debian
sudo apt-get install audiowaveform
 
# Or build from source
# https://github.com/bbc/audiowaveform

With audiowaveform installed, Potato precomputes waveforms on the server. Without it, the browser generates them, which works for files under 30 minutes.

Waveform Caching

Server-side waveforms are worth caching because computing one for an hour-long recording costs seconds, and annotators pay that cost on every page load. The top-level audio_annotation block controls the cache:

yaml

audio_annotation:
  waveform_cache_dir: waveform_cache/     # Where pre-computed waveforms are stored
  waveform_look_ahead: 3                  # Pre-compute the next N instances
  waveform_cache_max_size: 50             # Maximum cached waveform files
  client_fallback_max_duration: 1800      # Longest file the browser will render, in seconds

Potato reads the block only when the task actually has an audio_annotation scheme.

Data Format

Simple Audio Reference

json

[
  {"id": "1", "audio_path": "audio/recording_001.wav"},
  {"id": "2", "audio_path": "audio/recording_002.wav"}
]

yaml

data_files:
  - "data/audio_data.json"
 
item_properties:
  id_key: id
  text_key: audio_path   # The field holding the audio URL

With Transcripts

json

[
  {
    "id": "1",
    "audio_path": "audio/call_001.wav",
    "transcript": "Hello, how can I help you today?"
  }
]

A flat string like this shows the transcript as context next to the waveform. If your transcript carries timings, from Whisper, a cloud ASR API, or a subtitle file, pass the file itself instead and Potato will render its turns as speaker bubbles synced to the audio:

json

[
  {
    "id": "1",
    "conversation": {"audio": "audio/call_001.wav",
                     "transcript": "audio/call_001.srt"}
  }
]

21 transcript and subtitle formats are recognized by content rather than extension. See Transcript Formats.

Output Format

A timeline answer is serialized whole into a single value, so the scheme's entry under labels holds one _data key whose value is a JSON string. Segment times are start_time and end_time, and per-segment answers from segment_schemes land under annotations:

json

{
  "instance_id": "audio_1",
  "user_id": "annotator1",
  "labels": {
    "speaker_turns": {
      "_data": "{\"segments\": [{\"id\": \"seg_1\", \"start_time\": 0.0, \"end_time\": 2.5, \"label\": \"Speaker A\", \"annotations\": {\"clarity\": \"4\"}}, {\"id\": \"seg_2\", \"start_time\": 2.5, \"end_time\": 5.2, \"label\": \"Speaker B\", \"annotations\": {}}]}"
    }
  },
  "spans": {},
  "links": {}
}

Parsed, that _data string is:

json

{
  "segments": [
    {
      "id": "seg_1",
      "start_time": 0.0,
      "end_time": 2.5,
      "label": "Speaker A",
      "annotations": {"clarity": "4"}
    },
    {
      "id": "seg_2",
      "start_time": 2.5,
      "end_time": 5.2,
      "label": "Speaker B",
      "annotations": {}
    }
  ]
}

Video annotation serializes the same way. The single-key shape is why a subsystem comparing two annotators sees _data against _data unless it goes through the reader in server_utils/annotation_values.py.

Keyboard Shortcuts

Potato provides keyboard shortcuts for efficient annotation:

Shortcut	Action
`Space`	Play/Pause
`[`	Set segment start at current position
`]`	Set segment end at current position
`1-9`	Assign label to current segment
`Delete`	Remove current segment
`Left Arrow`	Seek backward 5 seconds
`Right Arrow`	Seek forward 5 seconds
`Up Arrow`	Zoom in
`Down Arrow`	Zoom out
`Home`	Go to start
`End`	Go to end
`+`	Increase playback speed
`-`	Decrease playback speed

Example Configurations

Speaker Diarization

yaml

annotation_task_name: "Speaker Diarization"
task_dir: "."
port: 8000
 
data_files:
  - "data/recordings.json"
 
item_properties:
  id_key: id
  text_key: audio_path
 
annotation_schemes:
  - annotation_type: audio_annotation
    name: speakers
    mode: label
    description: "Identify who is speaking"
    labels:
      - Speaker 1
      - Speaker 2
      - Speaker 3
      - Overlap
      - Silence
    min_segments: 1
 
# Diarization recordings run long, so precompute a few ahead of the annotator.
audio_annotation:
  waveform_cache_dir: waveform_cache/
  waveform_look_ahead: 5
 
output_annotation_dir: "output/"
export_annotation_format: jsonl
 
user_config:
  allow_all_users: true

Transcription Review

yaml

annotation_task_name: "Transcription Quality Review"
task_dir: "."
port: 8000
 
data_files:
  - "data/transcripts.json"
 
item_properties:
  id_key: id
  text_key: audio_path
 
annotation_schemes:
  - annotation_type: audio_annotation
    name: errors
    mode: questions
    description: "Mark transcription errors"
    # Questions mode collects answers per marked region, so the error type and
    # the correction stay attached to the stretch of audio they describe.
    segment_schemes:
      - annotation_type: radio
        name: error_type
        description: "What kind of error is this?"
        labels: ["Substitution", "Deletion", "Insertion", "Speaker mix-up"]
      - annotation_type: text
        name: correction
        description: "What should the transcript say?"
 
  - annotation_type: radio
    name: overall_accuracy
    description: "Overall transcript accuracy"
    labels:
      - Accurate
      - Minor errors
      - Major errors
      - Unusable
 
output_annotation_dir: "output/"
export_annotation_format: jsonl

The configuration above reviews a transcript segment by segment against the waveform. When the transcript came out of an ASR pipeline and already carries timings, the speech_transcript scheme is a closer fit. It reads Whisper, Deepgram, AssemblyAI, SRT, WebVTT and other timed formats directly, and gives annotators segment cards with error tags and a correction box. See Transcript Formats and Annotating Whisper Transcripts.

Call Center QA

yaml

annotation_task_name: "Call Center Quality Assurance"
task_dir: "."
port: 8000
 
data_files:
  - "data/calls.json"
 
item_properties:
  id_key: call_id
  text_key: recording_path
 
annotation_schemes:
  # Segment-level annotation
  - annotation_type: audio_annotation
    name: conversation
    mode: both
    description: "Segment the conversation"
    labels:
      - Agent
      - Customer
      - Hold
      - Silence
    # Who is speaking is a label; how the turn went is a question about it.
    segment_schemes:
      - annotation_type: radio
        name: tone
        description: "Tone of this turn"
        labels: ["Calm", "Neutral", "Escalated"]
 
  # Call-level assessment
  - annotation_type: likert
    name: professionalism
    description: "Agent professionalism"
    size: 5
    min_label: "Poor"
    max_label: "Excellent"
 
  - annotation_type: likert
    name: resolution
    description: "Issue resolution"
    size: 5
    min_label: "Unresolved"
    max_label: "Fully resolved"
 
  - annotation_type: multiselect
    name: issues
    description: "Select any issues observed"
    labels:
      - Long hold time
      - Agent interrupted
      - Incorrect information
      - Missing greeting
      - Unprofessional language
 
  - annotation_type: text
    name: notes
    description: "Additional observations"
    rows: 4
 
output_annotation_dir: "output/"
export_annotation_format: jsonl

Supported Audio Formats

WAV (recommended for best quality)
MP3
OGG
FLAC
M4A
WebM

Performance Tips

Install audiowaveform - Server-side waveforms are what keep hour-long files usable
Enable caching - Set waveform_cache_dir to store pre-generated waveforms
Use WAV for quality - Compressed formats may introduce artifacts
Pre-process audio - Normalize levels, trim unnecessary silence
Consider file sizes - Large files slow down loading
Look ahead - Raise waveform_look_ahead so the next instances are ready before the annotator reaches them

Troubleshooting

Waveform Not Loading

Check audio file path is correct
Verify file format is supported
Install audiowaveform for long files
Check browser console for errors

Slow Performance

Install the audiowaveform tool
Enable waveform caching with waveform_cache_dir
Reduce audio file sizes
Raise waveform_look_ahead to precompute upcoming instances

Segments Not Saving

Ensure output directory is writable
Check annotation format configuration
Verify segment has both start and end times