Audio Annotation

Segment audio in Potato and label time regions on an interactive waveform, with playback speed control, zoom, and per-segment questions for transcription.

Potato's audio annotation tool enables annotators to segment audio files and assign labels to time regions through a waveform-based interface.

Features

Waveform visualization
Time-based segment creation
Label assignment to segments
Playback controls with variable speed
Zoom and scroll navigation
Keyboard shortcuts
Server-side waveform caching

Basic Configuration

The default mode asks the annotator to mark regions and label each one, so a diarization task needs only the labels:

yaml

annotation_schemes:
  - name: "speakers"
    description: "Mark when each speaker is talking"
    annotation_type: "audio_annotation"
    labels:
      - name: "Speaker 1"
        color: "#3B82F6"
      - name: "Speaker 2"
        color: "#10B981"

Configuration Options

Field	Type	Default	Description
`name`	string	Required	Unique identifier for the annotation
`description`	string	Required	Instructions shown to annotators
`annotation_type`	string	Required	Must be `"audio_annotation"`
`mode`	string	`"label"`	Annotation mode: `"label"`, `"questions"`, or `"both"`
`labels`	list	Conditional	Required for `label` or `both` modes
`segment_schemes`	list	Conditional	Required for `questions` or `both` modes
`min_segments`	integer	0	Minimum segments required
`max_segments`	integer	null	Maximum segments allowed (null = unlimited)
`zoom_enabled`	boolean	true	Enable zoom controls
`playback_rate_control`	boolean	false	Show playback speed selector
`waveform`	boolean	true	Show the waveform
`spectrogram`	boolean	false	Show a spectrogram alongside the waveform
`spectrogram_options`	dict	—	FFT size, frequency range, and color map for the spectrogram
`source_field`	string	`""`	Display field this schema annotates, when the item carries more than one audio file

Potato refuses to load a scheme that omits the input its mode needs, so label and both fail without labels, and questions and both fail without segment_schemes.

Label Configuration

Giving each label a key_value lets an annotator label a region without leaving the waveform, which matters in diarization where segments come in quick succession:

yaml

labels:
  - name: "speech"
    color: "#3B82F6"
    key_value: "1"
  - name: "music"
    color: "#10B981"
    key_value: "2"
  - name: "silence"
    color: "#64748B"
    key_value: "3"

Annotation Modes

Label Mode (Default)

Segments receive category labels. Use this mode when every segment answers the same one-choice question:

yaml

annotation_schemes:
  - name: "emotion"
    description: "Label the emotion in each segment"
    annotation_type: "audio_annotation"
    mode: "label"
    labels:
      - name: "happy"
        color: "#22C55E"
      - name: "sad"
        color: "#3B82F6"
      - name: "angry"
        color: "#EF4444"
      - name: "neutral"
        color: "#64748B"

Questions Mode

Each segment answers dedicated questions. The sub-schemes in segment_schemes are ordinary annotation schemes rendered per segment, so each one needs whatever its own type requires, such as size, min_label, and max_label for a likert scale:

yaml

annotation_schemes:
  - name: "transcription"
    description: "Transcribe each segment"
    annotation_type: "audio_annotation"
    mode: "questions"
    segment_schemes:
      - name: "transcript"
        annotation_type: "text"
        description: "Enter the transcription"
      - name: "confidence"
        annotation_type: "likert"
        description: "How confident are you?"
        size: 5
        min_label: "Not confident"
        max_label: "Very confident"

Both Mode

Combines labeling with per-segment questionnaires. It costs the annotator a label and a form for every region, so reserve it for passes that need both:

yaml

annotation_schemes:
  - name: "detailed_diarization"
    description: "Label speakers and add notes"
    annotation_type: "audio_annotation"
    mode: "both"
    labels:
      - name: "Speaker A"
        color: "#3B82F6"
      - name: "Speaker B"
        color: "#10B981"
    segment_schemes:
      - name: "notes"
        annotation_type: "text"
        description: "Any notes about this segment?"

Global Audio Configuration

Potato computes waveform data on the server and caches it, because a browser that renders the waveform itself has to download and decode the whole file first. Configure the cache in your config file:

yaml

audio_annotation:
  waveform_cache_dir: "waveform_cache/"
  waveform_look_ahead: 5
  waveform_cache_max_size: 1000
  client_fallback_max_duration: 1800

Field	Default	Description
`waveform_cache_dir`	—	Directory for cached waveform data
`waveform_look_ahead`	5	Number of upcoming instances to pre-compute
`waveform_cache_max_size`	100	Maximum number of cached waveform files
`client_fallback_max_duration`	1800	Longest file, in seconds, the browser will render when no cached waveform is available

Raise waveform_look_ahead when annotators move through items faster than the server precomputes, and raise waveform_cache_max_size when the study covers more files than the cache holds.

Examples

Speaker Diarization

Diarization is where playback speed earns its place, since annotators slow the audio down to place a boundary on a speaker change and speed it up across long single-speaker stretches:

yaml

annotation_schemes:
  - name: "diarization"
    description: "Identify who is speaking at each moment"
    annotation_type: "audio_annotation"
    mode: "label"
    labels:
      - name: "Interviewer"
        color: "#8B5CF6"
        key_value: "1"
      - name: "Guest"
        color: "#EC4899"
        key_value: "2"
      - name: "Overlap"
        color: "#F59E0B"
        key_value: "3"
    zoom_enabled: true
    playback_rate_control: true

Sound Event Detection

Setting min_segments: 1 stops an annotator submitting a recording with nothing marked. Use it only where every recording contains at least one event:

yaml

annotation_schemes:
  - name: "sound_events"
    description: "Mark all sound events"
    annotation_type: "audio_annotation"
    labels:
      - name: "speech"
        color: "#3B82F6"
      - name: "music"
        color: "#10B981"
      - name: "applause"
        color: "#F59E0B"
      - name: "laughter"
        color: "#EC4899"
      - name: "silence"
        color: "#64748B"
    min_segments: 1

Transcription Review

Pairing a free-text field with a quality rating records why a transcript was left unchanged, which separates a clean segment from an unintelligible one in the output:

yaml

annotation_schemes:
  - name: "transcription_review"
    description: "Review and correct the transcription for each segment"
    annotation_type: "audio_annotation"
    mode: "questions"
    segment_schemes:
      - name: "transcript"
        annotation_type: "text"
        description: "Enter or correct the transcription"
      - name: "quality"
        annotation_type: "radio"
        description: "Audio quality"
        labels:
          - "Clear"
          - "Noisy"
          - "Unintelligible"

Already have a transcript?

Everything above starts from a blank waveform, where the annotator listens, marks boundaries, and labels what they find. A transcript that already exists, from Whisper or from downloaded captions, calls for the other path instead. Potato reads 21 transcript and subtitle formats and renders the turns as speaker bubbles synced to the audio, so annotators label existing segments instead of re-creating them.

json

{"id": "int_001", "conversation": {"audio": "media/int_001.mp3",
                                   "transcript": "media/int_001.srt"}}

See Transcript Formats for the format list, and Annotating Whisper Transcripts for the walkthrough. Waveform segmentation and transcript annotation can also run in the same task when you need both.

Keyboard Shortcuts

Key	Action
`Space`	Play/pause
`←` / `→`	Seek 5 seconds
`[` / `]`	Set segment start/end
`Enter`	Create segment
`Del`	Delete segment
`+` / `-`	Zoom in/out
`0`	Fit to view

In label and both modes, each label with a key_value adds its own key.

Data Format

Input Data

Your data file should include audio file paths or URLs:

json

[
  {
    "id": "audio_001",
    "audio_url": "https://example.com/audio/recording1.mp3"
  },
  {
    "id": "audio_002",
    "audio_url": "/data/audio/recording2.wav"
  }
]

Point text_key at the field holding the audio path, since that is the field Potato hands to the annotation widget:

yaml

item_properties:
  id_key: id
  text_key: audio_url

Output Format

json

{
  "id": "audio_001",
  "annotations": {
    "diarization": [
      {
        "start": 0.0,
        "end": 5.5,
        "label": "Interviewer"
      },
      {
        "start": 5.5,
        "end": 12.3,
        "label": "Guest"
      },
      {
        "start": 12.3,
        "end": 14.0,
        "label": "Overlap"
      }
    ]
  }
}

For questions mode, segments include nested responses:

json

{
  "start": 0.0,
  "end": 5.5,
  "transcript": "Hello and welcome to the show.",
  "quality": "Clear"
}

Supported Audio Formats

MP3 (recommended)
WAV
OGG
M4A

Best Practices

Pre-cache waveforms - Use server-side caching for large datasets
Enable playback control - Variable speed helps with precise segmentation
Use keyboard shortcuts - Faster than clicking, and the annotator's eyes stay on the waveform
Define clear boundaries - Specify what constitutes segment start/end
Choose appropriate mode - Use "label" for classification, "questions" for detailed annotation
Set segment limits - Use min_segments to ensure coverage