Speaker Diarization Annotation

This tutorial builds two speaker annotation tasks in Potato. In the first, annotators mark speaker turns from scratch on a waveform. In the second, they correct the speaker labels a diarization system produced upstream. For the full audio scheme reference, see the source documentation.

What diarization produces, and where Potato sits

Speaker diarization splits a recording into stretches that belong to one speaker and gives each stretch a speaker identifier. It answers "who spoke when", and it comes up wherever a recording has more than one voice in it, from meeting minutes and call center analytics to podcast production, research interviews, and legal recordings.

Potato annotates diarization rather than performing it. Speech recognition and speaker diarization run in whatever system you already use for them, such as pyannote, WhisperX, or a cloud transcription API, and Potato ingests the output for humans to check and correct. The two tasks below cover the two cases that follow, one where no diarizer output exists and one where it exists and needs review.

A waveform task for marking turns

With no upstream output to correct, annotators draw the turns themselves on a waveform. The schema is audio_annotation in label mode, and item_properties.text_key names the field holding the audio URL, which is how Potato finds the media for each item.

annotation_task_name: "Speaker Diarization"
task_dir: "."
 
data_files:
  - "data/conversations.json"
 
item_properties:
  id_key: id
  text_key: audio_url
 
output_annotation_dir: "annotation_output/"
export_annotation_format: "jsonl"
 
annotation_schemes:
  - annotation_type: audio_annotation
    name: speakers
    description: "Mark each stretch of speech and label who is speaking"
    mode: label
    zoom_enabled: true
    playback_rate_control: true
    min_segments: 1
    label_requirement:
      required: true
    labels:
      - name: Speaker 1
        color: "#FF6B6B"
        key_value: "1"
      - name: Speaker 2
        color: "#4ECDC4"
        key_value: "2"
      - name: Speaker 3
        color: "#45B7D1"
        key_value: "3"
      - name: Overlap
        color: "#FFEAA7"
        key_value: "4"
      - name: Silence
        color: "#9CA3AF"
        key_value: "5"
 
user_config:
  allow_all_users: true

The Overlap and Silence labels are there to keep the timeline complete. Without a label for crosstalk, an annotator has to decide which of two simultaneous speakers owns the region, and two annotators will decide differently on the same second of audio. Without one for silence, a gap in the timeline is ambiguous between "nobody spoke" and "nobody got to it yet".

min_segments takes effect only alongside label_requirement.required: true. With both set, Next stays blocked until the annotator has marked at least one segment and labeled every segment they marked, and the message says which of the two is missing.

How a segment gets made

The annotator has three ways to create one, and the right one depends on how precise the boundary has to be. Right-click and drag on the zoomed waveform draws a segment directly, and the view auto-scrolls when the drag reaches an edge, so a turn longer than the visible window still takes one gesture. Right-click and drag on the overview draws a coarse segment anywhere in the recording without navigating there first. The [ and ] keys set the boundaries at the playhead and Enter creates the segment, which is the fastest route when the annotator is listening straight through. Left-click keeps its usual job of moving the playhead in all three cases, so navigation and drawing never collide.

Questions on each turn

A speaker label alone will not carry speech acts, speaker roles, or a per-turn quality judgment. Setting mode: both keeps the labels and attaches a set of questions to each segment, which the annotator answers in the segment panel after selecting a region:

annotation_schemes:
  - annotation_type: audio_annotation
    name: turns
    description: "Mark each speaker turn and describe it"
    mode: both
    labels:
      - name: Moderator
        color: "#EF4444"
        key_value: "1"
      - name: Participant 1
        color: "#3B82F6"
        key_value: "2"
      - name: Participant 2
        color: "#10B981"
        key_value: "3"
      - name: Overlap
        color: "#FCD34D"
        key_value: "4"
    segment_schemes:
      - annotation_type: radio
        name: speech_type
        description: "What kind of turn is this?"
        labels: ["Statement", "Question", "Response", "Interruption", "Backchannel"]
      - annotation_type: radio
        name: speaker_confidence
        description: "How sure are you about the speaker?"
        labels: ["Certain", "Probable", "Guess"]

Two limits come with segment questions. Keyboard shortcuts are not bound inside them, because a segment's fields only exist while that segment is selected and a global key would fire against whichever segment happened to be open, so these answers are given by clicking. label_requirement is not enforced on a sub-scheme either, since marking one required would block Next on a question the annotator may not have opened, so completeness on those fields is checked in analysis instead.

A speaker_confidence field of this kind is worth the extra click on hard audio. It separates the turns an annotator was sure about from the ones they guessed, which is the difference between a disagreement that means your label set is wrong and one that means the recording is bad.

Correcting diarizer output

When a diarization system has already segmented the recording, redrawing its boundaries on a waveform wastes the work. The audio_dialogue display renders a diarized transcript as speaker bubbles synced to the audio, each with a play button for that turn alone, so annotators read and listen rather than drawing. It reads WhisperX-style diarized JSON, where each segment carries a start, an end, text, and a speaker, along with subtitle and caption formats, and turns with no speaker render as unassigned for the annotator to fill in.

audio_dialogue is a display rather than an annotation type, so it goes under instance_display.fields and the annotation schemes stay separate. A turn_level scheme bound to that field puts the same question on every turn:

annotation_task_name: "Diarization Review"
task_dir: "."
 
data_files:
  - "data/interviews.json"
 
item_properties:
  id_key: id
  text_key: title
 
output_annotation_dir: "annotation_output/"
export_annotation_format: "jsonl"
 
instance_display:
  fields:
    - key: conversation
      type: audio_dialogue
      label: "Transcript"
      display_options:
        audio_key: audio
        turns_key: turns
        show_timestamps: true
        speakers:
          - id: interviewer
            name: "Interviewer"
            color: "#7c3aed"
            side: left
          - id: participant
            name: "Participant"
            color: "#059669"
            side: right
 
annotation_schemes:
  - annotation_type: radio
    name: speaker_correct
    description: "Is the speaker label on this turn right?"
    labels: ["Correct", "Wrong speaker", "Two speakers in one turn"]
    turn_level: true
    turn_binding:
      field: conversation
 
  - annotation_type: radio
    name: recording_quality
    description: "Overall recording quality"
    labels:
      - "Excellent, all speakers clear"
      - "Good, most speech understandable"
      - "Fair, some difficulty"
      - "Poor, significant problems"
 
user_config:
  allow_all_users: true

The speakers roster fixes a color and a side for each known speaker, so a mislabeled turn shows up as a bubble on the wrong side of the transcript before the annotator has read a word of it. Clicking a turn's speaker name reassigns it, including to a speaker who was not in the roster. A stable turn_id on each turn is what per-turn answers are stored against, so keep those identifiers fixed if you re-ingest the transcripts after fixing anything upstream.

Building the data file from a folder of ASR output is one command:

potato transcripts ./whisper_out --media-dir ./audio -o data/interviews.json

Multiple annotators on the same audio

Speaker boundaries are where annotators disagree most, and one pass over a recording gives you nothing to compare. num_annotators_per_item sets the coverage target for the project:

num_annotators_per_item: 3

Three annotators on a pilot subset is enough to show whether the label set holds up before the full recording set goes out. For choosing the size of that subset and reading the result, see the quality control guide.

Keyboard shortcuts

Audio annotation is paced by the transport controls, so give annotators the shortcuts on day one:

What the output file holds

Segments are written under the scheme name, each with its own identifier, its boundaries in seconds, its label, and any answers from the segment questions:

{
  "turns": {
    "segments": [
      {
        "id": "segment_1",
        "start_time": 0.0,
        "end_time": 5.2,
        "label": "Moderator",
        "annotations": {
          "speech_type": "Statement",
          "speaker_confidence": "Certain"
        }
      },
      {
        "id": "segment_2",
        "start_time": 5.2,
        "end_time": 12.8,
        "label": "Participant 1",
        "annotations": {
          "speech_type": "Response",
          "speaker_confidence": "Certain"
        }
      }
    ]
  }
}

Boundaries are stored in seconds rather than samples, so the same file can be lined up against a transcript from any automatic speech recognition (ASR) system without a conversion step. Each segment keeps its own identifier as well, which is what lets a second annotator's pass be compared against the first segment by segment instead of turn count by turn count.

Long recordings

Waveform rendering is the part that breaks on hour-long audio. Installing the BBC's audiowaveform tool lets Potato compute waveforms on the server rather than in the annotator's browser, which is the difference between a task that opens immediately and one that stalls on every item:

brew install audiowaveform          # macOS
sudo apt-get install audiowaveform  # Ubuntu/Debian

Without it, Potato falls back to generating the waveform in the browser, which suits files under about 30 minutes. Pre-computing the next few instances keeps annotators from waiting between items:

audio_annotation:
  waveform_cache_dir: waveform_cache/
  waveform_look_ahead: 3
  waveform_cache_max_size: 50

Next steps

• Combine with transcription for full meeting notes
• Add emotion detection per speaker
• Read the transcript format reference for the formats audio_dialogue accepts
• Read the audio annotation documentation for the full option list