Skip to content
Tutorials7 min read

Annotating ASR Transcripts: A Worked Example

A walkthrough from a folder of Whisper output to labeled speaker turns, covering the annotation unit, diarization, the config, and export with timings intact.

Potato Team

This tutorial takes one project from a folder of Whisper output to labeled speaker turns. The project is 40 recorded research interviews that need to be coded for topic and for who said what. The reference guides lay out the options, and this post picks one at each step. For the format-by-format detail instead, see Transcript Formats.

From a folder of ASR output to labeled turns, in four stepsCheck what you kept, decide who labels speakers, build the config, export with timings

Step 0: inspect the transcript folder

Before anything else, find out what is in the folder:

bash
potato transcripts ./whisper_out --dry-run
text
Scanned 40 file(s):
  interview_01.json      Whisper JSON      42 turns    891.4s  undiarized
  interview_02.json      Whisper JSON      51 turns   1120.8s  undiarized
  interview_03.txt       plain text         1 turns      0.0s  undiarized
  ...

40 item(s), 1683 turn(s).

The output shows two things. Every file is undiarized, so nothing in this corpus knows who is speaking. And interview_03.txt came through as one turn with a zero duration, because a .txt from Whisper contains text and nothing else, with no timings in it to recover. That file needs its .json found or the audio re-run, since nothing downstream will fix it.

Step 1: pick the annotation unit

The unit question decides more about your agreement numbers than the label set does. Whisper's segments are roughly utterance-sized, and they break at pauses rather than at anything grammatical. For interview coding, those segments are usually the right unit, because an interviewee's answer arrives as several segments, and coding each one separately gives you a finer-grained record than coding the whole answer at once. The segments in this project are usable as-is, so the unit is the turn.

Auto-captions from a video platform are where this goes wrong, because their cue boundaries fall wherever the caption box filled up. Asking annotators to rate "each sentence" over cues like that produces disagreement about where the sentences are instead of about the thing you wanted to measure. If that is your input, see Annotating YouTube Subtitles.

Step 2: decide who assigns speakers

Whisper does not diarize, so you have three options:

Re-run with WhisperX. WhisperX is automatic and fast, and it is wrong often enough on overlapping speech that someone will need to check it anyway:

bash
whisperx interview_01.mp3 --model medium --diarize --output_format json

Use a cloud API with diarization on. Deepgram with diarize=true, AssemblyAI with speaker_labels, AWS Transcribe, and Rev.ai all work, and Potato reads all four natively.

Let annotators assign speakers while they listen. For 40 two-person interviews, this is the option we would pick. Two speakers with clearly different roles is the easy case for a person and not always the easy case for a model, and the annotator is listening to the audio regardless.

We go with the third. Undiarized turns render as Unassigned with a picker, and the assignment saves with the annotations.

Transcript turns with hatched gray backgrounds, each labeled Unassigned with a dropdownUndiarized turns arrive as Unassigned, with a picker on each one

Step 3: build the data file

Build it with the converter:

bash
potato transcripts ./whisper_out --media-dir ./audio -o data/interviews.json

Transcripts pair to their audio by basename, so interview_01.json finds interview_01.mp3. Whisper's doubled interview_01.mp3.json naming is handled, and the item id comes out as interview_01. The output looks like this:

json
{
  "id": "interview_01",
  "conversation": {
    "audio": "audio/interview_01.mp3",
    "turns": [
      {"turn_id": "t0", "speaker": null, "start": 0.0, "end": 6.5,
       "text": "So I want to start with how the team was structured."}
    ]
  }
}

You can skip this step entirely if you would rather keep the transcripts as files. A data file can point straight at them, and Potato reads and normalizes on render:

json
{"id": "interview_01", "conversation": {"audio": "audio/interview_01.mp3",
                                        "transcript": "whisper_out/interview_01.json"}}

Step 4: write the config

Save this as config.yaml:

yaml
annotation_task_name: "Interview Coding"
task_dir: .
output_annotation_dir: annotation_output/
data_files:
  - data/interviews.json
 
item_properties:
  id_key: id
  text_key: conversation
 
instance_display:
  fields:
    - key: conversation
      type: audio_dialogue
      label: "Interview"
      span_target: true
      display_options:
        show_timestamps: true
        scroll_height: 520px
        allow_speaker_assignment: auto
        speakers:
          - id: interviewer
            name: "Interviewer"
            color: "#7c3aed"
            side: left
          - id: participant
            name: "Participant"
            color: "#059669"
            side: right
 
annotation_schemes:
  - annotation_type: radio
    name: turn_topic
    description: "What is this turn about?"
    labels: [structure, workload, tooling, morale, other]
    turn_level: true
    turn_binding:
      field: conversation
 
  - annotation_type: span
    name: quotes
    description: "Highlight anything quotable in the writeup"
    labels:
      - name: quotable
        key_value: "q"

Three parts of that config matter. The speaker roster gives the two roles stable names, colors, and sides before anyone has assigned a single turn. Without it, speakers still get colors, but they are assigned deterministically per transcript rather than consistently across the corpus.

turn_level: true with turn_binding attaches the topic question to every turn rather than to the interview as a whole, which is what makes the turn the annotation unit in practice. span_target: true plus the span scheme lets a highlight run across turn boundaries, which matters when the quotable passage spans a question and its answer. Offsets stay stable when a speaker gets reassigned.

Start the server:

bash
potato start config.yaml -p 8000

Each speaker bubble has a play button that plays only that turn and stops. Annotators mention the play button more than anything else, because checking a specific line against the audio stops being a scrubbing exercise.

Transcript turns rendered as colored speaker bubbles with per-turn play buttons and an inline label questionEach turn gets a play button that plays only that turn, plus its own label question

Step 5: check topic and speaker agreement

With two annotators per interview, you have two kinds of agreement to look at. Topic labels are ordinary categorical agreement at the turn level. Because turn ids are deterministic, the same file always produces the same ids, and the two annotators' labels line up without any alignment step.

Speaker assignment is worth checking separately. If your two annotators disagree about who is talking on 15% of turns, the disagreement is a signal about the audio, and those turns need a second listen before you trust any topic label attached to them. See Inter-Annotator Agreement for the measures, and Agreement for Spans for the highlights, which need a different treatment because annotators choose boundaries as well as labels.

Step 6: export

Standard JSON, JSONL, and CSV work as usual. When you want the time alignment to survive into a speech-analysis tool, export tiered annotations to ELAN or Praat:

bash
python -m potato.export --config config.yaml --format eaf --output ./out/
python -m potato.export --config config.yaml --format textgrid --output ./out/

Both formats round-trip, because Potato reads EAF and TextGrid as input too, so you can annotate in Potato, refine in ELAN, and read the result back in.

Four common failures in transcript projects

Check for these four problems as you go.

Timings are 1000× off. Something upstream mixed seconds and milliseconds. Whisper and Deepgram emit float seconds, while AssemblyAI, whisper.cpp offsets, and Whisper's TSV emit integer milliseconds.

Sentence-level schemes over cue-based input. Step 1 covers this problem. It is the most expensive of the four, because you do not find out until you compute agreement.

Trusting diarization nobody checked. A diarization error propagates into every label attached to that turn, and it looks like annotator disagreement when you go looking for the cause.

Someone kept the .txt. It has no timings, and they cannot be recovered without a re-run. --dry-run catches the problem before you have built anything on top.

Further reading