# Annotating ASR Transcripts: A Worked Example

Source: https://www.potatoannotator.com/blog/annotating-asr-transcripts

This walks one project end to end: 40 recorded research interviews, already transcribed with Whisper, that need to be coded for topic and for who said what. It is the concrete version of the two reference guides, with every decision made rather than described.

If you want the format-by-format detail instead, that lives in [Transcript Formats](/docs/annotation-types/transcript-formats).

![From a folder of ASR output to labeled turns, in four steps](/images/blog/annotating-asr-transcripts.svg "Check what you kept, decide who labels speakers, build the config, export with timings")

## Step 0: look at what you actually have

Before anything else, find out what is in the folder. This takes ten seconds and saves a day:

```bash
potato transcripts ./whisper_out --dry-run
```

```
Scanned 40 file(s):
  interview_01.json      Whisper JSON      42 turns    891.4s  undiarized
  interview_02.json      Whisper JSON      51 turns   1120.8s  undiarized
  interview_03.txt       plain text         1 turns      0.0s  undiarized
  ...

40 item(s), 1683 turn(s).
```

Two things to read out of that. Every file is undiarized, so nothing in this corpus knows who is speaking. And `interview_03.txt` came through as one turn with a zero duration, because a `.txt` from Whisper contains text and nothing else. There are no timings in it to recover.

That third file needs its `.json` found, or the audio re-run. Nothing downstream will fix it.

## Step 1: pick the annotation unit before you pick the labels

The unit question decides more about your agreement numbers than the label set does.

Whisper's segments are roughly utterance-sized, and they break at pauses rather than at anything grammatical. For interview coding that is usually the right unit: an interviewee's answer arrives as several segments, and coding each one separately gives you a finer-grained record than coding the whole answer at once.

Where this goes wrong is with auto-captions from a video platform, where the cue boundaries fall wherever the caption box filled up. Asking annotators to rate "each sentence" over cues like that produces disagreement about where the sentences are, not about the thing you wanted to measure. If that is your input, see [Annotating YouTube Subtitles](/docs/guides/annotating-youtube-subtitles).

Here the segments are usable as-is, so the unit is the turn.

## Step 2: decide who assigns speakers

Whisper does not diarize. Three options, and this is a real decision rather than a formality:

**Re-run with [WhisperX](https://github.com/m-bain/whisperX).** Automatic, fast, and wrong often enough on overlapping speech that someone will need to check it anyway.

```bash
whisperx interview_01.mp3 --model medium --diarize --output_format json
```

**Use a cloud API with diarization on.** Deepgram with `diarize=true`, AssemblyAI with `speaker_labels`, AWS Transcribe, or Rev.ai. Potato reads all four natively.

**Let annotators assign speakers while they listen.** For 40 two-person interviews this is the option we would pick. Two speakers with clearly different roles is the easy case for a person and not always the easy case for a model, and the annotator is listening to the audio regardless.

We go with the third. Undiarized turns render as *Unassigned* with a picker, and the assignment saves with the annotations.

![Transcript turns with hatched grey backgrounds, each labeled Unassigned with a dropdown](/images/docs/transcript-unassigned-turns.png "Undiarized turns arrive as Unassigned, with a picker on each one")

## Step 3: build the data file

```bash
potato transcripts ./whisper_out --media-dir ./audio -o data/interviews.json
```

Transcripts pair to their audio by basename, so `interview_01.json` finds `interview_01.mp3`. Whisper's doubled `interview_01.mp3.json` naming is handled, and the item id comes out as `interview_01`.

The result:

```json
{
  "id": "interview_01",
  "conversation": {
    "audio": "audio/interview_01.mp3",
    "turns": [
      {"turn_id": "t0", "speaker": null, "start": 0.0, "end": 6.5,
       "text": "So I want to start with how the team was structured."}
    ]
  }
}
```

You can skip this step entirely if you would rather keep the transcripts as files. A data file can point straight at them, and Potato reads and normalizes on render:

```json
{"id": "interview_01", "conversation": {"audio": "audio/interview_01.mp3",
                                        "transcript": "whisper_out/interview_01.json"}}
```

## Step 4: write the config

```yaml
annotation_task_name: "Interview Coding"
task_dir: .
data_files:
  - data/interviews.json

item_properties:
  id_key: id
  text_key: conversation

instance_display:
  fields:
    - key: conversation
      type: audio_dialogue
      label: "Interview"
      span_target: true
      display_options:
        show_timestamps: true
        scroll_height: 520px
        allow_speaker_assignment: auto
        speakers:
          - id: interviewer
            name: "Interviewer"
            color: "#7c3aed"
            side: left
          - id: participant
            name: "Participant"
            color: "#059669"
            side: right

annotation_schemes:
  - annotation_type: radio
    name: turn_topic
    description: "What is this turn about?"
    labels: [structure, workload, tooling, morale, other]
    turn_level: true
    turn_binding:
      field: conversation

  - annotation_type: span
    name: quotes
    description: "Highlight anything quotable in the writeup"
    target_field: conversation
    labels:
      - name: quotable
        key_value: "q"
```

Three things are doing work here.

The **speaker roster** gives the two roles stable names, colors, and sides before anyone has assigned a single turn. Without it, speakers still get colors, but they are assigned deterministically per transcript rather than consistently across the corpus.

`turn_level: true` with `turn_binding` attaches the topic question to every turn rather than to the interview as a whole. That is what makes the turn the annotation unit in practice.

`span_target: true` plus the `span` scheme lets a highlight run across turn boundaries, which matters when the quotable passage spans a question and its answer. Offsets stay stable when a speaker gets reassigned.

Run it:

```bash
python potato/flask_server.py start config.yaml -p 8000
```

![Transcript turns rendered as colored speaker bubbles with per-turn play buttons and an inline label question](/images/docs/transcript-speaker-bubbles.png "Each turn gets a play button that plays only that turn, plus its own label question")

Each bubble has a play button that plays just that turn and stops. In practice that is the feature annotators comment on: checking a specific line against the audio stops being a scrubbing exercise.

## Step 5: check agreement on the right thing

Two annotators per interview, and you now have two kinds of agreement to look at.

Topic labels are ordinary categorical agreement at the turn level. Because turn ids are deterministic, the same file always produces the same ids, and the two annotators' labels line up without any alignment step.

Speaker assignment is worth checking separately. If your two annotators disagree about who is talking on 15% of turns, that is a signal about the audio, and it means automatic diarization would have been wrong at least that often without telling you.

See [Inter-Annotator Agreement](/docs/guides/inter-annotator-agreement) for the measures, and [Agreement for Spans](/docs/guides/agreement-for-spans) for the highlights, which need a different treatment because annotators choose boundaries as well as labels.

## Step 6: export

Standard JSON, JSONL, and CSV work as usual. When you want the time alignment to survive into a speech-analysis tool, export tiered annotations to ELAN or Praat:

```bash
python -m potato.export --config config.yaml --format eaf --output ./out/
python -m potato.export --config config.yaml --format textgrid --output ./out/
```

Both round-trip, because Potato reads EAF and TextGrid as input too. Annotate here, refine in [ELAN](https://archive.mpi.nl/tla/elan), read the result back in.

## The four things that go wrong

**Someone kept the `.txt`.** No timings, not recoverable, re-run required. `--dry-run` catches it before you have built anything on top.

**Timings are 1000× off.** Something upstream mixed seconds and milliseconds. Whisper and Deepgram emit float seconds; AssemblyAI, whisper.cpp offsets, and Whisper's TSV emit integer milliseconds.

**Sentence-level schemes over cue-based input.** Covered above, and the most expensive of the four because you do not find out until you compute agreement.

**Trusting diarization nobody checked.** A diarization error propagates into every label attached to that turn, and it looks like annotator disagreement when you go looking for the cause.

## Further reading

- [How to Annotate Whisper Transcripts](/docs/guides/annotating-whisper-transcripts)
- [How to Annotate YouTube Subtitles](/docs/guides/annotating-youtube-subtitles)
- [Transcript Formats](/docs/annotation-types/transcript-formats)
- [Transcript Format Ingestion](/showcase/transcript-formats), a runnable design with six formats side by side
- [Potato 2.7.1: The Transcript Already Exists](/blog/potato-2-7-1-transcripts)