# How to Annotate Whisper Transcripts

Source: https://www.potatoannotator.com/docs/guides/annotating-whisper-transcripts

**Whisper gives you timed segments, and an annotation tool should read them as they are. Keep the `.json` output rather than the `.txt`, run WhisperX if you need speaker labels, then point Potato at the folder.** No conversion script, no reformatting step. For the format-by-format detail, see [Transcript Formats](/docs/annotation-types/transcript-formats).

[Whisper](https://github.com/openai/whisper) is OpenAI's open-source [speech recognition](https://en.wikipedia.org/wiki/Speech_recognition) model, and it has become the default first step for anyone with a pile of audio to work through. What it produces is a set of timed segments, which is most of what an annotation project needs. The gap is everything between "I have transcripts" and "annotators are labeling them."

> **Note:** Potato does not transcribe. ASR runs upstream and Potato ingests the output. This guide covers which upstream choices matter, but the transcription itself happens before Potato sees anything.

## Which Whisper output file should I keep?

Whisper writes several files and the choice matters more than it looks:

```bash
whisper interview_01.mp3 --model medium --output_format json --word_timestamps True
```

| File | Contains | Use it? |
|------|----------|---------|
| `.json` | Segments with start and end times, optionally per-word timings | **Yes, this one** |
| `.srt` / `.vtt` | Segments with timings, no metadata | Yes, works fine |
| `.tsv` | Start and end in milliseconds plus text | Yes |
| `.txt` | Text only, **no timings** | No, nothing to sync to audio |

If the `.txt` is all you kept, the alignment is gone and you cannot recover it without re-running the model. That is the most common reason a transcript arrives in Potato as one undifferentiated block of text.

## Does Whisper label speakers?

No, and this surprises people constantly. Plain Whisper transcribes; it does not tell you who is talking. Every turn arrives unassigned, which is fine for content annotation and painful for anything speaker-related.

[Speaker diarization](https://en.wikipedia.org/wiki/Speaker_diarisation) is a separate step. Three ways to handle it:

**Run [WhisperX](https://github.com/m-bain/whisperX)**, which wraps Whisper with [pyannote](https://github.com/pyannote/pyannote-audio) diarization:

```bash
whisperx interview_01.mp3 --model medium --diarize --output_format json
```

Its output carries a `speaker` field per segment (`SPEAKER_00`, `SPEAKER_01`, and so on) and Potato reads it directly.

**Use a cloud API with diarization enabled.** Deepgram with `diarize=true`, AssemblyAI with `speaker_labels`, AWS Transcribe, and Rev.ai all produce speaker labels, and Potato reads all four natively.

![Three transcript turns with hatched grey backgrounds, each labeled Unassigned with a dropdown caret.](/images/docs/transcript-unassigned-turns.png "Undiarized turns render as Unassigned with a speaker picker")

**Let annotators do it.** When turns arrive undiarized, each bubble renders as *Unassigned* with a picker. On messy audio, overlapping speech, or a room with background noise, a person listening is often more accurate than automatic diarization. For a small corpus this is a reasonable trade rather than a fallback.

## How do I build the data file?

Point the converter at your Whisper output folder:

```bash
potato transcripts ./whisper_out --media-dir ./audio -o data/interviews.json
```

Transcripts pair to media by basename, so `interview_01.json` finds `interview_01.mp3`. Item ids come from the filename, and Whisper's doubled `interview_01.mp3.json` extension is handled without producing an id of `interview_01.mp3`.

Check what it understood before you commit to it:

```bash
potato transcripts ./whisper_out --dry-run
```

```
Scanned 3 file(s):
  interview_01.json      Whisper JSON      42 turns    891.4s  undiarized
  interview_02.json      WhisperX JSON     51 turns   1120.8s  2 speaker(s): SPEAKER_00, SPEAKER_01
  interview_03.json      Whisper JSON      38 turns    754.2s  undiarized

3 item(s), 131 turn(s).
```

A file reporting `plain text` or zero turns is your problem file. Almost always it is a `.txt` that got mixed in with the `.json` output.

## Can I skip the conversion step?

Yes. A data file can point straight at the transcript files on disk:

```json
{"id": "int_001", "conversation": {"audio": "media/int_001.mp3",
                                   "transcript": "media/int_001.srt"}}
```

Potato reads the file, detects the format from its contents, and normalizes it when the instance renders. Your transcripts stay as files you can diff and re-export instead of being baked into a data blob.

## What does the config look like?

The generated data file has this shape:

```json
{
  "id": "interview_01",
  "conversation": {
    "audio": "audio/interview_01.mp3",
    "turns": [
      {"turn_id": "t0", "speaker": "SPEAKER_00", "start": 0.0, "end": 6.5,
       "text": "Welcome back."}
    ]
  }
}
```

And the matching config, which `potato transcripts --emit-config` will print for you:

```yaml
annotation_task_name: "Interview Annotation"
task_dir: .
data_files:
  - data/interviews.json

item_properties:
  id_key: id
  text_key: conversation

instance_display:
  fields:
    - key: conversation
      type: audio_dialogue
      label: "Transcript"
      span_target: true
      display_options:
        show_timestamps: true
        allow_speaker_assignment: auto

annotation_schemes:
  - annotation_type: span
    name: topics
    description: "Highlight topic mentions"
    target_field: conversation
    labels:
      - name: policy
      - name: personal
```

![Transcript turns rendered as colored speaker bubbles with per-turn play buttons, timestamps, and an inline label question under each turn.](/images/docs/transcript-speaker-bubbles.png "The audio_dialogue display, with a per-turn question attached to every turn")

The `audio_dialogue` display renders turns as speaker bubbles synced to the audio, with a play button on each turn that plays only that turn. `span_target: true` lets spans run across turn boundaries, and the offsets stay stable when a speaker gets reassigned.

If you would rather have annotators check the transcription itself than label its content, `speech_transcript` gives you segment cards with error tags and a correction box. For overlap and interruption work, `voice_interaction` gives you a dual-track timeline. Both read the same files.

## How do annotators assign speakers?

Define a roster so speakers get stable names, colors, and sides:

```yaml
display_options:
  allow_speaker_assignment: auto
  speakers:
    - id: interviewer
      name: "Interviewer"
      color: "#7c3aed"
      side: left
    - id: participant
      name: "Participant"
      color: "#059669"
      side: right
```

`auto` turns the picker on when there are undiarized turns to label. Set it to `true` to allow reassignment even when the source did label speakers, which is what you want if you are correcting diarization errors rather than filling in blanks.

Assignments persist with the annotations, keyed to a stable turn id, so they survive a reload.

## How do I get the annotations back out?

JSON, JSONL, and CSV export work as usual. When the time alignment needs to survive, export tiered annotations to ELAN EAF or Praat TextGrid:

```bash
python -m potato.export --config config.yaml --format eaf --output ./out/
python -m potato.export --config config.yaml --format textgrid --output ./out/
```

Both round-trip. Potato reads EAF and TextGrid as input too, so you can annotate in Potato, refine in [ELAN](https://archive.mpi.nl/tla/elan) or [Praat](https://www.fon.hum.uva.nl/praat/), and read the result back in.

## Further reading

- [Transcript Formats](/docs/annotation-types/transcript-formats), every supported format and how it is detected
- [Annotating YouTube Subtitles](/docs/guides/annotating-youtube-subtitles), the caption-file version of this workflow
- [Audio Annotation](/docs/guides/audio-annotation), for waveform work that does not start from a transcript
- [Transcript Format Ingestion](/showcase/transcript-formats), a runnable showcase design with six formats side by side
- [Speaker Diarization](/showcase/speaker-diarization), a runnable showcase design
- [Inter-Annotator Agreement](/docs/guides/inter-annotator-agreement), for measuring agreement on turn-level labels