Train Speech and Audio Models on Your Annotations

New in v2.14.0

This tutorial trains speech and audio models on the answers annotators give in Potato and sends each model's output back to the annotators as pre-labels. The page is written for a study owner who already corrects transcripts, speaker turns, or animal calls in Potato and wants a model that learns from those corrections. The main path imports a speaker diarization file, has annotators correct the speaker turns, trains a diarization model on the corrected turns, and exports the corrections as RTTM. Three shorter sections then fine-tune Whisper on corrected transcript turns, train a box detector on spectrograms, and train a tagger for sound events on the waveform. For the training loop in general, read the model training overview first, and Train your first model if you have never started a run.

The four audio trainers

Each trainer learns from one kind of scheme, and the Training page offers a scheme only the trainers that fit the scheme:

Every audio trainer reads recordings from files on the server's disk that soundfile opens, e.g., WAV, FLAC, or OGG, and MP3 with a recent libsndfile. A path such as /media/meet000.wav names a file in the media/ folder of the task directory, the folder the annotation page plays recordings from. An item whose recording is an http(s) URL is left out of training with a warning. Download remote recordings into media/ before training. The trainers reference lists every parameter.

Install the trainers into the environment that runs Potato:

pip install "potato-annotation[train-audio]"

The [train-audio] extra installs PyTorch, transformers, and soundfile. The spectrogram detectors need [train-vision] instead, or pip install "potato-annotation[train]" for every text, image, audio, and LLM trainer at once. A trainer whose packages are missing stays on the Training page as "not installed", with the command that installs the packages.

Importing a diarization as an RTTM file

Speaker diarization splits a recording into turns by who is speaking. pyannote and NeMo write diarizations as RTTM, a text format with one SPEAKER line per turn, and dscore and md-eval score RTTM files. A line names the recording's file id, the channel, the turn's onset and duration in seconds, and the speaker:

SPEAKER meet000 1 0.200 1.055 <NA> <NA> SPEAKER_01 <NA> <NA>
SPEAKER meet000 1 1.539 0.755 <NA> <NA> SPEAKER_00 <NA> <NA>

Put the RTTM files and the recordings in one folder, with each recording named after its file id (meet000.wav for meet000), and import the folder:

potato import -f rttm -i diarization/ -o interviews \
  --speaker-map "SPEAKER_00=Interviewer,SPEAKER_01=Guest"

The importer makes one item per recording, copies the recording into interviews/media/, and writes config.yaml, data/diarization.jsonl, and a README. On the 16-recording test project behind this page, the summary line read:

Imported 16 item(s), 1 question(s), 0 answer(s) from 0 annotator(s), and 16 model suggestion(s) on 16 item(s) from RTTM speaker diarization.

The importer takes three flags of its own:

• --rttm-scheme audio|tiered picks the scheme the turns land on. The default audio makes an audio_annotation scheme named speakers whose labels are the speakers. tiered makes a tiered_annotation scheme named tiers with one speaker tier. Use tiered for a study that transcribes turn by turn afterwards.
• --speaker-map renames speakers, and a speaker the map does not name keeps its RTTM name. Use the map only when a name means the same person in every recording. A diarizer numbers speakers within each file, and SPEAKER_00 in one interview and SPEAKER_00 in the next can be different people.
• --media-dir DIR names the folder holding the recordings when the recordings are not beside the RTTM files.

The importer reads only SPEAKER lines and counts the other line types as skipped. Zero-length turns, turns with a negative onset, turns with no speaker name (<NA>), and a turn listed twice are skipped as well. RTTM separates fields with spaces, and the importer reads a speaker name that contains a space only up to the space, with a warning. Importing a Project covers the import in full.

The diarization config

The generated config already shows the imported turns as pre-annotations. Add an admin key and a model_training block to the end of config.yaml. The complete file below passes python -m potato.validate_cli config.yaml --strict:

port: 8000
annotation_task_name: diarization (imported)
task_dir: .
output_annotation_dir: annotation_output/
data_files:
- data/diarization.jsonl
item_properties:
  id_key: id
  text_key: text
user_config:
  allow_all_users: true
  users: []
site_dir: default
pre_annotation:
  enabled: true
  field: predictions
  allow_modification: true
annotation_schemes:
- annotation_type: audio_annotation
  name: speakers
  description: Check and correct who speaks when.
  source_field: audio_url
  mode: label
  labels:
  - name: Guest
  - name: Interviewer
admin_api_key: "change-me"
model_training:
  enabled: true

The importer sorts the speaker labels alphabetically, and Guest comes before Interviewer. The model's write-back goes to the item field predictions by default, the same field the import's pre_annotation block reads. Potato refuses to start when an enabled pre_annotation block names a different field from writeback.field, because no training pre-label would then show. Leave writeback out of the config until the first run has a held-out score worth showing to annotators.

Correcting the speaker turns

Start the server from the project folder:

cd interviews
potato start config.yaml -p 8000

Each recording opens with the diarizer's turns drawn on the waveform. Annotators move a turn's edges, change a turn's speaker, delete a turn of noise, and draw any turn the diarizer missed. The pre-annotations are stored only once an annotator saves the item. An item nobody opened has no answer and does not train. Speaker Diarization Annotation covers the waveform controls and keyboard shortcuts.

The diarization trainer needs at least 4 annotated recordings in the train split, and the default split puts about 70% of the labeled recordings in train. Correct at least 8 recordings before the first run, and more for a held-out score you can trust.

Training the diarization model

Open http://localhost:8000/admin, enter the admin key, and go to /admin/training. Pick diarization as the trainer and speakers as the scheme. The Training page also offers hf-audio-tagger for the same scheme. hf-audio-tagger suits labels that name kinds of sound rather than speakers. For two-person interviews, set the Parameters field to:

{"max_speakers": 2}

Click Train now. The same run can be started from a script with the admin key:

curl -X POST -H "X-API-Key: change-me" -H "Content-Type: application/json" \
  http://localhost:8000/admin/training/api/runs \
  -d '{"trainer": "diarization", "schemas": ["speakers"], "params": {"max_speakers": 2}}'

diarization follows the recipe of pyannote's segmentation model (Bredin and Laurent, 2021) without pyannote or pyannote's gated weights, and fine-tunes WavLM (microsoft/wavlm-base-plus) with a frame classifier for up to max_speakers speakers. The model sees chunk_s seconds of audio at a time, 10 by default. Each chunk's loss is computed under the best matching of the model's speaker outputs to the chunk's annotated speakers. The matching matters because a speaker name refers to different people in different recordings, and a classifier over names would have no consistent target. The parameters below shape the predicted turns:

Remote training sends a run to a GPU server over ssh or to a SLURM cluster.

Reading the diarization score

The run page and GET /admin/training/api/status report the run's score on held-out recordings. For diarization, the headline is diarization_accuracy, defined as 1 minus the diarization error rate (DER) and floored at 0. DER adds up missed speech, false alarm, and speaker confusion and divides the sum by the annotated speech. Potato measures DER on a 10 ms grid with no collar, under the best one-to-one mapping of predicted to annotated speakers. The run reports the three error types separately as well. Read the three numbers to see whether the model misses speech, invents speech, or mixes up speakers. DER ignores whether a turn is called Interviewer or SPEAKER_01 and counts whether the speaker changes at the right times.

A metric name includes the split, as in val_der or test_diarization_accuracy. A recording with annotated turns and no prediction counts as wrong. A model therefore cannot raise the score by skipping hard recordings.

Publishing the predicted turns

Once the held-out score supports showing the model's turns to annotators, add the scheme to writeback and restart the server:

model_training:
  enabled: true
  writeback:
    schemas: [speakers]
    include_unlabeled: true

include_unlabeled: true also predicts on the recordings nobody has corrected yet. Without the setting, a run predicts only on the held-out val and test recordings. Train again after the restart. The write-back report in GET /admin/training/api/status gives the count. On the test project, the report read "Published 7 prediction(s) as prelabels." for 3 val recordings and 4 uncorrected recordings, and the recordings in the train split kept the imported RTTM turns.

The model names speakers in order of first speech within each recording, using the scheme's labels first and then SPEAKER_00, SPEAKER_01, and so on. The first speaker heard in each recording is therefore Guest in this project, whether or not that person is the guest. The model has no voice embeddings to recognize a person by. A speaker who is silent through the overlap between two chunks and speaks again later gets a new name. Tell annotators to check every predicted turn's speaker name, and not only the turn's edges. Model in the loop covers reviewing pre-labels least confident first and retraining as corrections arrive.

Exporting the corrected turns as RTTM

The segments export writes the corrected turns back out as RTTM, one file per annotator:

python -m potato.export --config config.yaml --format segments --output ./export/

The export writes export/segments.jsonl, one row per turn, and export/rttm/<annotator>.rttm. The file id in each RTTM line is the recording's name without its extension. pyannote.metrics, dscore, or md-eval can then score the diarizer's original output against the corrected file unchanged. On the test project, the first lines of export/rttm/ana.rttm read:

SPEAKER meet000 1 0.200 1.055 <NA> <NA> Guest <NA> <NA>
SPEAKER meet000 1 1.539 0.755 <NA> <NA> Interviewer <NA> <NA>

Diarization on a speaker tier

Import with --rttm-scheme tiered when annotators will add other tiers, such as a transcription tier, to the same timeline:

potato import -f rttm -i diarization/ -o interviews --rttm-scheme tiered

The import writes a tiered_annotation scheme named tiers with one speaker tier. Add admin_api_key and model_training as above, with schemas: [tiers] under writeback, and the config passes validate_cli --strict. diarization trains on the speaker tier and writes predicted turns back onto the same tier. The trainer picks the tier named speaker or speakers. Failing that, the trainer picks the one tier whose name contains "speaker", and a tier named non-speaker does not count. The last fallback is the only tier with labels. When two tiers qualify, or several tiers have labels and none is named for speakers, the run leaves the recordings out with a warning that names the tiers.

Fine-tuning Whisper on corrected transcripts

whisper-asr fine-tunes Whisper on the transcript turns annotators corrected, then suggests corrections on the turns nobody has checked. The section starts from a folder of recordings with no transcript.

Transcribing the recordings

Put the recordings in the project's media/ folder and transcribe the recordings on the server's machine:

pip install "potato-annotation[transcribe]"
potato transcripts media --transcribe --asr-model small.en --asr-word-timestamps \
  --media-url-prefix /media -o data/interviews.json

potato transcripts --transcribe runs WhisperX when WhisperX is installed and works on the machine, and faster-whisper otherwise. The [transcribe] extra installs faster-whisper, and WhisperX is a separate install (pip install whisperx) that brings its own PyTorch. Potato falls back to faster-whisper when WhisperX is missing, does not import, cannot run on the device, or cannot load the model, and the summary names the engine for each file. --asr-engine whisperx or --asr-engine faster-whisper runs one engine with no fallback. --media-url-prefix /media writes each recording's path as /media/<file>, the address the annotation page plays the recording from.

--asr-word-timestamps times each word by forced alignment and lets WhisperX split a segment at a sentence end. On the test recordings, WhisperX returned one turn per 10-second recording without the flag and three turns per recording with the flag. Shorter turns give whisper-asr more examples, and Whisper's input window is 30 seconds. Transcript Formats and How to Annotate Whisper Transcripts cover the transcription options, speaker labels, and existing Whisper output.

The transcript config

A speech_transcript scheme shows each turn of the transcript with its time span and a correction box. The complete config below passes validate_cli --strict:

annotation_task_name: "Interview transcript correction"
task_dir: .
output_annotation_dir: annotation_output/
data_files:
  - data/interviews.json
item_properties:
  id_key: id
  text_key: conversation
admin_api_key: "change-me"
annotation_schemes:
  - annotation_type: speech_transcript
    name: transcript
    description: "Play each turn and type the right text wherever the transcript is wrong."
    segments_key: conversation
    error_types: [asr_error]
model_training:
  enabled: true
  trainer_params:
    whisper-asr:
      model_name: openai/whisper-small
      language: en

segments_key: conversation names the field potato transcripts writes the transcript under. error_types: [asr_error] leaves one error flag, because the study covers transcription errors only. language: en fixes the language for training and prediction. Left unset, a multilingual Whisper detects the language once from up to 16 training turns, and the run log warns when fewer than three in four of the sampled turns agree.

Corrected turns as training examples

Each corrected turn is one example, made of the turn's audio cut from the recording by the turn's start and end, together with the corrected text. A turn nobody corrected is not an example, because the scheme saves only the turns an annotator touched, and an untouched turn may be right or may be unread. A flagged turn with no correction is not an example either. Turns longer than 30 seconds are left out with a warning. The trainer refuses to run with fewer than 10 corrected turns in the train split.

Correct every wrong turn on the recordings you open, because the held-out score is computed over the corrected turns. On the 12 test recordings, WhisperX with tiny.en got 11 of 36 turns wrong, with errors such as "Epsilanti" for Ypsilanti, "Cary Town" for Kerrytown, and a dropped "I" at the start of a turn.

Training and comparing against the transcript

Pick whisper-asr and the transcript scheme on the Training page and click Train now. The headline score is word_accuracy, which is 1 minus the word error rate (WER) over the corrected turns. The run also reports wer_asr, the original transcript's WER on the same turns, and a turn the model left alone is scored as the transcript's text. Compare the two numbers before publishing anything. A fine-tuned model is worth showing to annotators only when the model's WER is below wer_asr.

The test run behind this page shows why the comparison matters. With 8 corrected turns in train, min_instances lowered to 4, and openai/whisper-tiny trained for 3 epochs, the run scored a test WER of 0.31 against 0.19 for the original transcript. A model at that level would replace correct text with wrong text. Collect more corrections, or start from a larger checkpoint, and train again before adding writeback.

Publishing suggested corrections

Once the model beats the transcript, add the scheme to writeback, restart, and train again:

model_training:
  enabled: true
  trainer_params:
    whisper-asr:
      model_name: openai/whisper-small
      language: en
  writeback:
    schemas: [transcript]
    include_unlabeled: true

Prediction transcribes every turn of every held-out and unlabeled recording. A turn whose new text differs from the transcript becomes a suggested correction, flagged asr_error, with the model's mean token probability as the suggestion's confidence. The suggestions fill the correction boxes, and an annotator keeps, edits, or clears each suggestion.

Transcribing new recordings with the fine-tuned model

By default the run also writes ct2/, an int8 CTranslate2 copy of the model, among the run's artifacts. The copy needs the ctranslate2 package, which the [transcribe] extra installs with faster-whisper. Pass the copy's path to potato transcripts to transcribe the next batch of recordings with the fine-tuned model:

potato transcripts media/batch2 --transcribe \
  --asr-model annotation_output/training/runs/<run_id>/artifacts/ct2 \
  --media-url-prefix /media/batch2 -o data/batch2.json

WhisperX and faster-whisper both load the copy. The run page shows each run's id, and Configuration describes the run folders.

Detecting calls on spectrograms

A spectrogram_annotation scheme has annotators draw time-frequency boxes around calls. torchvision-detect and ultralytics-detect train on those boxes by rendering each recording as a spectrogram image the way the widget draws the spectrogram. The rendering puts frequency on a linear axis from the scheme's min_freq to max_freq, or half the sample rate when max_freq is unset, at 256 pixels high and 100 pixels per second. Each box becomes an image box. Time and frequency map linearly to pixels. A box's IoU on the image therefore equals the box's IoU in seconds by hertz. A box with no frequency range, as an Audacity label or a BirdNET row gives, covers the whole band. A recording of 41 seconds or more renders at least 4,096 pixels wide and is cut into windows and stitched back.

Boxes usually come from an import. potato import reads Raven selection tables, Audacity label tracks, and BirdNET-Analyzer output into a spectrogram_annotation project, as Bioacoustics in Potato and Verify BirdNET Detections on a Spectrogram describe. The trainers learn from the boxes annotators drew or corrected, and the detectors read the spectrogram pixels rather than BirdNET or Perch embeddings.

The config below passes validate_cli --strict and sets the detector's score_threshold to 0.5:

annotation_task_name: "Dawn chorus calls"
task_dir: .
output_annotation_dir: annotation_output/
data_files:
  - data/recordings.jsonl
item_properties:
  id_key: id
  text_key: id
admin_api_key: "change-me"
annotation_schemes:
  - annotation_type: spectrogram_annotation
    name: calls
    description: "Draw a box around each call and pick its type."
    source_field: audio_url
    max_freq: 8000
    labels:
      - name: whistle
      - name: chirp
model_training:
  enabled: true
  trainer_params:
    torchvision-detect:
      score_threshold: 0.5
  writeback:
    schemas: [calls]
    include_unlabeled: true

The spectrum of a 16 kHz recording ends at 8 kHz, and max_freq: 8000 shows the whole band. The default score_threshold of 0.25 keeps more of the detector's guesses. On the 24-recording test project, torchvision-detect at the default threshold predicted 34 boxes for 10 annotated calls on the test recordings, and each extra box is one more box for an annotator to delete.

Each data row names its recording, as in {"id": "site_00", "audio_url": "/media/site_00.wav"}. A detector trains only on recordings with at least one box, because a recording with no boxes looks the same as a recording nobody checked. The detectors need 20 annotated recordings in the train split by default (min_instances).

torchvision-detect fine-tunes a Faster R-CNN from torchvision's COCO weights, and ultralytics-detect trains a YOLO detector. The YOLO detector is faster to train and to run. Ultralytics is licensed AGPL-3.0, and a model trained and distributed with Ultralytics is subject to the AGPL's terms. The Training page asks you to confirm the license before an ultralytics-detect run, and an API run needs "accept_licence": true. Both detectors are scored by mAP at IoU 0.5 on the rendered spectrogram, and both write predicted boxes back in start_s, end_s, low_hz, and high_hz, marked as model output, to prefill the widget. Vision models covers the detectors' parameters and tiling.

Tagging sound events with hf-audio-tagger

hf-audio-tagger learns the labeled time segments of an audio_annotation scheme, such as speech, music, calls, or coughs. The trainer fine-tunes a speech encoder, facebook/wav2vec2-base by default, to say which labels are active in each frame of about 20 ms. Each label is scored separately. Overlapping segments with different labels are therefore learned as annotated. Predicted segments go back as editable regions on the waveform. The config below passes validate_cli --strict:

annotation_task_name: "Dawn chorus events"
task_dir: .
output_annotation_dir: annotation_output/
data_files:
  - data/recordings.jsonl
item_properties:
  id_key: id
  text_key: id
admin_api_key: "change-me"
annotation_schemes:
  - annotation_type: audio_annotation
    name: events
    description: "Mark every call on the waveform and pick its type."
    source_field: audio_url
    mode: label
    labels:
      - name: whistle
      - name: chirp
model_training:
  enabled: true
  writeback:
    schemas: [events]
    include_unlabeled: true

The tagger treats any stretch of a recording that no annotator marked as "no label here". The assumption suits a study where annotators mark every event, and the scheme's description above asks for "every call". Where annotators mark only some events, the model learns to miss the rest. The headline score is segment F1, where a predicted segment matches an annotated segment with the same label at IoU of at least 0.5, and the run also reports the matched segments' mean IoU. The trainer needs 10 annotated recordings in the train split by default.

The Training page offers both hf-audio-tagger and diarization for an audio_annotation scheme. Pick hf-audio-tagger when a label names a kind of sound and means the same thing in every recording. Pick diarization when the labels are speakers whose names are local to each recording.

The runs behind this page

The commands and configs on this page were run against Potato 2.14.0 on a laptop, with the smaller models named below. The diarization path ran end to end, from an RTTM import of 16 synthetic two-speaker recordings with 20% of the turns given the wrong speaker, through corrections on 12 recordings, a diarization run, and write-back, to the segments export. The run used a one-layer test encoder in place of WavLM. The run's score therefore says nothing about real speech. The Whisper section used 12 recordings of synthesized speech, transcribed by WhisperX with tiny.en and fine-tuned from openai/whisper-tiny. The spectrogram and tagger sections used 24 synthetic recordings of tones at 3 kHz and 6 kHz. Every config block passes python -m potato.validate_cli --strict.

Further reading

• Model training overview explains the training loop, the Training page, and held-out scores.
• Trainers lists every trainer with its schemes, install, and parameters.
• Configuration documents every model_training key, write-back, and the routes.
• Model in the loop covers reviewing pre-labels and retraining automatically.
• Audio Annotation sets up classification, event, transcription, and diarization tasks.
• Transcript Formats lists every transcript format Potato reads.
• The upstream reference documents diarization, whisper-asr, hf-audio-tagger, and spectrograms parameter by parameter.

References

• Bain, M., Huh, J., Han, T., and Zisserman, A. (2023). WhisperX: Time-Accurate Speech Transcription of Long-Form Audio. Interspeech 2023, 4489–4493. doi:10.21437/Interspeech.2023-78
• Bredin, H., and Laurent, A. (2021). End-To-End Speaker Segmentation for Overlap-Aware Resegmentation. Interspeech 2021, 3111–3115. doi:10.21437/Interspeech.2021-560
• Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., Wu, J., Zhou, L., Ren, S., Qian, Y., Qian, Y., Wu, J., Zeng, M., Yu, X., and Wei, F. (2022). WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing. IEEE Journal of Selected Topics in Signal Processing. doi:10.1109/JSTSP.2022.3188113
• Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. (2023). Robust Speech Recognition via Large-Scale Weak Supervision. Proceedings of the 40th International Conference on Machine Learning, PMLR 202, 28492–28518. proceedings.mlr.press