# Train Speech and Audio Models on Your Annotations

Source: https://www.potatoannotator.com/docs/model-training/audio-models

*New in v2.14.0*

This tutorial trains speech and audio models on the answers annotators give in Potato and sends each model's output back to the annotators as pre-labels. The page is written for a study owner who already corrects transcripts, speaker turns, or animal calls in Potato and wants a model that learns from those corrections. The main path imports a speaker diarization file, has annotators correct the speaker turns, trains a diarization model on the corrected turns, and exports the corrections as RTTM. Three shorter sections then fine-tune Whisper on corrected transcript turns, train a box detector on spectrograms, and train a tagger for sound events on the waveform. For the training loop in general, read the [model training overview](/docs/model-training/overview) first, and [Train your first model](/docs/model-training/first-model) if you have never started a run.

## The four audio trainers

Each trainer learns from one kind of scheme, and the Training page offers a scheme only the trainers that fit the scheme:

| Trainer | Learns from | Model | Held-out score | Install |
|---|---|---|---|---|
| `diarization` | the speaker tier of a [`tiered_annotation`](/docs/guides/linguistic-fieldwork) scheme, or an [`audio_annotation`](/docs/annotation-types/audio-annotation) scheme whose labels are speakers | a speech encoder, WavLM by default | diarization accuracy (1 − DER) | `potato-annotation[train-audio]` |
| `whisper-asr` | the turns annotators corrected in a `speech_transcript` scheme | Whisper, `whisper-small` by default | word accuracy (1 − WER) | `potato-annotation[train-audio]` |
| `torchvision-detect`, `ultralytics-detect` | the boxes of a [`spectrogram_annotation`](/docs/annotation-types/spectrogram-annotation) scheme | a Faster R-CNN or a YOLO detector on rendered spectrograms | mAP at IoU 0.5 | `potato-annotation[train-vision]`, or `pip install ultralytics` (AGPL-3.0) |
| `hf-audio-tagger` | the labeled segments of an `audio_annotation` scheme | a speech encoder, wav2vec 2.0 by default | segment F1 at IoU 0.5 | `potato-annotation[train-audio]` |

Every audio trainer reads recordings from files on the server's disk that soundfile opens, e.g., WAV, FLAC, or OGG, and MP3 with a recent libsndfile. A path such as `/media/meet000.wav` names a file in the `media/` folder of the task directory, the folder the annotation page plays recordings from. An item whose recording is an `http(s)` URL is left out of training with a warning. Download remote recordings into `media/` before training. The [trainers reference](/docs/model-training/trainers#audio-and-speech) lists every parameter.

Install the trainers into the environment that runs Potato:

```bash
pip install "potato-annotation[train-audio]"
```

The `[train-audio]` extra installs PyTorch, transformers, and soundfile. The spectrogram detectors need `[train-vision]` instead, or `pip install "potato-annotation[train]"` for every text, image, audio, and LLM trainer at once. A trainer whose packages are missing stays on the Training page as "not installed", with the command that installs the packages.

## Importing a diarization as an RTTM file

[Speaker diarization](https://en.wikipedia.org/wiki/Speaker_diarisation) splits a recording into turns by who is speaking. pyannote and NeMo write diarizations as RTTM, a text format with one `SPEAKER` line per turn, and dscore and md-eval score RTTM files. A line names the recording's file id, the channel, the turn's onset and duration in seconds, and the speaker:

```text
SPEAKER meet000 1 0.200 1.055 <NA> <NA> SPEAKER_01 <NA> <NA>
SPEAKER meet000 1 1.539 0.755 <NA> <NA> SPEAKER_00 <NA> <NA>
```

Put the RTTM files and the recordings in one folder, with each recording named after its file id (`meet000.wav` for `meet000`), and import the folder:

```bash
potato import -f rttm -i diarization/ -o interviews \
  --speaker-map "SPEAKER_00=Interviewer,SPEAKER_01=Guest"
```

The importer makes one item per recording, copies the recording into `interviews/media/`, and writes `config.yaml`, `data/diarization.jsonl`, and a README. On the 16-recording test project behind this page, the summary line read:

```text
Imported 16 item(s), 1 question(s), 0 answer(s) from 0 annotator(s), and 16 model suggestion(s) on 16 item(s) from RTTM speaker diarization.
```

The importer takes three flags of its own:

- **`--rttm-scheme audio|tiered`** picks the scheme the turns land on. The default `audio` makes an `audio_annotation` scheme named `speakers` whose labels are the speakers. `tiered` makes a `tiered_annotation` scheme named `tiers` with one `speaker` tier. Use `tiered` for a study that transcribes turn by turn afterwards.
- **`--speaker-map`** renames speakers, and a speaker the map does not name keeps its RTTM name. Use the map only when a name means the same person in every recording. A diarizer numbers speakers within each file, and `SPEAKER_00` in one interview and `SPEAKER_00` in the next can be different people.
- **`--media-dir DIR`** names the folder holding the recordings when the recordings are not beside the RTTM files.

The importer reads only `SPEAKER` lines and counts the other line types as skipped. Zero-length turns, turns with a negative onset, turns with no speaker name (`<NA>`), and a turn listed twice are skipped as well. RTTM separates fields with spaces, and the importer reads a speaker name that contains a space only up to the space, with a warning. [Importing a Project](/docs/tools/importing-projects#rttm-speaker-diarization) covers the import in full.

## The diarization config

The generated config already shows the imported turns as pre-annotations. Add an admin key and a `model_training` block to the end of `config.yaml`. The complete file below passes `python -m potato.validate_cli config.yaml --strict`:

```yaml
port: 8000
annotation_task_name: diarization (imported)
task_dir: .
output_annotation_dir: annotation_output/
data_files:
- data/diarization.jsonl
item_properties:
  id_key: id
  text_key: text
user_config:
  allow_all_users: true
  users: []
site_dir: default
pre_annotation:
  enabled: true
  field: predictions
  allow_modification: true
annotation_schemes:
- annotation_type: audio_annotation
  name: speakers
  description: Check and correct who speaks when.
  source_field: audio_url
  mode: label
  labels:
  - name: Guest
  - name: Interviewer
admin_api_key: "change-me"
model_training:
  enabled: true
```

The importer sorts the speaker labels alphabetically, and `Guest` comes before `Interviewer`. The model's write-back goes to the item field `predictions` by default, the same field the import's `pre_annotation` block reads. Potato refuses to start when an enabled `pre_annotation` block names a different field from `writeback.field`, because no training pre-label would then show. Leave `writeback` out of the config until the first run has a held-out score worth showing to annotators.

## Correcting the speaker turns

Start the server from the project folder:

```bash
cd interviews
potato start config.yaml -p 8000
```

Each recording opens with the diarizer's turns drawn on the waveform. Annotators move a turn's edges, change a turn's speaker, delete a turn of noise, and draw any turn the diarizer missed. The pre-annotations are stored only once an annotator saves the item. An item nobody opened has no answer and does not train. [Speaker Diarization Annotation](/blog/speaker-diarization-annotation) covers the waveform controls and keyboard shortcuts.

The `diarization` trainer needs at least 4 annotated recordings in the train split, and the default split puts about 70% of the labeled recordings in train. Correct at least 8 recordings before the first run, and more for a held-out score you can trust.

## Training the diarization model

Open `http://localhost:8000/admin`, enter the admin key, and go to `/admin/training`. Pick `diarization` as the trainer and `speakers` as the scheme. The Training page also offers `hf-audio-tagger` for the same scheme. `hf-audio-tagger` suits labels that name kinds of sound rather than speakers. For two-person interviews, set the Parameters field to:

```json
{"max_speakers": 2}
```

Click **Train now**. The same run can be started from a script with the admin key:

```bash
curl -X POST -H "X-API-Key: change-me" -H "Content-Type: application/json" \
  http://localhost:8000/admin/training/api/runs \
  -d '{"trainer": "diarization", "schemas": ["speakers"], "params": {"max_speakers": 2}}'
```

`diarization` follows the recipe of pyannote's segmentation model ([Bredin and Laurent, 2021](https://doi.org/10.21437/Interspeech.2021-560)) without pyannote or pyannote's gated weights, and fine-tunes [WavLM](https://arxiv.org/abs/2110.13900) (`microsoft/wavlm-base-plus`) with a frame classifier for up to `max_speakers` speakers. The model sees `chunk_s` seconds of audio at a time, 10 by default. Each chunk's loss is computed under the best matching of the model's speaker outputs to the chunk's annotated speakers. The matching matters because a speaker name refers to different people in different recordings, and a classifier over names would have no consistent target. The parameters below shape the predicted turns:

| Parameter | Default | Effect |
|---|---|---|
| `max_speakers` | 3 | Most speakers active within one chunk, up to 6 |
| `chunk_s` | 10 | Seconds the model sees at once |
| `threshold` | 0.5 | Frame probability at which a speaker counts as speaking |
| `min_gap_s`, `min_duration_s` | 0.2, 0.2 | A shorter gap inside a turn is closed, and a shorter turn is dropped |
| `model_name` | `microsoft/wavlm-base-plus` | A WavLM or wav2vec 2.0 checkpoint, or a local path |

[Remote training](/docs/model-training/remote-training) sends a run to a GPU server over ssh or to a SLURM cluster.

## Reading the diarization score

The run page and `GET /admin/training/api/status` report the run's score on held-out recordings. For `diarization`, the headline is `diarization_accuracy`, defined as 1 minus the diarization error rate (DER) and floored at 0. DER adds up missed speech, false alarm, and speaker confusion and divides the sum by the annotated speech. Potato measures DER on a 10 ms grid with no collar, under the best one-to-one mapping of predicted to annotated speakers. The run reports the three error types separately as well. Read the three numbers to see whether the model misses speech, invents speech, or mixes up speakers. DER ignores whether a turn is called `Interviewer` or `SPEAKER_01` and counts whether the speaker changes at the right times.

A metric name includes the split, as in `val_der` or `test_diarization_accuracy`. A recording with annotated turns and no prediction counts as wrong. A model therefore cannot raise the score by skipping hard recordings.

## Publishing the predicted turns

Once the held-out score supports showing the model's turns to annotators, add the scheme to `writeback` and restart the server:

```yaml
model_training:
  enabled: true
  writeback:
    schemas: [speakers]
    include_unlabeled: true
```

`include_unlabeled: true` also predicts on the recordings nobody has corrected yet. Without the setting, a run predicts only on the held-out val and test recordings. Train again after the restart. The write-back report in `GET /admin/training/api/status` gives the count. On the test project, the report read "Published 7 prediction(s) as prelabels." for 3 val recordings and 4 uncorrected recordings, and the recordings in the train split kept the imported RTTM turns.

The model names speakers in order of first speech within each recording, using the scheme's labels first and then `SPEAKER_00`, `SPEAKER_01`, and so on. The first speaker heard in each recording is therefore `Guest` in this project, whether or not that person is the guest. The model has no voice embeddings to recognize a person by. A speaker who is silent through the overlap between two chunks and speaks again later gets a new name. Tell annotators to check every predicted turn's speaker name, and not only the turn's edges. [Model in the loop](/docs/model-training/model-in-the-loop) covers reviewing pre-labels least confident first and retraining as corrections arrive.

## Exporting the corrected turns as RTTM

The `segments` export writes the corrected turns back out as RTTM, one file per annotator:

```bash
python -m potato.export --config config.yaml --format segments --output ./export/
```

The export writes `export/segments.jsonl`, one row per turn, and `export/rttm/<annotator>.rttm`. The file id in each RTTM line is the recording's name without its extension. pyannote.metrics, dscore, or md-eval can then score the diarizer's original output against the corrected file unchanged. On the test project, the first lines of `export/rttm/ana.rttm` read:

```text
SPEAKER meet000 1 0.200 1.055 <NA> <NA> Guest <NA> <NA>
SPEAKER meet000 1 1.539 0.755 <NA> <NA> Interviewer <NA> <NA>
```

## Diarization on a speaker tier

Import with `--rttm-scheme tiered` when annotators will add other tiers, such as a transcription tier, to the same timeline:

```bash
potato import -f rttm -i diarization/ -o interviews --rttm-scheme tiered
```

The import writes a `tiered_annotation` scheme named `tiers` with one `speaker` tier. Add `admin_api_key` and `model_training` as above, with `schemas: [tiers]` under `writeback`, and the config passes `validate_cli --strict`. `diarization` trains on the speaker tier and writes predicted turns back onto the same tier. The trainer picks the tier named `speaker` or `speakers`. Failing that, the trainer picks the one tier whose name contains "speaker", and a tier named `non-speaker` does not count. The last fallback is the only tier with labels. When two tiers qualify, or several tiers have labels and none is named for speakers, the run leaves the recordings out with a warning that names the tiers.

## Fine-tuning Whisper on corrected transcripts

`whisper-asr` fine-tunes [Whisper](https://proceedings.mlr.press/v202/radford23a.html) on the transcript turns annotators corrected, then suggests corrections on the turns nobody has checked. The section starts from a folder of recordings with no transcript.

### Transcribing the recordings

Put the recordings in the project's `media/` folder and transcribe the recordings on the server's machine:

```bash
pip install "potato-annotation[transcribe]"
potato transcripts media --transcribe --asr-model small.en --asr-word-timestamps \
  --media-url-prefix /media -o data/interviews.json
```

`potato transcripts --transcribe` runs [WhisperX](https://doi.org/10.21437/Interspeech.2023-78) when WhisperX is installed and works on the machine, and faster-whisper otherwise. The `[transcribe]` extra installs faster-whisper, and WhisperX is a separate install (`pip install whisperx`) that brings its own PyTorch. Potato falls back to faster-whisper when WhisperX is missing, does not import, cannot run on the device, or cannot load the model, and the summary names the engine for each file. `--asr-engine whisperx` or `--asr-engine faster-whisper` runs one engine with no fallback. `--media-url-prefix /media` writes each recording's path as `/media/<file>`, the address the annotation page plays the recording from.

`--asr-word-timestamps` times each word by forced alignment and lets WhisperX split a segment at a sentence end. On the test recordings, WhisperX returned one turn per 10-second recording without the flag and three turns per recording with the flag. Shorter turns give `whisper-asr` more examples, and Whisper's input window is 30 seconds. [Transcript Formats](/docs/annotation-types/transcript-formats#transcribing-locally) and [How to Annotate Whisper Transcripts](/docs/guides/annotating-whisper-transcripts) cover the transcription options, speaker labels, and existing Whisper output.

### The transcript config

A `speech_transcript` scheme shows each turn of the transcript with its time span and a correction box. The complete config below passes `validate_cli --strict`:

```yaml
annotation_task_name: "Interview transcript correction"
task_dir: .
output_annotation_dir: annotation_output/
data_files:
  - data/interviews.json
item_properties:
  id_key: id
  text_key: conversation
admin_api_key: "change-me"
annotation_schemes:
  - annotation_type: speech_transcript
    name: transcript
    description: "Play each turn and type the right text wherever the transcript is wrong."
    segments_key: conversation
    error_types: [asr_error]
model_training:
  enabled: true
  trainer_params:
    whisper-asr:
      model_name: openai/whisper-small
      language: en
```

`segments_key: conversation` names the field `potato transcripts` writes the transcript under. `error_types: [asr_error]` leaves one error flag, because the study covers transcription errors only. `language: en` fixes the language for training and prediction. Left unset, a multilingual Whisper detects the language once from up to 16 training turns, and the run log warns when fewer than three in four of the sampled turns agree.

### Corrected turns as training examples

Each corrected turn is one example, made of the turn's audio cut from the recording by the turn's start and end, together with the corrected text. A turn nobody corrected is not an example, because the scheme saves only the turns an annotator touched, and an untouched turn may be right or may be unread. A flagged turn with no correction is not an example either. Turns longer than 30 seconds are left out with a warning. The trainer refuses to run with fewer than 10 corrected turns in the train split.

Correct every wrong turn on the recordings you open, because the held-out score is computed over the corrected turns. On the 12 test recordings, WhisperX with `tiny.en` got 11 of 36 turns wrong, with errors such as "Epsilanti" for Ypsilanti, "Cary Town" for Kerrytown, and a dropped "I" at the start of a turn.

### Training and comparing against the transcript

Pick `whisper-asr` and the `transcript` scheme on the Training page and click **Train now**. The headline score is `word_accuracy`, which is 1 minus the word error rate (WER) over the corrected turns. The run also reports `wer_asr`, the original transcript's WER on the same turns, and a turn the model left alone is scored as the transcript's text. Compare the two numbers before publishing anything. A fine-tuned model is worth showing to annotators only when the model's WER is below `wer_asr`.

The test run behind this page shows why the comparison matters. With 8 corrected turns in train, `min_instances` lowered to 4, and `openai/whisper-tiny` trained for 3 epochs, the run scored a test WER of 0.31 against 0.19 for the original transcript. A model at that level would replace correct text with wrong text. Collect more corrections, or start from a larger checkpoint, and train again before adding `writeback`.

### Publishing suggested corrections

Once the model beats the transcript, add the scheme to `writeback`, restart, and train again:

```yaml
model_training:
  enabled: true
  trainer_params:
    whisper-asr:
      model_name: openai/whisper-small
      language: en
  writeback:
    schemas: [transcript]
    include_unlabeled: true
```

Prediction transcribes every turn of every held-out and unlabeled recording. A turn whose new text differs from the transcript becomes a suggested correction, flagged `asr_error`, with the model's mean token probability as the suggestion's confidence. The suggestions fill the correction boxes, and an annotator keeps, edits, or clears each suggestion.

### Transcribing new recordings with the fine-tuned model

By default the run also writes `ct2/`, an int8 CTranslate2 copy of the model, among the run's artifacts. The copy needs the `ctranslate2` package, which the `[transcribe]` extra installs with faster-whisper. Pass the copy's path to `potato transcripts` to transcribe the next batch of recordings with the fine-tuned model:

```bash
potato transcripts media/batch2 --transcribe \
  --asr-model annotation_output/training/runs/<run_id>/artifacts/ct2 \
  --media-url-prefix /media/batch2 -o data/batch2.json
```

WhisperX and faster-whisper both load the copy. The run page shows each run's id, and [Configuration](/docs/model-training/configuration#where-runs-are-stored) describes the run folders.

## Detecting calls on spectrograms

A `spectrogram_annotation` scheme has annotators draw time-frequency boxes around calls. `torchvision-detect` and `ultralytics-detect` train on those boxes by rendering each recording as a spectrogram image the way the widget draws the spectrogram. The rendering puts frequency on a linear axis from the scheme's `min_freq` to `max_freq`, or half the sample rate when `max_freq` is unset, at 256 pixels high and 100 pixels per second. Each box becomes an image box. Time and frequency map linearly to pixels. A box's IoU on the image therefore equals the box's IoU in seconds by hertz. A box with no frequency range, as an Audacity label or a BirdNET row gives, covers the whole band. A recording of 41 seconds or more renders at least 4,096 pixels wide and is cut into windows and stitched back.

Boxes usually come from an import. `potato import` reads Raven selection tables, Audacity label tracks, and BirdNET-Analyzer output into a `spectrogram_annotation` project, as [Bioacoustics in Potato](/docs/guides/bioacoustics) and [Verify BirdNET Detections on a Spectrogram](/blog/verify-birdnet-detections-spectrogram) describe. The trainers learn from the boxes annotators drew or corrected, and the detectors read the spectrogram pixels rather than BirdNET or Perch embeddings.

The config below passes `validate_cli --strict` and sets the detector's `score_threshold` to 0.5:

```yaml
annotation_task_name: "Dawn chorus calls"
task_dir: .
output_annotation_dir: annotation_output/
data_files:
  - data/recordings.jsonl
item_properties:
  id_key: id
  text_key: id
admin_api_key: "change-me"
annotation_schemes:
  - annotation_type: spectrogram_annotation
    name: calls
    description: "Draw a box around each call and pick its type."
    source_field: audio_url
    max_freq: 8000
    labels:
      - name: whistle
      - name: chirp
model_training:
  enabled: true
  trainer_params:
    torchvision-detect:
      score_threshold: 0.5
  writeback:
    schemas: [calls]
    include_unlabeled: true
```

The spectrum of a 16 kHz recording ends at 8 kHz, and `max_freq: 8000` shows the whole band. The default `score_threshold` of 0.25 keeps more of the detector's guesses. On the 24-recording test project, `torchvision-detect` at the default threshold predicted 34 boxes for 10 annotated calls on the test recordings, and each extra box is one more box for an annotator to delete.

Each data row names its recording, as in `{"id": "site_00", "audio_url": "/media/site_00.wav"}`. A detector trains only on recordings with at least one box, because a recording with no boxes looks the same as a recording nobody checked. The detectors need 20 annotated recordings in the train split by default (`min_instances`).

`torchvision-detect` fine-tunes a Faster R-CNN from torchvision's COCO weights, and `ultralytics-detect` trains a YOLO detector. The YOLO detector is faster to train and to run. Ultralytics is licensed AGPL-3.0, and a model trained and distributed with Ultralytics is subject to the AGPL's terms. The Training page asks you to confirm the license before an `ultralytics-detect` run, and an API run needs `"accept_licence": true`. Both detectors are scored by mAP at IoU 0.5 on the rendered spectrogram, and both write predicted boxes back in `start_s`, `end_s`, `low_hz`, and `high_hz`, marked as model output, to prefill the widget. [Vision models](/docs/model-training/vision-models) covers the detectors' parameters and tiling.

## Tagging sound events with `hf-audio-tagger`

`hf-audio-tagger` learns the labeled time segments of an `audio_annotation` scheme, such as speech, music, calls, or coughs. The trainer fine-tunes a speech encoder, `facebook/wav2vec2-base` by default, to say which labels are active in each frame of about 20 ms. Each label is scored separately. Overlapping segments with different labels are therefore learned as annotated. Predicted segments go back as editable regions on the waveform. The config below passes `validate_cli --strict`:

```yaml
annotation_task_name: "Dawn chorus events"
task_dir: .
output_annotation_dir: annotation_output/
data_files:
  - data/recordings.jsonl
item_properties:
  id_key: id
  text_key: id
admin_api_key: "change-me"
annotation_schemes:
  - annotation_type: audio_annotation
    name: events
    description: "Mark every call on the waveform and pick its type."
    source_field: audio_url
    mode: label
    labels:
      - name: whistle
      - name: chirp
model_training:
  enabled: true
  writeback:
    schemas: [events]
    include_unlabeled: true
```

The tagger treats any stretch of a recording that no annotator marked as "no label here". The assumption suits a study where annotators mark every event, and the scheme's description above asks for "every call". Where annotators mark only some events, the model learns to miss the rest. The headline score is segment F1, where a predicted segment matches an annotated segment with the same label at IoU of at least 0.5, and the run also reports the matched segments' mean IoU. The trainer needs 10 annotated recordings in the train split by default.

| Parameter | Default | Effect |
|---|---|---|
| `window_s` | 10 | Seconds of audio per training and prediction window |
| `threshold` | 0.5 | Frame probability at which a label is active |
| `min_gap_s`, `min_duration_s` | 0.1, 0.1 | A shorter gap inside a segment is closed, and a shorter segment is dropped |
| `freeze_feature_encoder` | `true` | Keeps the encoder's convolutional feature layers fixed |

The Training page offers both `hf-audio-tagger` and `diarization` for an `audio_annotation` scheme. Pick `hf-audio-tagger` when a label names a kind of sound and means the same thing in every recording. Pick `diarization` when the labels are speakers whose names are local to each recording.

## The runs behind this page

The commands and configs on this page were run against Potato 2.14.0 on a laptop, with the smaller models named below. The diarization path ran end to end, from an RTTM import of 16 synthetic two-speaker recordings with 20% of the turns given the wrong speaker, through corrections on 12 recordings, a `diarization` run, and write-back, to the `segments` export. The run used a one-layer test encoder in place of WavLM. The run's score therefore says nothing about real speech. The Whisper section used 12 recordings of synthesized speech, transcribed by WhisperX with `tiny.en` and fine-tuned from `openai/whisper-tiny`. The spectrogram and tagger sections used 24 synthetic recordings of tones at 3 kHz and 6 kHz. Every config block passes `python -m potato.validate_cli --strict`.

## Further reading

- [Model training overview](/docs/model-training/overview) explains the training loop, the Training page, and held-out scores.
- [Trainers](/docs/model-training/trainers#audio-and-speech) lists every trainer with its schemes, install, and parameters.
- [Configuration](/docs/model-training/configuration) documents every `model_training` key, write-back, and the routes.
- [Model in the loop](/docs/model-training/model-in-the-loop) covers reviewing pre-labels and retraining automatically.
- [Audio Annotation](/docs/guides/audio-annotation) sets up classification, event, transcription, and diarization tasks.
- [Transcript Formats](/docs/annotation-types/transcript-formats) lists every transcript format Potato reads.
- The upstream reference documents [`diarization`](https://github.com/davidjurgens/potato/blob/master/docs/ai-intelligence/model_training.md#diarization), [`whisper-asr`](https://github.com/davidjurgens/potato/blob/master/docs/ai-intelligence/model_training.md#whisper-asr), [`hf-audio-tagger`](https://github.com/davidjurgens/potato/blob/master/docs/ai-intelligence/model_training.md#hf-audio-tagger), and [spectrograms](https://github.com/davidjurgens/potato/blob/master/docs/ai-intelligence/model_training.md#spectrograms) parameter by parameter.

## References

- Bain, M., Huh, J., Han, T., and Zisserman, A. (2023). WhisperX: Time-Accurate Speech Transcription of Long-Form Audio. *Interspeech 2023*, 4489–4493. [doi:10.21437/Interspeech.2023-78](https://doi.org/10.21437/Interspeech.2023-78)
- Bredin, H., and Laurent, A. (2021). End-To-End Speaker Segmentation for Overlap-Aware Resegmentation. *Interspeech 2021*, 3111–3115. [doi:10.21437/Interspeech.2021-560](https://doi.org/10.21437/Interspeech.2021-560)
- Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., Wu, J., Zhou, L., Ren, S., Qian, Y., Qian, Y., Wu, J., Zeng, M., Yu, X., and Wei, F. (2022). WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing. *IEEE Journal of Selected Topics in Signal Processing*. [doi:10.1109/JSTSP.2022.3188113](https://doi.org/10.1109/JSTSP.2022.3188113)
- Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. (2023). Robust Speech Recognition via Large-Scale Weak Supervision. *Proceedings of the 40th International Conference on Machine Learning*, PMLR 202, 28492–28518. [proceedings.mlr.press](https://proceedings.mlr.press/v202/radford23a.html)