# Spectrogram Annotation: Time-Frequency Boxes on Audio

Source: https://www.potatoannotator.com/docs/annotation-types/spectrogram-annotation

*New in v2.12.0*

A `spectrogram_annotation` scheme shows a [spectrogram](https://en.wikipedia.org/wiki/Spectrogram) of the item's audio. Annotators draw time-frequency boxes on it and give each box a label from the scheme's list, such as species or call types. It is meant for [bioacoustics](https://en.wikipedia.org/wiki/Bioacoustics): bird, bat, frog and marine-mammal calls, where a call is described by when it starts and ends and by its frequency range.

[Audio annotation](/docs/annotation-types/audio-annotation) marks segments in time only. Use `spectrogram_annotation` when the frequency range of each event matters, or when you import boxes from Raven, Audacity or BirdNET-Analyzer ([Bioacoustics](/docs/guides/bioacoustics)).

The spectrogram is computed in the browser: the page decodes the audio with Web Audio and runs a short-time Fourier transform with one column per pixel of the drawing. No third-party library is loaded, and the server needs no audio tools.

## Configuration

```yaml
item_properties:
  id_key: id
  text_key: recording

annotation_schemes:
  - annotation_type: spectrogram_annotation
    name: calls
    description: Draw a box around every call and label it
    source_field: audio_url
    labels:
      - name: Turdus migratorius
        color: "#2563eb"
      - name: Cardinalis cardinalis
        color: "#dc2626"
      - unknown
    max_freq: 12000
```

```json
{"id": "site3_0612", "recording": "site3_0612.wav", "audio_url": "/media/site3_0612.wav"}
```

`source_field` names the item field that holds the audio URL, and files in the project's `media/` folder are served at `/media/<name>`. Any format the browser can decode works (WAV, FLAC, MP3, Ogg, M4A). For a WAV file the page reads the sample rate from the file header, so the frequency axis ends at the recording's own Nyquist frequency. For other formats, give the rate in an item field named `sample_rate`, or the browser's own rate is used.

| Key | Default | Meaning |
|---|---|---|
| `labels` | required | Label names, or `{name, color, key_value}` objects. Labels 1 to 9 get the keys `1` to `9` unless `key_value` sets another key |
| `source_field` | `audio_url` | Item field holding the audio URL |
| `min_freq` | `0` | Lowest frequency shown, in Hz |
| `max_freq` | Nyquist frequency | Highest frequency shown, in Hz |
| `fft_size` | `1024` | FFT window length in samples: 256, 512, 1024, 2048 or 4096. Larger windows resolve frequency more finely and time more coarsely |
| `colormap` | `viridis` | `viridis`, `magma` or `grayscale` |
| `height` | `280` | Height of the spectrogram in pixels |
| `min_boxes` | `0` | Boxes required before the item counts as done |
| `max_boxes` | none | Most boxes allowed on one item |
| `label_requirement` | none | `required: true` makes the scheme count as unanswered until the annotator has saved it once. Saving with no boxes counts as an answer, so use `min_boxes` to require boxes |

## Controls

| Action | Mouse or touch | Keyboard (focus in the widget) |
|---|---|---|
| Draw a box | Drag on empty spectrogram | `N` adds a box at the play position |
| Choose the label for new boxes | Click a label button | `1` to `9` |
| Relabel a box | Select it, then click a label button, or use its menu in the list | Select it, then `1` to `9` |
| Move a box | Drag it | Arrow keys |
| Resize a box | Drag its top-left or bottom-right corner | `Shift` and the arrow keys |
| Delete a box | Delete in the list | `Delete` or `Backspace` |
| Play or pause | Play button | `Space` |
| Play one box | Play in the list | `P` |
| Zoom in time | `+` and `-` buttons, up to 16× | |

The list under the spectrogram shows every box with its times and frequencies, a label menu, and Play and Delete buttons. A box imported from a detector is drawn with a dashed border and its confidence next to the label until someone changes it.

## Stored answer

```json
[{"start_s": 0.5, "end_s": 1.1, "low_hz": 2900.0, "high_hz": 3100.0, "label": "whistle"},
 {"start_s": 3.0, "end_s": 3.8, "low_hz": 5900.0, "high_hz": 6600.0, "label": "trill",
  "source": "import", "import_format": "raven", "edited": true}]
```

Times are seconds from the start of the file, and frequencies are Hz. `low_hz` and `high_hz` are `null` for a box that covers all frequencies, which is how an Audacity label without a spectral selection or a BirdNET CSV row imports. Boxes the annotator draws get `"source": "human"`. Imported boxes keep `source`, `ai_model`, `confidence` and `import_format`, and get `"edited": true` once they are moved, resized or relabelled.

Pre-annotations use the same list, under the scheme name in the `pre_annotation` field. A pre-filled box counts as the annotator's answer: if they click Next without touching a clip, the boxes are saved as their reviewed answer with no `edited` flag, so a review study can count them as accepted unchanged.

## Export

| Format | Output |
|---|---|
| `raven` | One Raven selection table per recording, with the label in an `Annotation` column (`--option label_column=Species` renames it) and `Confidence` when a box has one |
| `audacity_labels` | One Audacity label file per recording, with a spectral line under each box that has a frequency range |
| `jsonl`, `csv` and the other general formats | The JSON list as the scheme's value |

```bash
python -m potato.export -c config.yaml -f raven -o export/
python -m potato.export -c config.yaml -f audacity_labels -o export/
```

A box with no frequency range is written to Raven from 0 Hz to the Nyquist frequency, taken from the item's `sample_rate`, from the header of a WAV file, or from `max_freq`. Without any of these, `High Freq (Hz)` is left empty and the export warns.

The whole file is decoded in the browser, and the spectrogram has at most 4096 columns, so a long recording is drawn coarsely in time. Cut recordings into clips of a few minutes or less.

## Example

```bash
python potato/flask_server.py start examples/audio/bioacoustics-boxes/config.yaml -p 8000
```

## Further Reading

- [Bioacoustics](/docs/guides/bioacoustics) - Importing Raven, Audacity and BirdNET output and exporting corrected boxes
- [Audio Annotation](/docs/annotation-types/audio-annotation) - Time segments on a waveform
- [Camera Traps](/docs/guides/camera-traps) - The image side of wildlife monitoring

For implementation details, see the [source documentation](https://github.com/davidjurgens/potato/blob/master/docs/annotation-types/multimedia/spectrogram_annotation.md).