Spectrogram Annotation: Time-Frequency Boxes on Audio

Draw labelled time-frequency boxes on a spectrogram in Potato for bird, bat, frog and marine-mammal calls, with Raven and Audacity export.

New in v2.12.0

A spectrogram_annotation scheme shows a spectrogram of the item's audio. Annotators draw time-frequency boxes on it and give each box a label from the scheme's list, such as species or call types. It is meant for bioacoustics: bird, bat, frog and marine-mammal calls, where a call is described by when it starts and ends and by its frequency range.

Audio annotation marks segments in time only. Use spectrogram_annotation when the frequency range of each event matters, or when you import boxes from Raven, Audacity or BirdNET-Analyzer (Bioacoustics).

The spectrogram is computed in the browser: the page decodes the audio with Web Audio and runs a short-time Fourier transform with one column per pixel of the drawing. No third-party library is loaded, and the server needs no audio tools.

Configuration

yaml

item_properties:
  id_key: id
  text_key: recording
 
annotation_schemes:
  - annotation_type: spectrogram_annotation
    name: calls
    description: Draw a box around every call and label it
    source_field: audio_url
    labels:
      - name: Turdus migratorius
        color: "#2563eb"
      - name: Cardinalis cardinalis
        color: "#dc2626"
      - unknown
    max_freq: 12000

json

{"id": "site3_0612", "recording": "site3_0612.wav", "audio_url": "/media/site3_0612.wav"}

source_field names the item field that holds the audio URL, and files in the project's media/ folder are served at /media/<name>. Any format the browser can decode works (WAV, FLAC, MP3, Ogg, M4A). For a WAV file the page reads the sample rate from the file header, so the frequency axis ends at the recording's own Nyquist frequency. For other formats, give the rate in an item field named sample_rate, or the browser's own rate is used.

Key	Default	Meaning
`labels`	required	Label names, or `{name, color, key_value}` objects. Labels 1 to 9 get the keys `1` to `9` unless `key_value` sets another key
`source_field`	`audio_url`	Item field holding the audio URL
`min_freq`	`0`	Lowest frequency shown, in Hz
`max_freq`	Nyquist frequency	Highest frequency shown, in Hz
`fft_size`	`1024`	FFT window length in samples: 256, 512, 1024, 2048 or 4096. Larger windows resolve frequency more finely and time more coarsely
`colormap`	`viridis`	`viridis`, `magma` or `grayscale`
`height`	`280`	Height of the spectrogram in pixels
`min_boxes`	`0`	Boxes required before the item counts as done
`max_boxes`	none	Most boxes allowed on one item
`label_requirement`	none	`required: true` makes the scheme count as unanswered until the annotator has saved it once. Saving with no boxes counts as an answer, so use `min_boxes` to require boxes

Controls

Action	Mouse or touch	Keyboard (focus in the widget)
Draw a box	Drag on empty spectrogram	`N` adds a box at the play position
Choose the label for new boxes	Click a label button	`1` to `9`
Relabel a box	Select it, then click a label button, or use its menu in the list	Select it, then `1` to `9`
Move a box	Drag it	Arrow keys
Resize a box	Drag its top-left or bottom-right corner	`Shift` and the arrow keys
Delete a box	Delete in the list	`Delete` or `Backspace`
Play or pause	Play button	`Space`
Play one box	Play in the list	`P`
Zoom in time	`+` and `-` buttons, up to 16×

The list under the spectrogram shows every box with its times and frequencies, a label menu, and Play and Delete buttons. A box imported from a detector is drawn with a dashed border and its confidence next to the label until someone changes it.

Stored answer

json

[{"start_s": 0.5, "end_s": 1.1, "low_hz": 2900.0, "high_hz": 3100.0, "label": "whistle"},
 {"start_s": 3.0, "end_s": 3.8, "low_hz": 5900.0, "high_hz": 6600.0, "label": "trill",
  "source": "import", "import_format": "raven", "edited": true}]

Times are seconds from the start of the file, and frequencies are Hz. low_hz and high_hz are null for a box that covers all frequencies, which is how an Audacity label without a spectral selection or a BirdNET CSV row imports. Boxes the annotator draws get "source": "human". Imported boxes keep source, ai_model, confidence and import_format, and get "edited": true once they are moved, resized or relabelled.

Pre-annotations use the same list, under the scheme name in the pre_annotation field. A pre-filled box counts as the annotator's answer: if they click Next without touching a clip, the boxes are saved as their reviewed answer with no edited flag, so a review study can count them as accepted unchanged.

Export

Format	Output
`raven`	One Raven selection table per recording, with the label in an `Annotation` column (`--option label_column=Species` renames it) and `Confidence` when a box has one
`audacity_labels`	One Audacity label file per recording, with a spectral line under each box that has a frequency range
`jsonl`, `csv` and the other general formats	The JSON list as the scheme's value

bash

python -m potato.export -c config.yaml -f raven -o export/
python -m potato.export -c config.yaml -f audacity_labels -o export/

A box with no frequency range is written to Raven from 0 Hz to the Nyquist frequency, taken from the item's sample_rate, from the header of a WAV file, or from max_freq. Without any of these, High Freq (Hz) is left empty and the export warns.

The whole file is decoded in the browser, and the spectrogram has at most 4096 columns, so a long recording is drawn coarsely in time. Cut recordings into clips of a few minutes or less.

Example