Verify BirdNET Detections on a Spectrogram

Bird-survey teams who run BirdNET-Analyzer on ARU or AudioMoth recordings can follow this tutorial to have a person check every BirdNET detection in Potato 2.13.1, compute BirdNET's precision for each species from that check, and export the checked boxes as Raven selection tables. BirdNET reports a confidence score for each species in each 3-second window of a recording. BirdNET's documentation asks users to check detections by hand before choosing a score cut-off, and points to Wood and Kahl (2024) for the method. The worked example uses 20 minutes of a public dawn-chorus recording with expert boxes for every call. The precision this page reports can therefore be checked against an answer key. Bioacoustics documents every import and export option, and this page follows one review from BirdNET's tables to the exported results.

The recordings and the expert boxes

The example data come from Powdermill Nature Reserve in Rector, Pennsylvania, where AudioMoth recorders captured the dawn chorus between April and July 2018 (Chronister et al., 2021). The set has 385 minutes of audio in five-minute WAV segments, with 16,052 boxes for 48 species. One annotator drew the boxes in Raven Pro, a second identified the species again, and a third settled the disagreements. The boxes for each segment are in a Raven selection table named after the segment, such as Recording_2_Segment_01.Table.1.selections.txt for Recording_2_Segment_01.wav, and the Species column gives each species as a four-letter banding code. The data are released under CC0 1.0 on Dryad, and the authors ask that you cite the paper.

Dryad serves the files through a browser download. Zenodo record 4656848 lists the same four files under the Dryad DOI and downloads from the command line. The Zenodo WAV files are 44.1 kHz, and the dataset's README describes 32 kHz recordings. BirdNET's scores on the Dryad copy may differ slightly from the scores below. This tutorial uses the Zenodo copies of wav_Files.zip (1.4 GB) and annotation_Files.zip, and takes the first four segments of Recording 2, made on May 8, 2018 from 5:32 AM. Those 20 minutes have 891 expert boxes for 13 species.

Running BirdNET-Analyzer

BirdNET-Analyzer installs from PyPI and needs Python 3.11 or later. The first run downloads the model, about 224 MB. Put the four WAV files in a folder named recordings/ and run BirdNET on the folder:

pip install birdnet-analyzer==2.4.0
python -m birdnet_analyzer.analyze recordings/ --lat 40.16 --lon -79.27 --week 18 --min_conf 0.1

--lat, --lon, and --week restrict BirdNET to the 150 species BirdNET's range model expects at Powdermill in the second week of May. BirdNET splits the year into 48 weeks, four per month. --min_conf 0.1 keeps detections down to a score of 0.1, below BirdNET's default of 0.25. The import step below sets the cut-off for the review, and the low-scoring rows stay in BirdNET's tables for trying a different cut-off later without running BirdNET again. Without -o, BirdNET writes one Recording_2_Segment_01.BirdNET.selection.table.txt beside each recording. One window from the fourth clip shows the format:

Selection	View	Channel	Begin Time (s)	End Time (s)	Low Freq (Hz)	High Freq (Hz)	Common Name	Species Code	Confidence	Begin Path	File Offset (s)
36	Spectrogram 1	1	39.0	42.0	0	15000	Wood Thrush	woothr	0.9909	recordings/Recording_2_Segment_04.wav	39.0
37	Spectrogram 1	1	39.0	42.0	0	15000	American Redstart	amered	0.9909	recordings/Recording_2_Segment_04.wav	39.0
38	Spectrogram 1	1	39.0	42.0	0	15000	Black-capped Chickadee	bkcchi	0.3593	recordings/Recording_2_Segment_04.wav	39.0

Every row is a 3-second window from 0 Hz to 15 kHz, the top of BirdNET's default band-pass filter. BirdNET does not measure where a call sits in frequency, and one window can name several species. The four tables have 1,097 rows.

Importing the detections into Potato

Install Potato and import the folder with a cut-off of 0.5:

pip install potato-annotation==2.13.1
potato import -i recordings/ -f birdnet --min-confidence 0.5 -o birdnet-review/

Imported 4 item(s), 1 question(s), 0 answer(s) from 0 annotator(s), and 621 model suggestion(s) on 4 item(s) from BirdNET-Analyzer output.
  left out: 476 detections below confidence 0.5
  warning: 476 detection(s) below --min-confidence 0.5 were left out
  wrote birdnet-review/data/recordings.jsonl
  wrote birdnet-review/media/Recording_2_Segment_01.wav
  ...
  wrote birdnet-review/config.yaml
  wrote birdnet-review/README.md
 
Run it:
  potato start birdnet-review/config.yaml -p 8000

Each recording became one item, and the recording's WAV file was copied into media/. The 621 detections at 0.5 or more became boxes in the item's predictions field. Each box keeps BirdNET's score, "source": "ai", and BirdNET's species code under attributes, and the box's label is BirdNET's Common Name.

A recording in which BirdNET finds nothing above --min_conf gets one placeholder row labeled nocall with a confidence of 1.0. Potato 2.13.1 leaves that row out of the import. Potato 2.12.1 imported the row as a BirdNET box and added nocall to the label list, so on 2.12.1 delete those boxes in the review or remove the row before the import. None of the four clips in this example has a nocall row.

The review config

The import writes a working config with one spectrogram_annotation scheme whose labels are the ten species BirdNET reported. The config below makes three changes to the import's config. The first is a task name, and the second is max_freq: 15000, the top of BirdNET's boxes. The third adds the three species that the Powdermill experts found and BirdNET did not report. A reviewer can then label the missed calls. In your own survey, take those extra species from your site list. The config is complete and passes python -m potato.validate_cli config.yaml --strict on 2.12.1:

annotation_task_name: BirdNET review, Powdermill May 8
task_dir: .
output_annotation_dir: annotation_output/
data_files:
  - data/recordings.jsonl
item_properties:
  id_key: id
  text_key: recording
user_config:
  allow_all_users: true
  users: []
instance_display:
  fields:
    - key: recording
      type: text
      label: Recording
pre_annotation:
  enabled: true
  field: predictions
  allow_modification: true
 
annotation_schemes:
  - annotation_type: spectrogram_annotation
    name: calls
    description: Check BirdNET's boxes, relabel or delete the wrong ones, and draw one box for each species BirdNET missed.
    source_field: audio_url
    max_freq: 15000
    labels:
      # The ten species BirdNET reported at 0.5 or above
      - American Redstart
      - Black-and-white Warbler
      - Black-capped Chickadee
      - Black-throated Green Warbler
      - Eastern Towhee
      - Kentucky Warbler
      - Louisiana Waterthrush
      - Ovenbird
      - Scarlet Tanager
      - Wood Thrush
      # Species on the site list that BirdNET did not report in these clips
      - Brown-headed Cowbird
      - Hooded Warbler
      - Tufted Titmouse

The pre_annotation block draws BirdNET's boxes on each clip when the clip opens. The first nine labels get the keys 1 to 9, and the rest are chosen with the mouse.

Reviewing the boxes

Start the server from the project folder, and have each reviewer register with a username and password:

cd birdnet-review
potato start config.yaml -p 8000

Decide what counts as a correct detection before anyone starts. BirdNET labels a 3-second window. The rule in this tutorial is therefore that a box is correct when the named species calls inside the window. A reviewer applies the rule to each box in one of three ways:

• Accept a box by leaving the box unchanged. BirdNET's boxes have a dashed border and show the score next to the label until a reviewer changes the box.
• Relabel a box whose window contains a different species by choosing that species in the box's label menu. The box keeps BirdNET's score and species code and gets "edited": true.
• Delete a box whose window has no call of the named species.

Then draw one box around the clearest call of each species that the reviewer hears and BirdNET did not report. Clicking Next saves the clip, and a clip left untouched is saved with BirdNET's boxes as the reviewer's answer.

Work from the list under the spectrogram rather than from the boxes on the spectrogram. The first clip opens with 103 boxes, and BirdNET's full-height 3-second boxes cover the entire spectrogram of a 5-minute clip. Each row of the list has the label menu, a Play button for the box's 3 seconds, and a Delete button. These clips have no empty spectrogram left. In Potato 2.13.1, a drag over a BirdNET box that is not selected draws a new box, and a click selects a box so that a drag then moves the box. To draw a missed call, choose the species in the label bar and drag across the call. In Potato 2.12.1, a drag on a BirdNET box moved the box instead. On that version, press Space to play the clip, press Space again to pause at the call, and press N to add a box at the paused position. The arrow keys move a selected box, and Shift with the arrow keys resizes the box. The + button zooms in time up to 16×.

The test run behind this page

The numbers on this page come from a scripted reviewer on Potato 2.12.1 that used the Powdermill expert boxes as the answer key. The import and the Raven export were run again on Potato 2.13.1, and their output on this page is from 2.13.1. The script saved each clip through the same requests the annotation page sends. A BirdNET box was accepted when an expert box of the same species overlapped the box's window by at least 0.1 seconds. A box was relabeled when the window contained an expert box of another species on the label list that BirdNET had not reported in the same window, and deleted otherwise. For each species the experts found and no remaining box named, the script drew the expert's longest box. Of the 621 BirdNET boxes, 601 were accepted, 12 relabeled, and 8 deleted, and 7 boxes were drawn for missed species. A person listening to each window can judge a call at the edge of a window differently from the 0.1-second rule.

Exporting Raven selection tables

The Raven export writes one selection table per recording:

python -m potato.export -c config.yaml -f raven -o export/

The command wrote export/Recording_2_Segment_01.selections.txt through Recording_2_Segment_04.selections.txt, with 620 boxes in all. Each file is named after the recording, and Raven Pro or Raven Lite opens the table beside the audio. The label goes in an Annotation column, and --option label_column=Species renames the column. With several reviewers, each file name adds the reviewer's name, as in Recording_2_Segment_04.alice.selections.txt. -f audacity_labels writes Audacity label tracks instead.

From Potato 2.13.1, the Raven table records how each box came out of the review. Rows 24 and 25 of the fourth clip's table cover the window at 39 to 42 seconds shown earlier:

Selection	View	Channel	Begin Time (s)	End Time (s)	Low Freq (Hz)	High Freq (Hz)	Annotation	Confidence	Box Source
24	Spectrogram 1	1	39	42	0	15000	Wood Thrush	0.9909	ai
25	Spectrogram 1	1	39	42	0	15000	Black-capped Chickadee	1	ai (edited)

BirdNET gave American Redstart 0.9909 and Black-capped Chickadee 0.3593 in that window, and the reviewer relabeled the Redstart box as a chickadee. The relabeled box gets confidence 1 and the Box Source value ai (edited), a box BirdNET reported and the reviewer kept keeps BirdNET's score as ai, and a box the reviewer drew is human with confidence 1. Across the four tables, the 620 boxes split into 601 ai, 12 ai (edited), and 7 human. Potato 2.12.1 had no Box Source column and kept the Redstart's score on the relabeled row, so on that version take edits from the JSONL export below.

Per-species precision from the review

The JSONL export keeps every box as the reviewer saved the box, with source, edited, and BirdNET's species code:

python -m potato.export -c config.yaml -f jsonl -o export/

The script below looks up BirdNET's original label for each reviewed box by the box's species code, and counts each BirdNET box as kept, relabeled, or deleted. Precision is the share of BirdNET's boxes for a species that the reviewer kept under the same species. The script assumes one reviewer per clip. Run the script in birdnet-review/:

import collections
import json
 
# BirdNET's boxes as imported, and BirdNET's name for each species code
imported = collections.Counter()
name_of = {}
for line in open("data/recordings.jsonl"):
    item = json.loads(line)
    for box in item["predictions"]["calls"]:
        imported[box["label"]] += 1
        name_of[box["attributes"]["Species Code"]] = box["label"]
 
# The reviewed boxes, one record per clip and reviewer
kept = collections.Counter()
relabeled = collections.Counter()
drawn = collections.Counter()
for line in open("export/annotations.jsonl"):
    record = json.loads(line)
    for box in json.loads(record["labels"]["calls"]["_data"]):
        if box.get("source") == "human":
            drawn[box["label"]] += 1
            continue
        birdnet_label = name_of[box["attributes"]["Species Code"]]
        if box["label"] == birdnet_label:
            kept[birdnet_label] += 1
        else:
            relabeled[birdnet_label] += 1
 
print(f"{'species':30} {'BirdNET':>7} {'kept':>5} {'relab':>5} {'del':>4} {'prec':>5} {'drawn':>5}")
for species in sorted(set(imported) | set(drawn), key=lambda s: -imported[s]):
    n = imported[species]
    deleted = n - kept[species] - relabeled[species]
    precision = f"{kept[species] / n:.2f}" if n else "-"
    print(f"{species:30} {n:7} {kept[species]:5} {relabeled[species]:5} {deleted:4} {precision:>5} {drawn[species]:5}")

species                        BirdNET  kept relab  del  prec drawn
Wood Thrush                        389   386     2    1  0.99     0
American Redstart                   48    44     4    0  0.92     1
Louisiana Waterthrush               47    39     2    6  0.83     0
Black-throated Green Warbler        44    44     0    0  1.00     0
Scarlet Tanager                     34    31     2    1  0.91     0
Black-capped Chickadee              34    32     2    0  0.94     1
Kentucky Warbler                    12    12     0    0  1.00     1
Eastern Towhee                       5     5     0    0  1.00     0
Ovenbird                             4     4     0    0  1.00     1
Black-and-white Warbler              4     4     0    0  1.00     0
Tufted Titmouse                      0     0     0    0     -     1
Brown-headed Cowbird                 0     0     0    0     -     1
Hooded Warbler                       0     0     0    0     -     1

At a cut-off of 0.5, the reviewer kept 601 of BirdNET's 621 boxes under the same species, a precision of 0.97. Louisiana Waterthrush had the lowest precision at 0.83, with 6 of the 47 Louisiana Waterthrush boxes deleted. Wood Thrush made up 389 of the 621 boxes. The overall precision of 0.97 is therefore mostly a measure of Wood Thrush, and the per-species rows are the ones to read. One wrong box among Ovenbird's four would have moved Ovenbird's precision from 1.00 to 0.75, and species with a dozen boxes or fewer need more clips before the precision of those species means much.

Precision at other cut-offs and missed calls

Powdermill has an expert box for every identifiable call. The overlap rule from the test run can therefore score BirdNET's full tables at any cut-off without a second review. The table compares BirdNET's rows at each score with the expert boxes for the four clips:

A review of BirdNET's boxes measures precision and cannot measure the calls BirdNET never reported. At 0.5, 687 of the 891 expert boxes fall in a correct BirdNET window of the same species. Eastern Towhee had 5 of 45 expert calls in a correct window, Ovenbird 4 of 27, and Black-capped Chickadee 34 of 99. Wood Thrush had 492 of 495. The one box per missed species in the review gives species presence per clip, and counting missed calls needs a complete reference like Powdermill's or a person drawing every call. These rates come from 20 minutes of one May morning at one site, and a different site, season, or recorder needs a separate check.

Other tools for checking BirdNET output

BirdNET-Analyzer's Review tab implements the Wood and Kahl workflow. The Segments tab cuts out the audio of each detection, and in the Review tab a reviewer marks each segment of one species positive or negative. The tab then fits a logistic regression of the reviewer's answers on BirdNET's score and calculates a cut-off. For one person choosing a cut-off per species, the Review tab is the shortest route. Raven Pro runs BirdNET through Raven's Learning Detector and has a Selection Review interface for large numbers of detector selections, and the Cornell Lab store lists Raven Pro 1.6 at $50 to $100. Cornell's Raven Workbench is free and in beta, and Cornell plans to replace Raven Pro with the Workbench apps. Kaleidoscope Pro from Wildlife Acoustics lists BirdNET classifiers among Kaleidoscope's features. Arbimon is a free hosted platform from Rainforest Connection, and Arbimon's help center documents pattern matching and validating the presence of species in uploaded recordings.

Potato fits when several reviewers share the check in a browser, when recordings stay on your own server, and when the check includes relabeling and drawing missed calls rather than a yes or no per segment. Camera trap and bioacoustics tools compared sets these tools beside each other.

Limits of Potato for BirdNET review

• No detector. Potato reads BirdNET's tables and does not run BirdNET. BirdNET's models are licensed CC BY-NC-SA 4.0, and BirdNET's README treats all educational and research use as non-commercial.
• No agreement on spectrogram boxes. Potato 2.13.1 computes no agreement between reviewers on spectrogram_annotation boxes. For a second reviewer on every clip, set num_annotators_per_item: 2 and add a clip-level question, as in the two-reviewer BirdNET config.
• No sound measurements. Raven Pro and Raven Workbench measure frequency, duration, and energy, and Potato stores box edges only.

Where to go next

• Bioacoustics, for every import and export option for Raven, Audacity, and BirdNET files
• Spectrogram Annotation, for the scheme's options, controls, and stored answer
• Ecology in Potato, for the camera-trap and bioacoustics workflows in one place
• Camera trap and bioacoustics tools compared, for Raven, Kaleidoscope, Arbimon, and BirdNET beside Potato
• Detecting rubber-stamped pre-labels, for process signals that show whether reviewers accepted pre-labels without checking them

References

1. Chronister, L. M., Rhinehart, T. A., Place, A., & Kitzes, J. (2021). An annotated set of audio recordings of Eastern North American birds containing frequency, time, and species information. Ecology, 102(6). https://doi.org/10.1002/ecy.3329
2. Kahl, S., Wood, C. M., Eibl, M., & Klinck, H. (2021). BirdNET, a deep learning solution for avian diversity monitoring. Ecological Informatics, 61, 101236. https://doi.org/10.1016/j.ecoinf.2021.101236
3. Wood, C. M., & Kahl, S. (2024). Guidelines for appropriate use of BirdNET scores and other detector outputs. Journal of Ornithology, 165, 777-782. https://doi.org/10.1007/s10336-024-02144-5