Audio Annotation Services vs In-House Annotation Tools

An audio annotation service provides trained annotators and returns labeled audio for a price quoted per project. An in-house annotation tool provides only the software, and your team recruits, trains, and checks the annotators. Pay a vendor when the work needs languages or volume your team cannot staff, and annotate in-house when the recordings cannot leave your institution or the labels depend on judgment only your team has.

Labeled data for speech and audio covers transcripts, speaker diarization (who spoke when), sound events, emotion, and spoken intent. An audio annotation service is a company that staffs annotators for those tasks and delivers the labels. Another name for such a company is a data annotation vendor. An annotation tool is software that shows the recording to an annotator and records the labels. Transcripts are usually scored by word error rate, and segment and category labels by inter-rater reliability between independent annotators.

Potato is a free, open-source annotation tool (GPL-3.0-or-later) that runs on your own server, and the Potato team does not provide annotators. The guide below covers the decision between paying for labeled audio and producing labeled audio yourself. Audio Annotation Tools Compared compares the tools for the in-house route, and Audio Annotation sets up each audio task in Potato. Expert vs Crowd Annotation covers who should label, and Build or Buy an Annotation Tool covers the software decision for any data type.

Audio services named on vendor pages

Vendors describe audio work in a shared vocabulary of transcription, diarization, sound events, and paralinguistic or emotion labels. Vendors sell the work with a managed workforce and a quality process. The Potato team read each vendor's own audio or speech service page on November 2, 2026. The table below lists the vendors in alphabetical order and records what each page states.

CVAT.ai publishes the open-source CVAT tool and sells labeling services alongside the tool. Shaip's page lists TextGrid, ELAN, and CTM among Shaip's output formats. Potato reads all three formats, as the hybrid section below shows.

Questions to ask an audio annotation vendor

Vendor pages describe a quality process and sometimes a single accuracy or agreement percentage. You can judge a percentage only after you know how the vendor computed the percentage. Ask for the following before signing:

• Sample work on your own audio. Send the vendor a sample of your hardest recordings along with your guidelines. Score the returned labels yourself before committing the full budget.
• Agreement figures and the method behind each figure. Ask which coefficient the vendor reports, how many annotators labeled each item, and whether the figure covers your task or a different project. For transcripts, ask whether word error rate is measured against a second annotator or against a reviewer's correction.
• Language and dialect coverage. Ask how many annotators are native speakers of each dialect in your audio, and how code-switched speech is handled.
• Data handling and retention. Ask where the audio is stored, who can listen to the recordings, whether annotators work from home or from a controlled facility, and when the vendor deletes the audio after delivery.
• Rights to the audio and the labels. Confirm that your consent forms allow a third party to hear the recordings, and that the contract assigns the labels to you and bars reuse of your audio in the vendor's own datasets.
• Pricing unit. Ask whether the price is per audio minute, per audio hour, per label, or per annotator hour, and what a rework round costs.

The Partnership on AI publishes a responsible sourcing library for teams buying annotation work. In the library's guidelines, the Partnership on AI asks buyers to pay workers above the local living wage, run a pilot before launching, and provide verified instructions or training. The library includes a vendor comparison template and a living-wage spreadsheet.

In-house audio annotation with Potato

Running the work in-house means your team owns each part of the study described below. Potato covers the software, and your team covers the people and the time.

The tool. Potato's audio_annotation scheme shows a waveform and an optional spectrogram. Annotators drag across the waveform to create labeled segments. The speech_transcript scheme presents an existing transcript segment by segment for correction, and the tiered_annotation scheme holds ELAN-style tiers that export to EAF and TextGrid.

The annotators. Annotators can come from your own lab, from a university subject pool through SONA, or from Prolific. Potato connects to Prolific through participant IDs in the study link and completion codes at the end. Mechanical Turk closed on September 30, 2026. Running a Crowdsourced Study covers recruitment and fair pay.

Training and quality checks. A training phase gives annotators practice clips with feedback and admits only annotators who meet the passing criteria. Gold items and attention checks are mixed into the real task to score accuracy and catch annotators who are not listening.

Agreement. For audio segments, Potato compares every pair of annotators on temporal IoU and on detection F1 at a matching threshold of 0.5 by default. Segment agreement in Potato has no chance correction yet, and Agreement over Time explains the reason.

Data handling. The recordings stay on the server you run. Potato's local transcription uses faster-whisper on your own machine, and the whole setup runs air-gapped once the model weights are on the machine, as Air-Gapped and Offline Deployment describes.

The main cost of in-house annotation is your team's time. Run a short pilot to measure how long each clip takes. The annotation cost calculator turns that per-clip time into a budget.

Choosing between a service and in-house annotation

Each common situation below is paired with the route that usually fits the situation:

Hybrid setups with a vendor and your own server

Two hybrid patterns let a team buy capacity and keep control of quality.

Check vendor labels on a sample. Import the vendor delivery into Potato, have your own annotators label a random sample independently, and compare the two sets. potato import reads Praat TextGrid and ELAN EAF files as tiered projects with the vendor's segments as pre-annotations, as Importing a Project describes. Transcript Formats lists the 21 transcript formats that load for correction. WhisperX JSON, NIST CTM, SubRip, and WebVTT are among the 21 formats. Potato does not parse standalone RTTM diarization files, and an RTTM delivery needs converting to a supported format first.

Bring the vendor's annotators into your own Potato server. Potato's expert invite provider gives each contracted annotator a private link with a token. The vendor's annotators then label on your server, and the recordings are stored only on your server. Your training phase, gold items, and agreement report apply to the vendor's annotators in the same way as to your own annotators. The following minimal config sets up speaker turns on call recordings:

annotation_task_name: "Call center speaker turns"
task_dir: "."
data_files:
  - "data/clips.jsonl"
item_properties:
  id_key: id
  text_key: audio_url
output_annotation_dir: "annotation_output/"
 
login:
  type: url_direct
  url_argument: invite
 
crowdsourcing:
  provider: expert
  expert:
    invites_file: "expert_invites.yaml"
 
annotation_schemes:
  - annotation_type: audio_annotation
    name: speaker_turns
    description: "Mark each stretch where one speaker talks and choose the speaker."
    mode: label
    labels:
      - {name: agent, color: "#4ECDC4"}
      - {name: caller, color: "#FF6B6B"}
      - {name: overlap, color: "#F39C12"}
    spectrogram: true

expert_invites.yaml maps each token to a display name, and Potato rejects any token missing from the file. Send each annotator https://your-server/?invite=<token>. The admin dashboard reports annotations and time on task per annotator, and you can check the vendor's invoices against those figures. The expert annotator documentation covers token generation.

Frequently asked questions

Which companies offer audio annotation services?

Appen, CVAT.ai, iMerit, LXT, and Shaip are among the companies whose own pages offer audio or speech annotation. Every one of the five pages offers transcription, and most of the pages also offer diarization or sound events. The section "Audio services named on vendor pages" records what each page stated on November 2, 2026. Before choosing, request sample work on your own recordings and score the returned labels yourself.

What should developers look for in an audio annotation service?

Developers should look for sample work on the team's own audio, an agreement or word error rate figure with the method behind the figure, native-speaker coverage of the dialects in the data, and a delivery format the pipeline already reads. Check retention and reuse terms for the audio before sending any recordings. A delivery in TextGrid, EAF, CTM, or WhisperX JSON loads into Potato. Your own team can then check a sample of the vendor labels in Potato.

Should I use an audio annotation service or annotate audio in-house?

Use a service when the volume or the language coverage exceeds what your team can recruit and train in time. Annotate in-house when the recordings cannot go to a third party, when the labels need specialist judgment, or when the study must report how annotators were trained and how well the annotators agreed. A hybrid design has a vendor label the bulk of the audio and your team check a sample in-house.

Further reading

• Audio Annotation Tools Compared, for choosing the in-house tool
• Audio Annotation, for each audio task set up in Potato
• Potato for audio and speech annotation
• Expert vs Crowd Annotation
• Build or Buy an Annotation Tool
• Running a Crowdsourced Study on Prolific and SONA
• Annotating Whisper Transcripts