How to Annotate YouTube Subtitles

Download captions with yt-dlp and point Potato at the files. The thing to know first: auto-generated captions have no punctuation and no speaker labels, and their line breaks are meaningless, which constrains what you can sensibly annotate. Potato reads every caption format yt-dlp emits. See Transcript Formats for the full list.

Video platforms are a large and convenient source of naturalistic speech data. Closed captioning makes that speech text, and the text is small, easy to move around, and easy to annotate. The catch is that not all captions are the same kind of object, and treating them as if they were is where most caption-based annotation projects go wrong.

How do I download the captions?

yt-dlp fetches subtitles with or without the media:

# Human-written subtitles, if the uploader provided any
yt-dlp --write-subs --sub-langs en --skip-download <URL>
 
# Auto-generated captions
yt-dlp --write-auto-subs --sub-langs en --sub-format vtt --skip-download <URL>

Potato reads vtt, srt, json3, ttml, and the srv1, srv2, and srv3 variants, so whichever --sub-format you pick will parse. Choose based on what you want to keep: json3 preserves per-word timings, while WebVTT and SubRip are easier to read and edit by hand.

What is the difference between human subtitles and auto-captions?

They look alike in a text editor and behave very differently as annotation input:

Neither is strictly better. Human subtitles read well but get edited for readability, so they drop filler, compress repetition, and sometimes rewrite a sentence entirely. Auto-captions are closer to verbatim, which is what you want for anything about how people actually talk, but they arrive as an unpunctuated stream.

What can I annotate on auto-captions?

Do not design a sentence-level annotation scheme over auto-captions. Their cue boundaries fall wherever the caption window filled up, not where a sentence ended. A cue in an auto-caption file is a display artifact, not a linguistic unit. Ask annotators to rate sentences and they will each silently pick different sentence boundaries, and your agreement numbers will end up measuring that instead of the thing you cared about.

Two workable options:

• Annotate the caption windows as they are. Label each cue with whatever applies to it, and accept that the unit is arbitrary but consistent. Fine for topic, presence of a phenomenon, or anything you can judge on a fragment.
• Re-segment upstream. Run a punctuation-restoration model or an ASR system that punctuates, then re-time. More work, but you get real units.

If you have human subtitles, this problem mostly goes away. The cue boundaries were placed by someone reading along, so they usually respect clauses.

What about the video?

Captions are small and easy to redistribute. Video usually is not. Three options:

# Media hosted somewhere your annotators can reach
potato transcripts ./captions --media-url-prefix https://cdn.example.org/video -o data/talks.json
 
# Media downloaded locally as audio only
yt-dlp -f 'ba' -x --audio-format mp3 -o './audio/%(id)s.%(ext)s' <URL>
potato transcripts ./captions --media-dir ./audio -o data/talks.json

The third option is no media at all. Transcripts without media annotate fine; annotators read instead of listening. If that is your plan, say so in the instructions, because annotators otherwise assume the missing player is a bug and file it as one.

Warning: Check your rights before redistributing anything you did not create. Downloading captions or media for your own research is a different question from serving that media to a pool of annotators, and platform terms, copyright, and your institution's rules all bear on it. Resolve this before you build the pipeline, not after.

How do I set up the task?

The converter writes a data file with the transcript already normalized:

potato transcripts ./captions --media-dir ./audio -o data/talks.json

{
  "id": "talk_01",
  "conversation": {
    "audio": "audio/talk_01.mp3",
    "turns": [
      {"turn_id": "t0", "speaker": null, "start": 0.0, "end": 4.2,
       "text": "so the thing about caption windows is"}
    ]
  }
}

speaker: null is what an auto-caption gives you. Those turns render as Unassigned with a picker, so annotators can attribute them while they listen.

Auto-captions carry no speakers, so each cue is Unassigned until an annotator picks one

annotation_task_name: "Talk Annotation"
task_dir: .
data_files:
  - data/talks.json
 
item_properties:
  id_key: id
  text_key: conversation
 
instance_display:
  fields:
    - key: conversation
      type: audio_dialogue
      label: "Captions"
      span_target: true
      display_options:
        show_timestamps: true
        allow_speaker_assignment: auto
 
annotation_schemes:
  - annotation_type: radio
    name: cue_topic
    description: "What is this caption window about?"
    labels: [setup, argument, example, aside]
    turn_level: true
    turn_binding:
      field: conversation

turn_level: true attaches the question to every cue rather than to the video as a whole, which is the right unit when the cues are all you have. Run it:

python potato/flask_server.py start config.yaml -p 8000

You can also skip the converter entirely and point a data file at the caption files on disk. See Annotating Whisper Transcripts, where the rest of this workflow is the same.

Further reading

• Transcript Formats, every supported caption and transcript format
• Annotating Whisper Transcripts, the ASR-output version of this workflow
• Transcript Format Ingestion, a runnable showcase design with six formats side by side
• Video Annotation, for annotating the video rather than its captions
• Designing Data Formats for Annotation
• Timed Text Markup Language, the broadcast caption standard Potato also reads