Think-Aloud Mode

New in v2.7.0

Before LLMs, the gold standard for understanding judgment was the think-aloud protocol, and it never survived contact with annotation tooling. Think-Aloud Mode lets annotators just talk while they work.

The verbatim transcript is stored as the rationale — deliberately un-summarized, because paraphrasing a think-aloud protocol contaminates the artifact you are trying to collect. Labels can be committed by voice using set phrasings detected by a rule-based parser, with no LLM anywhere in the pipeline.

Speech-to-text runs fully locally via faster-whisper, which is CPU real-time on the 39 MB tiny.en model. No cloud APIs, no per-token cost, and nothing leaves the machine.

Recording human chain-of-thought

The point is not only to collect rationales. Think-Aloud captures how a person actually reasons to a label, so you can hold that human chain-of-thought up against a model's chain-of-thought on the same item.

Potato's process reward annotation segments how a model reasons, step by step. Think-Aloud captures how a person does. Both are capture surfaces, and putting them side by side on the same item is where the interesting divergences show up.

How it works

1. The annotator taps 🎤 Think aloud and speaks freely.
2. Audio is captured in complete six-second chunks and transcribed locally.
3. To commit a label by voice, they use one of the accepted phrasings:
   • "I label this Polite" / "I'd call it neutral"
   • "My answer is impolite"
   • "Final answer: polite" / "I go with neutral"
4. Detection auto-selects the matching option in the UI, the normal save pipeline fires, and the pill confirms: Heard: Impolite ✓. Saying a new phrase later changes the label — the last commitment wins.
5. With require_spoken_label: true, pressing Next with no committed label triggers a one-time nudge showing the expected phrasing. A second Next passes through, and clicking labels always works.

Everything mentioned while thinking is ignored. "This seems polite, but…" commits nothing. Only the set phrasings commit, which is what makes rule-based parsing sufficient. Mishearings are absorbed by fuzzy label matching, so "in polite" resolves to Impolite.

Setup

pip install faster-whisper   # local STT; first recording downloads the model (~39 MB)

The browser needs microphone permission. localhost counts as a secure context.

Configuration

thinkaloud:
  enabled: true
  schema: politeness          # scheme whose labels can be spoken (default: first radio)
  stt: auto                   # faster_whisper | mock | auto
  model: tiny.en              # tiny.en is CPU real-time; base.en is sturdier
  chunk_seconds: 6            # recording chunk length
  require_spoken_label: true  # nudge on Next without a committed label
  # stems:                    # override accepted phrasing regexes (advanced)
  # fillers: [um, uh, hmm, i guess, maybe]
  # language: en

What you get

• Verbatim rationale streams aligned to each (annotator, instance), with the label phrase separated out. The transcript minus the commitment phrase is the rationale.
• Deterministic hesitation signals: silent-chunk counts and filler-word counts over a configurable lexicon, computed with arithmetic rather than models.
• A review page at /thinkaloud/review (admin) with every session's transcript, voice-committed label, confidence, and hesitation stats.
• Hands-free annotation as a side effect, including genuine accessibility and RSI relief.

Data and API

Transcripts persist to {output_annotation_dir}/thinkaloud/transcripts.jsonl (append-only, one record per chunk).

Design notes

• The parser runs over a rolling window of the last two chunks, so phrases that straddle a chunk boundary still detect.
• Label matching is exact, then prefix, then difflib at a 0.8 threshold, preferring exact matches. "polite" never fuzzy-collides with "Impolite".
• Adding an STT backend means subclassing STTBackend in potato/thinkaloud/stt.py and registering it in create_stt.

Further reading

• Process reward annotation — the model-side chain-of-thought surface
• Behavioral tracking — timing analytics
• Quality control
• Source documentation