# Think-Aloud Mode

Source: https://www.potatoannotator.com/docs/features/think-aloud

*New in v2.7.0*

Before LLMs, the gold standard for understanding judgment was the [think-aloud protocol](https://en.wikipedia.org/wiki/Think_aloud_protocol), and it never survived contact with annotation tooling. Think-Aloud Mode lets annotators just talk while they work.

The verbatim transcript is stored as the rationale — deliberately un-summarized, because paraphrasing a think-aloud protocol contaminates the artifact you are trying to collect. Labels can be committed by voice using set phrasings detected by a rule-based parser, with no LLM anywhere in the pipeline.

Speech-to-text runs **fully locally** via [faster-whisper](https://github.com/SYSTRAN/faster-whisper), which is CPU real-time on the 39 MB `tiny.en` model. No cloud APIs, no per-token cost, and nothing leaves the machine.

![A verbatim spoken-reasoning transcript with a detected voice label — human chain-of-thought, to compare against the model's.](/images/docs/think-aloud.png)

## Recording human chain-of-thought

The point is not only to collect rationales. Think-Aloud captures *how a person actually reasons to a label*, so you can hold that human chain-of-thought up against a model's chain-of-thought on the same item.

Potato's [process reward annotation](/docs/features/process-reward-annotation) segments how a **model** reasons, step by step. Think-Aloud captures how a **person** does. Both are capture surfaces, and putting them side by side on the same item is where the interesting divergences show up.

## How it works

1. The annotator taps **🎤 Think aloud** and speaks freely.
2. Audio is captured in complete six-second chunks and transcribed locally.
3. To commit a label by voice, they use one of the accepted phrasings:
   - *"I label this **Polite**"* / *"I'd call it **neutral**"*
   - *"My answer is **impolite**"*
   - *"Final answer: **polite**"* / *"I go with **neutral**"*
4. Detection auto-selects the matching option in the UI, the normal save pipeline fires, and the pill confirms: **Heard: Impolite ✓**. Saying a new phrase later changes the label — the last commitment wins.
5. With `require_spoken_label: true`, pressing **Next** with no committed label triggers a one-time nudge showing the expected phrasing. A second Next passes through, and clicking labels always works.

Everything mentioned while *thinking* is ignored. "This seems polite, but…" commits nothing. Only the set phrasings commit, which is what makes rule-based parsing sufficient. Mishearings are absorbed by fuzzy label matching, so "in polite" resolves to *Impolite*.

## Setup

```bash
pip install faster-whisper   # local STT; first recording downloads the model (~39 MB)
```

The browser needs microphone permission. `localhost` counts as a secure context.

## Configuration

```yaml
thinkaloud:
  enabled: true
  schema: politeness          # scheme whose labels can be spoken (default: first radio)
  stt: auto                   # faster_whisper | mock | auto
  model: tiny.en              # tiny.en is CPU real-time; base.en is sturdier
  chunk_seconds: 6            # recording chunk length
  require_spoken_label: true  # nudge on Next without a committed label
  # stems:                    # override accepted phrasing regexes (advanced)
  # fillers: [um, uh, hmm, i guess, maybe]
  # language: en
```

| Option | Default | Description |
|--------|---------|-------------|
| `stt` | `auto` | `faster_whisper` (local), `mock` (tests and dev), or `auto`, which picks faster-whisper and errors helpfully if it is missing. |
| `model` | `tiny.en` | Any faster-whisper model id. |
| `chunk_seconds` | `6` | Each chunk is a complete audio file. |
| `stems` | built-ins | Regex stems for accepted phrasings; each captures the words that follow. |
| `fillers` | um, uh, hmm, … | Lexicon for the hesitation counter. |
| `require_spoken_label` | `true` | One-time Next-button nudge when nothing was committed. |

## What you get

- **Verbatim rationale streams** aligned to each (annotator, instance), with the label phrase separated out. The transcript minus the commitment phrase is the rationale.
- **Deterministic hesitation signals**: silent-chunk counts and filler-word counts over a configurable lexicon, computed with arithmetic rather than models.
- **A review page** at `/thinkaloud/review` (admin) with every session's transcript, voice-committed label, confidence, and hesitation stats.
- Hands-free annotation as a side effect, including genuine accessibility and RSI relief.

## Data and API

Transcripts persist to `{output_annotation_dir}/thinkaloud/transcripts.jsonl` (append-only, one record per chunk).

| Endpoint | Method | Auth | Purpose |
|----------|--------|------|---------|
| `/thinkaloud/api/chunk` | POST | session | Multipart audio chunk → transcript + detection |
| `/thinkaloud/api/text` | POST | session | Text chunk (no-audio path) |
| `/thinkaloud/api/state` | GET | session | Session aggregate for an instance |
| `/thinkaloud/review` | GET | admin | Transcript review page |
| `/thinkaloud/api/export` | GET | admin | All sessions as JSON |

## Design notes

- The parser runs over a rolling window of the last two chunks, so phrases that straddle a chunk boundary still detect.
- Label matching is exact, then prefix, then `difflib` at a 0.8 threshold, preferring exact matches. "polite" never fuzzy-collides with "Impolite".
- Adding an STT backend means subclassing `STTBackend` in `potato/thinkaloud/stt.py` and registering it in `create_stt`.

## Further reading

- [Process reward annotation](/docs/features/process-reward-annotation) — the model-side chain-of-thought surface
- [Behavioral tracking](/docs/features/behavioral-tracking) — timing analytics
- [Quality control](/docs/features/quality-control)
- [Source documentation](https://github.com/davidjurgens/potato/blob/main/docs/advanced/think_aloud.md)