# Shared Codebook for Qualitative Coding

Source: https://www.potatoannotator.com/docs/qda/codebook

Potato's codebook is the list of codes a project's coders apply. Potato keeps the codebook in the project database instead of the YAML config. A scheme opts in with `codebook: true`, and the scheme's options then come from the codebook. A coder can add a code in the middle of a session without anyone editing the config or restarting the server. Codes can sit under a parent code, and each code's entry has typed fields that include a definition, inclusion and exclusion rules, and examples. Potato stamps every saved annotation with the codebook revision in effect at the moment of saving. The codebook arrived with QDA Mode in Potato 2.6.0 and works in standard annotation, [Solo Mode](/docs/features/solo-mode), and [QDA Mode](/docs/qda/qda-mode).

A [codebook](https://en.wikipedia.org/wiki/Coding_(social_sciences)) in qualitative research lists the codes a team applies and says when each code applies. [MacQueen et al. (1998)](https://doi.org/10.1177/1525822X980100020301) argue that a codebook with the same fixed structure for every code gives a team of coders a stable frame and improves [intercoder agreement](https://en.wikipedia.org/wiki/Inter-rater_reliability). Potato gives every code the same set of typed fields, from a short definition to counter-examples. The [codebook development guide](/docs/guides/qualitative-codebook-development) covers how a team writes and revises the entries.

## Codebook storage and seeding

- **Storage.** Codes live in `<task_dir>/project.sqlite`. Memos and cases share the same project database. The YAML holds only the starting labels.
- **Seeding.** On the first server start, Potato copies the `labels` of every scheme marked `codebook: true` into an empty codebook. From then on the database is the source of truth, and the server ignores later edits to the YAML labels.
- **One codebook per project.** Every scheme marked `codebook: true` shows the same codes. A project with a `themes` scheme and a `quotes` scheme draws both from one list.
- **Scheme types.** `radio`, `multiselect`, `select`, and `span` accept `codebook: true`.
- **Revisions.** Adding, renaming, recoloring, moving, deleting, merging, or splitting a code raises the project's codebook revision by one. Potato records the revision on every annotation saved afterward.
- **Where coders see the codebook.** A Codebook toggle on the right edge of the annotation page opens a tray listing each code with the code's short definition. The `/codebook` page shows the full codebook, with every field of every code.

## Configuration

The config below sets up interview coding for a team in `extensible` mode. Coders can add a code in the middle of reading, and only an adjudicator may rename or delete codes. The two seed labels use the object form, and an object-form label starts the code with a color and a short definition.

```yaml
annotation_task_name: Clinic Access Interviews
task_dir: .
output_annotation_dir: annotation_output/
data_files:
- data/interviews.json
item_properties:
  id_key: id
  text_key: text
user_config:
  allow_all_users: true

codebook_mode: extensible

annotation_schemes:
- annotation_type: span
  name: codes
  description: Highlight a passage and apply a code
  codebook: true
  labels:
  - name: access barriers
    color: "#4682b4"
    description: Anything the speaker names as standing between them and care they wanted.
  - name: cost concerns
    color: "#b44682"
    description: Money, insurance, or the fear of a bill.
```

| Option | Default | Description |
|--------|---------|-------------|
| `annotation_schemes[].codebook` | `false` | Takes the scheme's options from the project codebook. |
| `annotation_schemes[].labels` | (none) | Seeds an empty codebook on first start. A label may be a string or an object with `name`, `color`, and `description` (or `tooltip`). The description becomes the code's short definition. |
| `codebook_mode` | `open` when `qda_mode.enabled` or `solo_mode.enabled` is true, otherwise `fixed` | Who may change codes: `fixed`, `extensible`, or `open`. Wins over `codebook.mode` when both are set. Any other value fails validation. |
| `codebook.mode` | same as `codebook_mode` | The nested spelling of `codebook_mode`. |
| `codebook.enabled` | unset | Turns the tray, the `/codebook` page, and the codebook API on or off. Unset, the codebook turns on when a scheme sets `codebook: true` or under QDA Mode or Solo Mode. A config that sets `codebook_mode` has the codebook on whatever `codebook.enabled` says. |
| `codebook.distiller.include` (or `include_types`) | `short_def`, `definition`, `use_when`, `avoid_when`, `example`, `counter_example` | Which fields of each code go into the text Potato adds to [ICL labeling](/docs/features/icl-labeling) prompts. |
| `codebook.distiller.include_doc_sections` | `preamble`, `general_instructions` | Which codebook-wide sections go into the same prompt text. |
| `codebook.distiller.scope` | `per_code` | `per_code` adds each code's fields. `whole_doc` adds only the codebook-wide sections. |
| `codebook.distiller.procedure` | `concat` | How the prompt text is assembled. `llm_summarize` and `select_icl_examples` are accepted and produce the same output as `concat` in 2.13.1. |
| `codebook.distiller.max_chars` | no limit | Truncates the prompt text to the given number of characters. |
| `codebook_invivo_key` | `i` | Key that opens the in-vivo composer while text is selected in a codebook-backed `span` scheme. Only the first character counts. |

`qda_mode.codebook.mode` passes validation. Potato 2.13.1 does not read the key when deciding who may edit. Under QDA Mode, set `codebook_mode` to change the default of `open`.

## Who can change codes

What a logged-in coder may do depends on the mode. An [adjudicator](/docs/guides/adjudication-and-disagreement) may make every structural change in `extensible` and `open` mode. In `fixed` mode neither coders nor adjudicators can add, rename, recolor, move, or delete a code. Admins and adjudicators can still merge and split codes from the tray's Curate section.

| Mode | Coders may | Typical use |
|------|------------|-------------|
| `fixed` | apply codes only | a reliability round, crowd work, a codebook agreed before coding |
| `extensible` | add codes | team coding with an agreed core and room for new codes |
| `open` | add, rename, recolor, move, and delete codes, and edit definitions | one analyst, or a small team developing the codebook together |

Potato locks the mode to `fixed` for any crowdsourcing configuration and logs a warning when the config asked for more. Potato applies the lock to keep recruited annotators from reshaping a shared codebook. The lock covers `crowdsourcing.provider` with any value except `expert`, a top-level `prolific:` or `mturk:` block, and `login.type: prolific` or `mturk`. An unrecognized provider name and an empty provider value lock the mode too.

`crowdsourcing.provider: expert` keeps the requested mode, because experts are invited by name. `login.type: url_direct` alone keeps the requested mode and logs a warning. Anyone with the link to such a study can register and then edit an open codebook.

Each kind of change has a separate route in 2.13.1:

| Change | In the browser | Over the API | Who |
|--------|----------------|--------------|-----|
| Add a code | the tray's "Add a code" box, or the in-vivo key | `POST /api/codebook` | `extensible` or `open` |
| Rename, recolor, or move under a parent | no control in 2.13.1 | `PATCH /api/codebook/<id>` | `open`, or an adjudicator in `extensible` |
| Delete a code | no control in 2.13.1 | `DELETE /api/codebook/<id>` | `open`, or an adjudicator in `extensible` |
| Merge one code into another | the tray's Curate section | `POST /api/codebook/admin/merge` | an admin or adjudicator, in any mode |
| Split a code by coder | the tray's Curate section | `POST /api/codebook/admin/split` | an admin or adjudicator, in any mode |
| Edit a definition or add an example | the pencil on a code in the tray, or `/codebook` | `PUT /api/codebook/blocks` | `open`, or an adjudicator in `extensible`. Anyone else's edit is queued for an admin or adjudicator to confirm. |

The tray's add box creates a top-level code. A code goes under a parent through the `parent_id` field of `POST` or `PATCH`.

### In-vivo coding from a text selection

On a `span` scheme marked `codebook: true`, a coder selects a passage and presses the in-vivo key. A small composer opens with a name taken from the selected words (up to 60 characters), and "Create & code" adds the code and highlights the passage with the new code in one step. While the coder types the name, existing codes with similar names appear as chips. A coder who chooses a chip applies the existing code instead of creating a near-duplicate such as `cost` beside `cost concerns`.

### Model-proposed changes

A model never changes a shared codebook directly. An agent or a [Solo Mode](/docs/features/solo-mode) model stages a change with `POST /api/codebook/proposals` and `actor_kind: "model"`. The change waits in the Curate section as a sentence such as "Merge «cost» into «cost concerns»". An admin or adjudicator confirms or rejects the proposal. A confirmed proposal runs through the same path as a human edit and is marked as model-originated in the change log.

## Nesting codes

A code with a `parent_id` sits under that parent, and a parent can sit under another parent. A move that would place a code under one of the code's own descendants fails with an error. Adding a second code with the same name under one parent fails the same way. Deleting a parent deletes every code under the parent.

The tray and `/codebook` show the tree. The annotation form shows the codes as one flat list in tree order, parents before children. A form option and a saved annotation both hold only the code's name. The same name under two parents therefore appears twice on the form, and the two codes cannot be told apart afterward. Keep code names unique across the whole codebook.

The `codebook` CSV export and the `.qdpx` export both write a codebook-backed scheme's codes as a flat list in 2.13.1. The CSV's `parent` and `color` columns are empty for those codes. A code hierarchy needed in another program has to be rebuilt there, or recorded in the code names.

## Effect of code changes on saved codings

Potato saves each coding with the code's name as the label, in the coder's `user_state.json`. A merge or split never deletes a row from the database. Potato applies each codebook change to the codebook alone and never rewrites a saved annotation:

- **Adding a code** leaves saved codings unchanged.
- **Renaming a code** changes the form from then on. Codings saved under the old name keep the old name, and exports and agreement statistics count those codings under the old name. Rename before a coding round starts, or recode the affected excerpts.
- **Deleting a code** removes the code and every code under the code from the form. Saved codings keep the name, and the change log keeps the deleted code's name.
- **Merging** archives the source code. An archived code leaves the form, and the code's row and history stay in the database. Codings saved with the source code keep the source code's name.
- **Splitting by coder** names one coder and a new or existing target code. The split is meant for two coders who used one code name for different things, and is designed to move only the named coder's uses. In 2.13.1 the annotation form does not record the per-coder links a split moves. A split therefore creates the target code and archives the source code for every coder whenever the source code has no child codes. Saved codings keep the name each coder chose.

Potato tells coders about changes in two ways. First, a banner on an excerpt saved before new codes were added names the new codes. Second, the tray's **Review** list holds every excerpt the coder saved before the latest change of any kind. Each excerpt in the list has a **Go** button. Both signals can be dismissed, and Potato never forces a recode. An admin or adjudicator sees the same list for the whole project at `GET /api/codebook/admin/stale`.

Definition edits have a separate counter. An edit to a code's short definition, definition, "Use when", or "Avoid when" field raises a semantic revision, and Potato records the semantic revision on every annotation saved afterward. An analyst can then separate codings made before and after a definition changed. Ticking **Minor edit (no re-review)** in the `/codebook` editor saves the wording without raising the semantic revision. The Review list in 2.13.1 tracks structural changes only.

Set `codebook_mode: fixed` for a reliability round, because Potato computes agreement over saved code names and a code renamed mid-round becomes two codes. [Inter-Annotator Agreement](/docs/guides/inter-annotator-agreement) covers the coefficients. The agreement timeline on `/admin/iaa` marks each codebook revision against agreement over time. [Agreement Drift](/docs/measurement/agreement-drift) explains how to read the timeline.

## Typed fields in each code's entry

Each code's entry is a list of typed fields. `/codebook` is the full-page editor, with a version history for every code, a diff between versions, and a restore that saves an older version as a new version. A "Paste markdown" button turns pasted codebook prose into fields by heading and asks the user to choose a type for any section the parser does not recognize. Two people saving the same code at once do not overwrite each other. The second save fails and returns the current text to reconcile against.

| Field | Heading in the editor | Semantic |
|-------|----------------------|----------|
| `short_def` | Short definition | yes |
| `definition` | Definition | yes |
| `use_when` | Use when | yes |
| `avoid_when` | Avoid when | yes |
| `example` | Examples | no |
| `counter_example` | Counter-examples | no |
| `rationale` | Rationale | no |
| `background` | Background | no |
| `downstream_usage` | Downstream usage | no |
| `keywords` | Keywords | no |
| `notes` | Notes | no |
| `custom` | the user's own heading | no |

The codebook as a whole has four sections outside the codes, titled Preamble, General instructions, Background, and How codes are used downstream. The tray shows each code's short definition, clamped to two lines. When a code has no short definition, the tray shows the full definition instead.

The distiller turns the chosen fields into text that Potato adds to [ICL labeling](/docs/features/icl-labeling) prompts, and rebuilds the text after every codebook change. `GET /api/codebook/distilled` returns the text a model currently receives. Add a `codebook.distiller` block to the config above to choose the fields:

```yaml
codebook:
  distiller:
    include: [definition, use_when, avoid_when, example]
    procedure: concat
```

## Seeding and migrating with the CLI

`potato codebook` adds a code for every label of every `codebook: true` scheme that the codebook does not have yet, and leaves existing codes alone:

```bash
potato codebook config.yaml
```

The command prints how many codes were created and how many were already present. Codes created by the CLI get the same ID on every machine, because the CLI computes each ID from the project name, the parent, and the code name. Run the CLI before the first server start when IDs need to match across copies of a project, because codes seeded by the server get random IDs. The CLI copies names only. Colors and definitions from object-form labels reach the codebook only through the server's first-start seed.

Because the server seeds only an empty codebook, the CLI is the way to add new YAML labels to a codebook that already has codes. To preview without writing, run the module directly. In 2.13.1, `potato codebook --dry-run` drops the flag and writes the codes.

```bash
python -m potato.codebook_cli config.yaml --dry-run
```

## API

All routes sit under `/api/codebook` and need a logged-in session. Admin routes accept an admin or an adjudicator.

| Method | Path | Purpose |
|--------|------|---------|
| GET | `/api/codebook` | the tree, the flat label list, the revision, the codebook schemes, and the caller's `can_add` and `can_edit` |
| GET | `/api/codebook/version` | the structural and content revisions, for cheap polling |
| GET | `/api/codebook/similar?name=` | existing codes with names close to `name` |
| GET | `/api/codebook/provenance?instance_id=` | whether the caller's coding of one item predates the current revision, and the codes added since |
| GET | `/api/codebook/stale` | the caller's Review list |
| POST | `/api/codebook` | add a code (`name`, optional `color` and `parent_id`) |
| PATCH | `/api/codebook/<id>` | rename (`name`), recolor (`color`), or move (`parent_id`) |
| DELETE | `/api/codebook/<id>` | delete a code and every code under the code |
| POST | `/api/codebook/proposals` | queue a change (`op`, `payload`, `actor_kind`) |
| GET | `/api/codebook/admin/stale` | the Review list for every coder (admin) |
| POST | `/api/codebook/admin/merge` | merge `src_id` into `dst_id` (admin) |
| POST | `/api/codebook/admin/split` | move `annotator`'s links from `src_id` to `target_id` or a new code named `new_name` (admin) |
| GET | `/api/codebook/admin/changes` | the change log (admin) |
| GET | `/api/codebook/admin/proposals` | pending proposals (admin) |
| POST | `/api/codebook/admin/proposals/<id>/confirm` | run a proposal (admin) |
| POST | `/api/codebook/admin/proposals/<id>/reject` | discard a proposal (admin) |

The definition editor uses `GET /api/codebook/document`, `GET` and `PUT /api/codebook/blocks`, `POST /api/codebook/parse`, `GET /api/codebook/history`, `GET /api/codebook/history/<id>`, `POST /api/codebook/restore`, and `GET /api/codebook/distilled`. Under QDA Mode, `/qda/codebook` returns a read-only view of the codebook.

## Codebook export and change history

The `codebook` export writes `codebook.csv`, one row per code per codebook scheme, with the columns `schema`, `annotation_type`, `code`, `parent`, `description`, `color`, and `n_uses`. The export reads the codebook database. Codes added during coding are included, and merged codes are left out. `description` holds the code's short definition, or the full definition when the code has no short definition. `n_uses` counts saved codings by name. A coding saved under a name the codebook no longer has counts toward no row.

```bash
python -m potato.export -c config.yaml --format codebook -o ./export/
```

The `quotation_report` export lists every coded passage with the passage's code. The `qdpx` export hands the codes and codings to another QDA program. [QDA Mode](/docs/qda/qda-mode) covers both exports, and [Export Formats](/docs/features/export-formats) lists the columns.

The history stays in `project.sqlite`. The `codebook_change` table logs every rename, recolor, move, delete, merge, split, and confirmed or rejected proposal, with the old and new values, the actor, whether the actor was a human or a model, and the revision. In 2.13.1, a rename, recolor, move, or delete made through `PATCH` or `DELETE` is logged with the actor `system`, and code additions appear in the `codes` table with the creator's username instead of in the log. `GET /api/codebook/admin/changes` returns the change log as JSON.

`codebook_snapshot` keeps every saved version of every definition, and `codebook_revision_history` records when each revision happened. A [project snapshot](/docs/tools/project-snapshots) and the archive from `potato deploy pull` both include `project.sqlite`. Either bundle is an audit trail of the codebook for a methods section.

Run the bundled example from a Potato checkout:

```bash
python potato/flask_server.py start examples/advanced/codebook-example/config.yaml -p 8000
```

The example codes five patient comments about clinic access with an open multiselect codebook of four themes. `examples/advanced/codebook-document-example/` adds a distiller block. Open `/codebook` in that example to see the definition editor.

## Further reading

- [QDA Mode](/docs/qda/qda-mode), for the codebook together with memos, cases, and search
- [Memos](/docs/qda/memos), for analytic notes beside the codes
- [Developing a Qualitative Codebook](/docs/guides/qualitative-codebook-development), on writing definitions and revising codes as a team
- [Intercoder Reliability in Qualitative Research](/docs/guides/intercoder-reliability-qualitative-research), on when to report agreement and which coefficient to use
- [Agreement Drift](/docs/measurement/agreement-drift), for agreement over time against codebook revisions

## References

MacQueen, K. M., McLellan, E., Kay, K., & Milstein, B. (1998). Codebook Development for Team-Based Qualitative Analysis. *CAM Journal*, 10(2), 31-36. https://doi.org/10.1177/1525822X980100020301

For implementation details, see the [source documentation](https://github.com/davidjurgens/potato/blob/master/docs/advanced/codebook.md).