Shared Codebook for Qualitative Coding

Potato's codebook is the list of codes a project's coders apply. Potato keeps the codebook in the project database instead of the YAML config. A scheme opts in with codebook: true, and the scheme's options then come from the codebook. A coder can add a code in the middle of a session without anyone editing the config or restarting the server. Codes can sit under a parent code, and each code's entry has typed fields that include a definition, inclusion and exclusion rules, and examples. Potato stamps every saved annotation with the codebook revision in effect at the moment of saving. The codebook arrived with QDA Mode in Potato 2.6.0 and works in standard annotation, Solo Mode, and QDA Mode.

A codebook in qualitative research lists the codes a team applies and says when each code applies. MacQueen et al. (1998) argue that a codebook with the same fixed structure for every code gives a team of coders a stable frame and improves intercoder agreement. Potato gives every code the same set of typed fields, from a short definition to counter-examples. The codebook development guide covers how a team writes and revises the entries.

Codebook storage and seeding

• Storage. Codes live in <task_dir>/project.sqlite. Memos and cases share the same project database. The YAML holds only the starting labels.
• Seeding. On the first server start, Potato copies the labels of every scheme marked codebook: true into an empty codebook. From then on the database is the source of truth, and the server ignores later edits to the YAML labels.
• One codebook per project. Every scheme marked codebook: true shows the same codes. A project with a themes scheme and a quotes scheme draws both from one list.
• Scheme types. radio, multiselect, select, and span accept codebook: true.
• Revisions. Adding, renaming, recoloring, moving, deleting, merging, or splitting a code raises the project's codebook revision by one. Potato records the revision on every annotation saved afterward.
• Where coders see the codebook. A Codebook toggle on the right edge of the annotation page opens a tray listing each code with the code's short definition. The /codebook page shows the full codebook, with every field of every code.

Configuration

The config below sets up interview coding for a team in extensible mode. Coders can add a code in the middle of reading, and only an adjudicator may rename or delete codes. The two seed labels use the object form, and an object-form label starts the code with a color and a short definition.

annotation_task_name: Clinic Access Interviews
task_dir: .
output_annotation_dir: annotation_output/
data_files:
- data/interviews.json
item_properties:
  id_key: id
  text_key: text
user_config:
  allow_all_users: true
 
codebook_mode: extensible
 
annotation_schemes:
- annotation_type: span
  name: codes
  description: Highlight a passage and apply a code
  codebook: true
  labels:
  - name: access barriers
    color: "#4682b4"
    description: Anything the speaker names as standing between them and care they wanted.
  - name: cost concerns
    color: "#b44682"
    description: Money, insurance, or the fear of a bill.

qda_mode.codebook.mode passes validation. Potato 2.13.1 does not read the key when deciding who may edit. Under QDA Mode, set codebook_mode to change the default of open.

Who can change codes

What a logged-in coder may do depends on the mode. An adjudicator may make every structural change in extensible and open mode. In fixed mode neither coders nor adjudicators can add, rename, recolor, move, or delete a code. Admins and adjudicators can still merge and split codes from the tray's Curate section.

Potato locks the mode to fixed for any crowdsourcing configuration and logs a warning when the config asked for more. Potato applies the lock to keep recruited annotators from reshaping a shared codebook. The lock covers crowdsourcing.provider with any value except expert, a top-level prolific: or mturk: block, and login.type: prolific or mturk. An unrecognized provider name and an empty provider value lock the mode too.

crowdsourcing.provider: expert keeps the requested mode, because experts are invited by name. login.type: url_direct alone keeps the requested mode and logs a warning. Anyone with the link to such a study can register and then edit an open codebook.

Each kind of change has a separate route in 2.13.1:

The tray's add box creates a top-level code. A code goes under a parent through the parent_id field of POST or PATCH.

In-vivo coding from a text selection

On a span scheme marked codebook: true, a coder selects a passage and presses the in-vivo key. A small composer opens with a name taken from the selected words (up to 60 characters), and "Create & code" adds the code and highlights the passage with the new code in one step. While the coder types the name, existing codes with similar names appear as chips. A coder who chooses a chip applies the existing code instead of creating a near-duplicate such as cost beside cost concerns.

Model-proposed changes

A model never changes a shared codebook directly. An agent or a Solo Mode model stages a change with POST /api/codebook/proposals and actor_kind: "model". The change waits in the Curate section as a sentence such as "Merge «cost» into «cost concerns»". An admin or adjudicator confirms or rejects the proposal. A confirmed proposal runs through the same path as a human edit and is marked as model-originated in the change log.

Nesting codes

A code with a parent_id sits under that parent, and a parent can sit under another parent. A move that would place a code under one of the code's own descendants fails with an error. Adding a second code with the same name under one parent fails the same way. Deleting a parent deletes every code under the parent.

The tray and /codebook show the tree. The annotation form shows the codes as one flat list in tree order, parents before children. A form option and a saved annotation both hold only the code's name. The same name under two parents therefore appears twice on the form, and the two codes cannot be told apart afterward. Keep code names unique across the whole codebook.

The codebook CSV export and the .qdpx export both write a codebook-backed scheme's codes as a flat list in 2.13.1. The CSV's parent and color columns are empty for those codes. A code hierarchy needed in another program has to be rebuilt there, or recorded in the code names.

Effect of code changes on saved codings

Potato saves each coding with the code's name as the label, in the coder's user_state.json. A merge or split never deletes a row from the database. Potato applies each codebook change to the codebook alone and never rewrites a saved annotation:

• Adding a code leaves saved codings unchanged.
• Renaming a code changes the form from then on. Codings saved under the old name keep the old name, and exports and agreement statistics count those codings under the old name. Rename before a coding round starts, or recode the affected excerpts.
• Deleting a code removes the code and every code under the code from the form. Saved codings keep the name, and the change log keeps the deleted code's name.
• Merging archives the source code. An archived code leaves the form, and the code's row and history stay in the database. Codings saved with the source code keep the source code's name.
• Splitting by coder names one coder and a new or existing target code. The split is meant for two coders who used one code name for different things, and is designed to move only the named coder's uses. In 2.13.1 the annotation form does not record the per-coder links a split moves. A split therefore creates the target code and archives the source code for every coder whenever the source code has no child codes. Saved codings keep the name each coder chose.

Potato tells coders about changes in two ways. First, a banner on an excerpt saved before new codes were added names the new codes. Second, the tray's Review list holds every excerpt the coder saved before the latest change of any kind. Each excerpt in the list has a Go button. Both signals can be dismissed, and Potato never forces a recode. An admin or adjudicator sees the same list for the whole project at GET /api/codebook/admin/stale.

Definition edits have a separate counter. An edit to a code's short definition, definition, "Use when", or "Avoid when" field raises a semantic revision, and Potato records the semantic revision on every annotation saved afterward. An analyst can then separate codings made before and after a definition changed. Ticking Minor edit (no re-review) in the /codebook editor saves the wording without raising the semantic revision. The Review list in 2.13.1 tracks structural changes only.

Set codebook_mode: fixed for a reliability round, because Potato computes agreement over saved code names and a code renamed mid-round becomes two codes. Inter-Annotator Agreement covers the coefficients. The agreement timeline on /admin/iaa marks each codebook revision against agreement over time. Agreement Drift explains how to read the timeline.

Typed fields in each code's entry

Each code's entry is a list of typed fields. /codebook is the full-page editor, with a version history for every code, a diff between versions, and a restore that saves an older version as a new version. A "Paste markdown" button turns pasted codebook prose into fields by heading and asks the user to choose a type for any section the parser does not recognize. Two people saving the same code at once do not overwrite each other. The second save fails and returns the current text to reconcile against.

The codebook as a whole has four sections outside the codes, titled Preamble, General instructions, Background, and How codes are used downstream. The tray shows each code's short definition, clamped to two lines. When a code has no short definition, the tray shows the full definition instead.

The distiller turns the chosen fields into text that Potato adds to ICL labeling prompts, and rebuilds the text after every codebook change. GET /api/codebook/distilled returns the text a model currently receives. Add a codebook.distiller block to the config above to choose the fields:

codebook:
  distiller:
    include: [definition, use_when, avoid_when, example]
    procedure: concat

Seeding and migrating with the CLI

potato codebook adds a code for every label of every codebook: true scheme that the codebook does not have yet, and leaves existing codes alone:

potato codebook config.yaml

The command prints how many codes were created and how many were already present. Codes created by the CLI get the same ID on every machine, because the CLI computes each ID from the project name, the parent, and the code name. Run the CLI before the first server start when IDs need to match across copies of a project, because codes seeded by the server get random IDs. The CLI copies names only. Colors and definitions from object-form labels reach the codebook only through the server's first-start seed.

Because the server seeds only an empty codebook, the CLI is the way to add new YAML labels to a codebook that already has codes. To preview without writing, run the module directly. In 2.13.1, potato codebook --dry-run drops the flag and writes the codes.

python -m potato.codebook_cli config.yaml --dry-run

API

All routes sit under /api/codebook and need a logged-in session. Admin routes accept an admin or an adjudicator.

The definition editor uses GET /api/codebook/document, GET and PUT /api/codebook/blocks, POST /api/codebook/parse, GET /api/codebook/history, GET /api/codebook/history/<id>, POST /api/codebook/restore, and GET /api/codebook/distilled. Under QDA Mode, /qda/codebook returns a read-only view of the codebook.

Codebook export and change history

The codebook export writes codebook.csv, one row per code per codebook scheme, with the columns schema, annotation_type, code, parent, description, color, and n_uses. The export reads the codebook database. Codes added during coding are included, and merged codes are left out. description holds the code's short definition, or the full definition when the code has no short definition. n_uses counts saved codings by name. A coding saved under a name the codebook no longer has counts toward no row.

python -m potato.export -c config.yaml --format codebook -o ./export/

The quotation_report export lists every coded passage with the passage's code. The qdpx export hands the codes and codings to another QDA program. QDA Mode covers both exports, and Export Formats lists the columns.

The history stays in project.sqlite. The codebook_change table logs every rename, recolor, move, delete, merge, split, and confirmed or rejected proposal, with the old and new values, the actor, whether the actor was a human or a model, and the revision. In 2.13.1, a rename, recolor, move, or delete made through PATCH or DELETE is logged with the actor system, and code additions appear in the codes table with the creator's username instead of in the log. GET /api/codebook/admin/changes returns the change log as JSON.

codebook_snapshot keeps every saved version of every definition, and codebook_revision_history records when each revision happened. A project snapshot and the archive from potato deploy pull both include project.sqlite. Either bundle is an audit trail of the codebook for a methods section.

Run the bundled example from a Potato checkout:

python potato/flask_server.py start examples/advanced/codebook-example/config.yaml -p 8000

The example codes five patient comments about clinic access with an open multiselect codebook of four themes. examples/advanced/codebook-document-example/ adds a distiller block. Open /codebook in that example to see the definition editor.

Further reading

• QDA Mode, for the codebook together with memos, cases, and search
• Memos, for analytic notes beside the codes
• Developing a Qualitative Codebook, on writing definitions and revising codes as a team
• Intercoder Reliability in Qualitative Research, on when to report agreement and which coefficient to use
• Agreement Drift, for agreement over time against codebook revisions

References

MacQueen, K. M., McLellan, E., Kay, K., & Milstein, B. (1998). Codebook Development for Team-Based Qualitative Analysis. CAM Journal, 10(2), 31-36. https://doi.org/10.1177/1525822X980100020301

For implementation details, see the source documentation.