# Quantitative Content Analysis in Potato: Codebooks and Krippendorff's Alpha

Source: https://www.potatoannotator.com/docs/guides/content-analysis

**A quantitative content analysis codes units such as articles, posts, speeches or headlines against a codebook, with at least two coders on a sample, and reports intercoder reliability per variable. In Potato 2.12.0 the codebook can load from a CSV, JSON or YAML file, and the `reliability` export writes Krippendorff's alpha per question with a bootstrap interval.**

[Content analysis](https://en.wikipedia.org/wiki/Content_analysis) in this sense is the social-science method of Krippendorff and others: a codebook defines the variables and their categories, coders apply it to units, and a chance-corrected agreement coefficient on a sample coded twice shows whether the codebook can be applied reliably. In Potato:

- the codebook is a `hierarchical_multiselect` question loaded from a file (`taxonomy_file`), or ordinary `radio` and `multiselect` questions;
- `max_annotations_per_item: 2` or more gives each unit several coders;
- `--format reliability` writes the reliability table.

## Load a codebook from a file

```yaml
annotation_schemes:
  - annotation_type: hierarchical_multiselect
    name: topic
    description: Which topics does the headline cover?
    taxonomy_file: codebooks/topics.csv
    show_codes: true
    show_search: true
```

```csv
code,label,parent,description
1,Economy,,"Taxes, jobs, prices and the budget"
101,Taxes,1,
102,Jobs and wages,1,
2,Environment,,
201,Climate,2,Climate change and emissions
```

Each row is a code: `code`, `label`, `parent` (the parent's code, empty at the top level) and an optional `description`, shown as the code's tooltip. Order is kept. `show_codes: true` shows `101 Taxes` instead of `Taxes`. A JSON or YAML file holding the taxonomy itself also works. [Hierarchical Multiselect](/docs/annotation-types/hierarchical-multiselect) covers the rest of the scheme's options.

**Published codebooks are not bundled.** The [Comparative Agendas Project](https://www.comparativeagendas.net/) topics and the [Manifesto Project](https://manifesto-project.wzb.eu/) categories can be written in this format for your own use, but Potato does not ship them: the Comparative Agendas Project codebook is licensed CC BY-NC-SA, and the Manifesto Project's terms do not allow redistribution. Supply your own copy.

## Assign two or more coders per unit

`max_annotations_per_item` sets how many coders see each unit. A common design has every coder code a shared reliability sample and the rest of the units coded once. [How Many Annotators?](/docs/guides/how-many-annotators) and [Task Assignment](/docs/features/task-assignment) cover the assignment strategies, and `/admin/iaa` shows agreement while coding is in progress.

## Export reliability

```bash
python -m potato.export --config config.yaml --format reliability --output ./export/
```

The export writes `reliability.csv` and `reliability.md` (the same table in Markdown), one row per question:

| Column | Meaning |
|---|---|
| `level` | Level of measurement: nominal (radio, select), ordinal (likert), interval (number, slider, VAS), or MASI for multiselect and hierarchical multiselect, with partial credit for overlapping sets |
| `alpha` | [Krippendorff's alpha](https://en.wikipedia.org/wiki/Krippendorff%27s_alpha) over the units coded by two or more coders |
| `ci95_low`, `ci95_high` | A percentile bootstrap interval, resampling units with replacement (`bootstrap` resamples, default 1000, seeded) |
| `percent_agreement` | Share of coder pairs on a unit that gave the same answer |
| `units`, `coders`, `pairable_values` | What the coefficient rests on |

The bootstrap resamples units, and it is not the coincidence-based bootstrap of Hayes and Krippendorff (2007). Krippendorff (2004) suggests relying on variables with an alpha of at least .800 and drawing tentative conclusions down to .667. Potato reports the numbers and leaves that judgment to you.

[Adjudication](/docs/guides/adjudication-and-disagreement) resolves the disagreements once reliability is established, and [Inter-Annotator Agreement](/docs/guides/inter-annotator-agreement) explains the coefficients.

## Example

```bash
python potato/flask_server.py start examples/classification/codebook-content-analysis/config.yaml -p 8000
```

## Further Reading

- [Hierarchical Multiselect](/docs/annotation-types/hierarchical-multiselect) - Tree-structured codebooks
- [Inter-Annotator Agreement](/docs/guides/inter-annotator-agreement) - Choosing and reading an agreement coefficient
- [Writing Annotation Guidelines](/docs/guides/writing-annotation-guidelines) - Turning a codebook into coder instructions
- [Adjudication and Disagreement](/docs/guides/adjudication-and-disagreement) - Resolving coder disagreements

For implementation details, see the [source documentation](https://github.com/davidjurgens/potato/blob/master/docs/guides/content-analysis.md).