Quantitative Content Analysis in Potato: Codebooks and Krippendorff's Alpha

Code news, posts or speeches against a codebook loaded from a CSV in Potato, assign several coders per unit, and export Krippendorff's alpha with bootstrap intervals.

A quantitative content analysis codes units such as articles, posts, speeches or headlines against a codebook, with at least two coders on a sample, and reports intercoder reliability per variable. In Potato 2.12.0 the codebook can load from a CSV, JSON or YAML file, and the reliability export writes Krippendorff's alpha per question with a bootstrap interval.

Content analysis in this sense is the social-science method of Krippendorff and others: a codebook defines the variables and their categories, coders apply it to units, and a chance-corrected agreement coefficient on a sample coded twice shows whether the codebook can be applied reliably. In Potato:

the codebook is a hierarchical_multiselect question loaded from a file (taxonomy_file), or ordinary radio and multiselect questions;
max_annotations_per_item: 2 or more gives each unit several coders;
--format reliability writes the reliability table.

Load a codebook from a file

yaml

annotation_schemes:
  - annotation_type: hierarchical_multiselect
    name: topic
    description: Which topics does the headline cover?
    taxonomy_file: codebooks/topics.csv
    show_codes: true
    show_search: true

csv

code,label,parent,description
1,Economy,,"Taxes, jobs, prices and the budget"
101,Taxes,1,
102,Jobs and wages,1,
2,Environment,,
201,Climate,2,Climate change and emissions

Each row is a code: code, label, parent (the parent's code, empty at the top level) and an optional description, shown as the code's tooltip. Order is kept. show_codes: true shows 101 Taxes instead of Taxes. A JSON or YAML file holding the taxonomy itself also works. Hierarchical Multiselect covers the rest of the scheme's options.

Published codebooks are not bundled. The Comparative Agendas Project topics and the Manifesto Project categories can be written in this format for your own use, but Potato does not ship them: the Comparative Agendas Project codebook is licensed CC BY-NC-SA, and the Manifesto Project's terms do not allow redistribution. Supply your own copy.

Assign two or more coders per unit

max_annotations_per_item sets how many coders see each unit. A common design has every coder code a shared reliability sample and the rest of the units coded once. How Many Annotators? and Task Assignment cover the assignment strategies, and /admin/iaa shows agreement while coding is in progress.

Export reliability

bash

python -m potato.export --config config.yaml --format reliability --output ./export/

The export writes reliability.csv and reliability.md (the same table in Markdown), one row per question:

Column	Meaning
`level`	Level of measurement: nominal (radio, select), ordinal (likert), interval (number, slider, VAS), or MASI for multiselect and hierarchical multiselect, with partial credit for overlapping sets
`alpha`	Krippendorff's alpha over the units coded by two or more coders
`ci95_low`, `ci95_high`	A percentile bootstrap interval, resampling units with replacement (`bootstrap` resamples, default 1000, seeded)
`percent_agreement`	Share of coder pairs on a unit that gave the same answer
`units`, `coders`, `pairable_values`	What the coefficient rests on

The bootstrap resamples units, and it is not the coincidence-based bootstrap of Hayes and Krippendorff (2007). Krippendorff (2004) suggests relying on variables with an alpha of at least .800 and drawing tentative conclusions down to .667. Potato reports the numbers and leaves that judgment to you.

Adjudication resolves the disagreements once reliability is established, and Inter-Annotator Agreement explains the coefficients.

Example