# Open-Source Qualitative Coding with Intercoder Reliability

Source: https://www.potatoannotator.com/use-cases/qualitative-research

**Potato codes interviews, documents, and open-ended answers with a living codebook, in-vivo coding, memos, and cases, and reports intercoder reliability as Krippendorff's alpha when two or more coders share the units. Potato is free and open source, runs on your own computer or server, and exchanges projects with NVivo, ATLAS.ti, MAXQDA, Quirkos, and QDA Miner as REFI-QDA `.qdpx` files.**

Qualitative coding has two common shapes, and Potato supports both. In interpretive work such as thematic analysis, an analyst reads the whole corpus, grows the codebook while reading, and writes memos, and QDA Mode sets Potato up for that. In quantitative content analysis, a fixed codebook is applied to units by several coders, and a chance-corrected agreement coefficient on a sample coded twice shows whether the codebook can be applied reliably. Potato assigns the units, keeps each coder's answers separate, and the `reliability` export writes Krippendorff's alpha per question with a bootstrap interval.

## Qualitative research tasks in Potato

Each row is a feature that ships with Potato, and the linked page has its configuration.

| Task | How Potato does it | Docs |
|---|---|---|
| Open coding with an evolving codebook | QDA Mode's living codebook, where codes are added, renamed, recolored, moved, or deleted while you code | [QDA Mode](/docs/qda/qda-mode) |
| In-vivo coding | Select a passage and press `i` to create a code from its words, with near-duplicate codes flagged | [QDA Mode](/docs/qda/qda-mode) |
| Memos and cases | Analytic notes on a passage or document, and cases that group excerpts by participant or document | [QDA Mode](/docs/qda/qda-mode) |
| Content analysis with a fixed codebook | A codebook loaded from CSV, JSON or YAML as a `hierarchical_multiselect` question | [Content analysis](/docs/guides/content-analysis) |
| Intercoder reliability | Two or more coders per unit, and Krippendorff's alpha per question in the `reliability` export | [Content analysis](/docs/guides/content-analysis) |
| Coding open-ended survey answers | Responses loaded as items from a CSV or JSON file and coded with the codebook | [Survey research](/use-cases/survey-research) |
| Quotation reports | One row per coded span with the quote, its offsets, the source, and the coder | [QDA Mode](/docs/qda/qda-mode#exporting-your-coding) |

## Exchanging projects with NVivo, ATLAS.ti, and MAXQDA

Since Potato 2.9, a project can come in from and go back out to the REFI-QDA exchange format, which NVivo, ATLAS.ti, MAXQDA, Quirkos and QDA Miner all read and write. The codebook keeps its hierarchy, along with the codings and which analyst made each one.

```bash
potato import -i project.qdpx -o my-study
python -m potato.export -c my-study/config.yaml -f qdpx -o ./export/
```

ATLAS.ti keeps only one level of subcode, so for a deeper codebook add `--option flatten_subcodes=true` to the export, which moves every code to the top level and renames each one `Parent > Child`.

## Working alongside NVivo, ATLAS.ti, MAXQDA, and Taguette

[NVivo](https://en.wikipedia.org/wiki/NVivo), ATLAS.ti, and MAXQDA are established desktop programs for computer-assisted qualitative data analysis, with paid licenses, and [Taguette](https://www.taguette.org/) is a free and open-source tool for qualitative research. Potato is web-based and free, so a team codes in a browser without installing anything, on a server the team controls. Potato treats team coding as a measured process, with assignment of units to coders, independent coding, reliability reported per question, and adjudication of disagreements. The same project can include surveys or annotation tasks.

## Frequently asked questions

### Is there a free, open-source alternative to NVivo or ATLAS.ti for team coding?

Potato is one. Potato is free and open source under the GPL-3.0 license, runs in a browser from your own server, and supports a codebook, memos, cases, in-vivo coding and intercoder reliability. Because Potato reads and writes REFI-QDA `.qdpx` files, a project can move between Potato and NVivo, ATLAS.ti or MAXQDA when part of a team uses those programs.

### How do I calculate Krippendorff's alpha between coders?

Assign each unit to at least two coders with `num_annotators_per_item`, and run the `reliability` export, which writes Krippendorff's alpha for each question with a bootstrap interval, percent agreement, and the number of units and coders behind each value. For data coded elsewhere, the [agreement calculator](/tools/inter-annotator-agreement) computes alpha and kappa from a CSV in your browser.

### Can I import a project from NVivo, ATLAS.ti, or MAXQDA?

Yes, through the REFI-QDA exchange format. Export the project from your QDA program as a `.qdpx` file and run `potato import` on the file, and the codebook, codings, and analyst identities come across.

### Should I report intercoder reliability in qualitative research?

The answer depends on the approach. Content analysis treats reliability as a requirement, while some interpretive traditions question whether reliability fits their aims. [O'Connor and Joffe (2020)](https://doi.org/10.1177/1609406919899220) review the debate and give practical guidelines for when and how to report reliability.

## Further reading

- [Qualitative data analysis software compared](/docs/guides/qualitative-coding-software-compared), for NVivo, ATLAS.ti, MAXQDA, and the free alternatives
- [Writing annotation guidelines](/docs/guides/writing-annotation-guidelines), for the codebook as a document coders work from
- [Adjudication and disagreement](/docs/guides/adjudication-and-disagreement), for reconciling coders' answers
- [Survey research](/use-cases/survey-research) and [digital humanities](/use-cases/digital-humanities), for neighboring fields on this site

## References

1. O'Connor and Joffe (2020). Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines. *International Journal of Qualitative Methods* 19. https://doi.org/10.1177/1609406919899220