Intercoder Reliability in Qualitative Research

Intercoder reliability (ICR) is a chance-corrected measure of how consistently two or more coders apply one codebook to the same excerpts. Qualitative studies that use content analysis or a fixed codebook usually report Krippendorff's alpha or Cohen's kappa on a double-coded sample of 10 to 25% of the data, with α ≥ 0.800 as the conventional standard and 0.667 as the floor for tentative conclusions. Reflexive thematic analysis leaves the coefficient out, because the method holds that coding is an interpretive act with no accurate answer for a second coder to match.

Intercoder reliability is the agreement between coders who code the same material independently, measured with a coefficient that subtracts the agreement expected by chance. A codebook, also called a coding frame, lists the codes with a definition, inclusion and exclusion rules, and examples for each code. Percent agreement is the share of excerpts on which coders gave the same codes, with no correction for chance. Cohen's kappa compares two coders, and Fleiss' kappa compares a fixed number of coders on every excerpt. Krippendorff's alpha handles any number of coders, missing codings, and nominal, ordinal, or interval codes. Thematic analysis is a family of methods for developing themes across a dataset, and the members of the family disagree about ICR.

Inter-Annotator Agreement Explained covers the arithmetic of the coefficients. The codebook development guide covers writing the codes. The tutorial Code Interview Transcripts With Two Coders and Report Alpha runs a full project in Potato from excerpts to alpha, and Qualitative research in Potato lists Potato's qualitative features in one place.

When qualitative studies report intercoder reliability

Content analysts present ICR as a requirement of the method, and O'Connor and Joffe (2020) apply ICR in thematic analysis as well. A coefficient earns a place wherever several coders split the material, because the coefficient shows that each coder applies the codebook the same way before coding alone. A study that counts how many interviews contain each code needs ICR for the same reason. Inside the team, coders who compute a coefficient for each code can see which codes the coders read differently, and O'Connor and Joffe judge the discussion of those codes often worth more than the final score. Reviewers in some fields ask for ICR, and the COREQ reporting checklist asks for the number of coders.

When the method argues against a coefficient

Reflexive thematic analysis (TA) treats the researcher's subjectivity as a resource and coding as an interpretive act. Braun and Clarke (2023) reject the idea that coding can be accurate, and Braun and Clarke call consensus coding and measuring intercoder agreement "conceptually incoherent" additions to a reflexive study. Braun and Clarke separate reflexive TA from two other kinds of thematic analysis. Coding reliability TA uses independent coders and an agreement coefficient, and codebook approaches such as framework analysis borrow some of the structured procedures of coding reliability TA. Some grounded theory researchers reject ICR as well, because grounded theory builds codes recursively as the analysis proceeds. A reflexive TA write-up describes the researcher's reflexivity and theoretical assumptions in place of a coefficient.

O'Connor and Joffe answer the epistemological objection by reading a high coefficient as evidence that researchers who share a conceptual framework reach a consensual interpretation of the data. The two authors see a larger risk in false precision, because a high figure can make a weak analysis look rigorous. Choose one position per study and state the position in the methods section.

Which coefficient fits which coding design

The coefficient follows from two decisions made before coding. The first decision is the unit, either fixed excerpts that every coder sees or segments that each coder marks. The second decision is whether an excerpt takes one code or several.

Several codes per excerpt is common in interview coding. O'Connor and Joffe score each code as 1 or 0 on every excerpt for each coder and compute the coefficient code by code. The per-code values show which definitions need work, whereas a single pooled figure lets reliable codes hide unreliable ones. Krippendorff (2004) makes the same argument for content analysis and takes the smallest value among the variables as the reliability of the whole.

Segment boundaries are the harder problem in interview coding. Coders who each pick the quotes to code split and lump differently, and automated comparisons can treat two quotes that differ by one punctuation mark as different units. A second Krippendorff paper from 2004 extends alpha to units that coders identify within a continuum of text. A simpler route is to fix the segments first. The first coder segments and codes the transcript, and the second coder receives the same segments with the codes removed. ICR then measures code applications on fixed excerpts and leaves boundary agreement unmeasured.

Conventional thresholds and where each comes from

Krippendorff (2004) suggests requiring α ≥ 0.800, and α ≥ 0.667 where tentative conclusions are still acceptable. The same paper says that apart from perfect agreement "there are no magical numbers", ties the acceptable level to the cost of drawing invalid conclusions, and asks researchers to consult confidence intervals. For kappa, O'Connor and Joffe summarize the bands of Landis and Koch (1977) as 0.41 to 0.60 moderate, 0.61 to 0.80 substantial, and 0.81 to 1 nearly perfect. McHugh (2012) considers Cohen's original interpretation too lenient for health research and calls a kappa of 0.60 to 0.79 moderate and 0.80 to 0.90 strong. O'Connor and Joffe also cite Miles and Huberman's standard for percent agreement, 80% agreement on 95% of codes. O'Connor and Joffe advise fixing the threshold before coding starts, with a higher bar for findings that inform medical, policy, or financial decisions.

Size of the double-coded sample

O'Connor and Joffe call 10 to 25% of the data units typical, drawn at random or by a stated rule such as one case from each stratum. Before the formal sample, both coders code one interview informally and compare the codings to catch misread definitions. After a revision, the next round uses a different subset, and the excerpts coded during the reliability rounds are recoded with the final codebook. A coder who joins later needs a separate reliability check. Report the number of excerpts and coders beside every coefficient, because a coefficient from a small sample has a wide interval.

From disagreements to codebook revision

Coders code the reliability sample independently and then meet to go through each disagreement. The team rewrites unclear definitions, adds exclusion rules and examples, and merges codes that coders cannot tell apart. The coders then test the revised codebook in a second independent round on fresh excerpts. O'Connor and Joffe warn that repeated rounds with the same coders can produce "interpretive convergence" between those coders without making the codebook clearer to a new reader. Report the coefficient from the independent coding, before the coders discuss any disagreement. The final coded data then come from consensus, a third coder's vote, or the senior researcher's decision, and the methods section names the rule.

A reliability round in Potato

The config below sets up a reliability round on interview excerpts about access to a health clinic. Each excerpt outside the sample goes to one coder, and two coders code a 20% sample drawn within each participant. With the codebook fixed for the round, no coder can add a code mid-round. Each code's definition appears as the code's tooltip:

annotation_task_name: Clinic access interviews, reliability round 1
task_dir: .
output_annotation_dir: annotation_output/
data_files:
  - data/excerpts.jsonl
item_properties:
  id_key: id
  text_key: text
user_config:
  allow_all_users: true
  users: []
 
# Two coders on a 20% sample drawn within each participant, one coder elsewhere
num_annotators_per_item:
  default: 1
  overlap_sample:
    fraction: 0.2
    count: 2
    stratify_by: participant_id
    seed: 7
assignment_strategy: random
 
# Coders apply the agreed codes and cannot add new ones during the round
codebook_mode: fixed
 
annotation_schemes:
  - annotation_type: multiselect
    name: codes
    description: Which codes apply to this excerpt? Hover over a code for its definition.
    humanize_labels: false
    labels:
      - {name: Access barriers, tooltip: "Distance, transport, opening hours, or waiting time keep the participant from care"}
      - {name: Cost concerns, tooltip: "Fees, insurance, or lost wages shape whether the participant seeks care"}
      - {name: Provider trust, tooltip: "Confidence or distrust in clinic staff, their competence, or their motives"}
      - {name: Administrative burden, tooltip: "Forms, referrals, phone queues, or eligibility rules the participant has to work through"}
      - {name: No code applies, tooltip: "The excerpt touches none of the codes above"}
 
adjudication:
  enabled: true
  adjudicator_users: [lead]
  min_annotations: 2

Each line of data/excerpts.jsonl has id, text, and participant_id. The "No code applies" code gives every excerpt an answer. Potato's reliability export drops an excerpt on which a coder chose no code, and without that code a split between one theme and no theme would never count against alpha. The two-coder tutorial explains the random assignment. A team that built the codebook in QDA Mode can also keep qda_mode on for the round with codebook_mode: fixed. Under QDA Mode, Potato skips the check that rejects search.annotator_claim alongside overlap. Leave annotator_claim off while the sample is being coded.

When both coders have finished, export the reliability table and freeze the round:

python -m potato.export --config config.yaml --format reliability --output export/
potato snapshot create config.yaml --tag icr-round-1 --note "Reliability round 1"

The export writes reliability.csv and reliability.md, with one row per question. For the codes question, the row gives Krippendorff's alpha with the MASI distance over each excerpt's set of codes, a 95% percentile bootstrap interval over excerpts, percent agreement on identical sets, and the numbers of excerpts and coders. The snapshot stores the round's config, data, codings, and alpha per question under the tag, as Project Snapshots describes. The lead then opens /adjudicate, settles each disagreement, and ticks "Flag for guideline update" wherever a definition caused the split. The adjudication export writes those flags to adjudication_log.jsonl. The flagged excerpts set the agenda for the codebook revision before the next round.

Potato's two agreement reports cover the scheme types a qualitative coder uses as follows:

The dashboard scores an excerpt once every coder assigned to the excerpt has finished, and Agreement Drift shows the same coefficients window by window during the main coding. Four gaps remain in Potato 2.13.1:

• No alpha per code. Both reports give one value per question, and the two-coder tutorial has a script that computes alpha for each code from the CSV export.
• No unitizing alpha. The character-level alpha for span questions treats every character as a fixed unit, and Agreement for Spans explains the difference.
• No other indices. Potato computes no Scott's pi, Holsti index, Gwet's AC1, or Brennan and Prediger's kappa.
• No threshold judgment. The export reports values without a label, and the dashboard badges any value from 0.6 to 0.8 "substantial". That band starts below Krippendorff's 0.667. Compare each value with the threshold the study fixed in advance.

Frequently asked questions

Should I use Krippendorff's alpha or Cohen's kappa for intercoder reliability?

Use Krippendorff's alpha when more than two coders take part, when coders coded different but overlapping excerpts, or when the codes are ordered, and either coefficient when two coders applied unordered codes to the same excerpts. On complete nominal data from two coders the two values usually land close together. The values part when the coders use a code at different rates, because kappa computes chance agreement from each coder's own code frequencies and alpha from the pooled frequencies.

What Krippendorff's alpha is acceptable for intercoder reliability?

An alpha of 0.800 or higher is the conventional standard, and 0.667 is the lowest value Krippendorff (2004) accepts for tentative conclusions. Krippendorff ties the cut-off to the cost of a wrong conclusion and sets far higher criteria when human lives hang on the result. Apply the threshold to each code rather than to an average across codes, and report the bootstrap interval beside the point value.

What Cohen's kappa is good enough for intercoder agreement?

A kappa from 0.81 to 0.90 counts as nearly perfect under Landis and Koch's (1977) bands and as strong under McHugh's (2012) stricter scale. McHugh reserves almost perfect for a kappa above 0.90. Landis and Koch call 0.61 to 0.80 substantial, while McHugh calls 0.60 to 0.79 only moderate for health research. Fix the bar before coding, and report kappa for each code.

Should I report percent agreement alongside Krippendorff's alpha?

Yes, as a descriptive figure beside alpha, with alpha as the reliability claim. Percent agreement ignores chance. Two coders who rarely apply a code agree on most excerpts even when the coders never agree on where the code applies. Krippendorff (2004) rejects percent agreement as a reliability measure, and McHugh (2012) advises calculating both percent agreement and kappa. Potato's reliability export writes both columns for every question.

Further reading

• Qualitative research in Potato, for whether to report ICR at all and Potato's qualitative features
• Qualitative Data Analysis Software Compared, for the coefficients each QDA program reports
• Code Interview Transcripts With Two Coders and Report Alpha, for a full coding project with per-code alpha
• Coding interview transcripts, for the steps from recordings to coded excerpts
• LLM-assisted qualitative coding, for a model as a second coder
• Shared Codebook for Qualitative Coding, for codebook modes and revision history
• Adjudication and Disagreement, for settling disagreements
• Agreement calculator, for alpha and kappa from a CSV of codes exported by any tool

References

1. Braun, V., & Clarke, V. (2023). Toward good practice in thematic analysis: Avoiding common problems and be(com)ing a knowing researcher. International Journal of Transgender Health, 24(1), 1-6. https://doi.org/10.1080/26895269.2022.2129597
2. Krippendorff, K. (2004). Reliability in content analysis: Some common misconceptions and recommendations. Human Communication Research, 30(3), 411-433. https://doi.org/10.1111/j.1468-2958.2004.tb00738.x
3. Krippendorff, K. (2004). Measuring the reliability of qualitative text analysis data. Quality & Quantity, 38(6), 787-800. https://doi.org/10.1007/s11135-004-8107-7
4. Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1). https://doi.org/10.2307/2529310
5. McHugh, M. L. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276-282. https://doi.org/10.11613/BM.2012.031
6. O'Connor, C., & Joffe, H. (2020). Intercoder reliability in qualitative research: Debates and practical guidelines. International Journal of Qualitative Methods, 19, 1-13. https://doi.org/10.1177/1609406919899220