Code Interview Transcripts With Two Coders and Report Alpha

Qualitative and mixed-methods researchers who code interviews as a team can follow this tutorial to have two coders code six openly licensed interview transcripts in Potato 2.13.1, report Krippendorff's alpha in a methods section, settle the disagreements, and hand the coded project to colleagues who work in NVivo, ATLAS.ti, or MAXQDA. The tutorial assumes a codebook already exists. Potato's QDA Mode suits the earlier stage of one analyst reading the interviews and growing the codebook, and the QDA Mode post walks through that stage. Qualitative data analysis software compared covers prices and features across tools, and Qualitative research in Potato lists Potato's qualitative features in one place.

The interviews

The transcripts come from What makes research software sustainable? Anonymized interview transcripts, deposited on Zenodo by Mário Rosado de Souza, Robert Haines, Markel Vigo, and Caroline Jay (reference 2). The deposit has 19 interviews with developers of research software, one PDF per interview, under the CC BY-SA 4.0 license. The study behind the interviews appeared at CHASE 2019 (reference 1). Any file you derive from the transcripts and share, such as the excerpt file below, falls under the same license, with attribution to the four authors. This tutorial uses the first six interviews. In all six, the interviewer asks what sustainable software is, which features make software sustainable, and what happens when software is not sustained. Download Interview01.pdf to Interview06.pdf from the Zenodo record.

Reliability needs fixed units, with both coders answering the same question about the same piece of text. The unit here is one participant answer. The script below reads the transcripts after pdftotext (from Poppler) has turned each PDF into text, keeps every participant answer of 20 words or more, and records the interviewer's last question beside each answer:

"""Split interview transcripts into one excerpt per participant answer.
 
Usage: python prepare_excerpts.py transcripts/*.txt > data/excerpts.jsonl
"""
import json
import re
import sys
from pathlib import Path
 
TURN = re.compile(r"^\s*(I\d*|P\d*)\s*[-:]\s*(.*)$", re.IGNORECASE)
MIN_WORDS = 20  # shorter answers ("Yeah", "Right") are not coded
 
for path in sys.argv[1:]:
    interview = Path(path).stem.replace("Interview", "I").replace("interview", "I")
    turns, speaker, words = [], None, []
    for line in Path(path).read_text(encoding="utf-8").splitlines():
        line = re.sub(r"!\s*$", "", line).strip()  # the PDFs end some lines with "!"
        match = TURN.match(line)
        if match:
            if speaker:
                turns.append((speaker, " ".join(words)))
            speaker, words = match.group(1).upper()[0], [match.group(2)]
        elif line and speaker:
            words.append(line)
    if speaker:
        turns.append((speaker, " ".join(words)))
 
    question, n = "", 0
    for speaker, text in turns:
        text = re.sub(r"\s+", " ", text).strip()
        if speaker == "I":
            if "?" in text:  # keep the last question, skipping "Mm" and "I see"
                question = text
        elif len(text.split()) >= MIN_WORDS:
            n += 1
            print(json.dumps({"id": f"{interview}-{n:02d}", "interview": interview,
                              "question": question, "text": text}, ensure_ascii=False))

Install Potato and run the script on the six transcripts:

pip install potato-annotation==2.13.1
mkdir -p transcripts data
for f in Interview0[1-6].pdf; do pdftotext -layout "$f" "transcripts/${f%.pdf}.txt"; done
python prepare_excerpts.py transcripts/*.txt > data/excerpts.jsonl

The six interviews give 52 excerpts, from 5 in the shortest interview to 15 in the longest. Each line of data/excerpts.jsonl is one excerpt, shown here with the answer shortened:

{"id": "I01-07", "interview": "I01", "question": "Okay. Um, regarding the software you developed, was sustainability a consideration?", "text": "Sustainability most certainly was a consideration. ..."}

The transcripts label speakers in two styles, I1: and I1 -, and the script reads both. Interview 1 labels the interviewer's opening question as a participant turn, and the first four excerpts of that interview have no question beside the answer. Read a few excerpts from each interview against the PDF before coding starts.

The codebook

The codebook has ten codes, written after reading the six interviews. Nine codes name a theme the participants raise, and a tenth code, "No code applies", gives every excerpt an answer. Potato's reliability export leaves out any excerpt on which a coder chose nothing. Without the tenth code, an excerpt where one coder saw a theme and the other coder saw none would drop out of alpha instead of counting as a disagreement. Each code's definition is the code's tooltip, and a coder sees the definition by hovering over the code.

A multiselect question records the codebook, because one answer often raises two themes. Potato can also load a codebook from a CSV file into a hierarchical_multiselect question (Content Analysis), and that question type suits a nested codebook with a hundred codes. In 2.13.1 the hierarchical question type has one gap for this tutorial's workflow. The adjudication page lets the adjudicator adopt one coder's answer but not combine the two. In 2.12.1 the codebook and qdpx exports also left out the question's codes.

The config

The config below is complete and passes python -m potato.validate_cli config.yaml --strict on 2.12.1 and on 2.13.1:

annotation_task_name: Research software sustainability interviews
task_dir: .
output_annotation_dir: annotation_output/
data_files:
  - data/excerpts.jsonl
item_properties:
  id_key: id
  text_key: text
 
user_config:
  allow_all_users: true
  users: []
 
# Both coders code every excerpt independently
num_annotators_per_item: 2
 
instance_display:
  fields:
    - {key: question, type: text, label: Interviewer}
    - {key: text, type: text, label: Participant, span_target: true}
 
annotation_schemes:
  - annotation_type: multiselect
    name: codes
    description: Which codes apply to this answer? Hover over a code for its definition.
    humanize_labels: false
    labels:
      - {name: Documentation, tooltip: "Manuals, API descriptions, examples, or software that explains itself"}
      - {name: Testing, tooltip: "Automated tests, or the effort and limits of writing them"}
      - {name: Code design, tooltip: "Modular, generic, narrowly scoped, or well-commented code"}
      - {name: Reuse of libraries, tooltip: "Building on maintained libraries instead of bespoke code"}
      - {name: Public source code, tooltip: "Version control, GitHub, or another place where anyone can find the code"}
      - {name: User community, tooltip: "People or organizations who use the software, rely on it, or answer questions about it"}
      - {name: More than one maintainer, tooltip: "Someone other than the original developer can build, maintain, or take over the software"}
      - {name: Funding, tooltip: "Grants, budgets, or money to keep the software maintained, or the lack of them"}
      - {name: Software decay, tooltip: "Software stops working as platforms, libraries, or servers change, or is abandoned"}
      - {name: No code applies, tooltip: "The answer touches none of the codes above"}
  - annotation_type: span
    name: quotes
    description: Highlight the words that show each code you ticked.
    target_field: text
    humanize_labels: false
    labels: [Documentation, Testing, Code design, Reuse of libraries, Public source code, User community, More than one maintainer, Funding, Software decay]
 
adjudication:
  enabled: true
  adjudicator_users: [lead]
  min_annotations: 2

num_annotators_per_item: 2 sends every excerpt to both coders, and Potato shows neither coder the other's codes. instance_display puts the interviewer's question above the answer, and span_target: true lets coders highlight in the answer only. The quotes question collects the words behind each code for the write-up. Potato computes reliability from the ticked codes and ignores the highlights. Without humanize_labels: false, Potato capitalizes every word of a code and displays "Reuse Of Libraries". The adjudication block makes the user lead the adjudicator for every excerpt both coders coded.

On a larger corpus, a common design has both coders code a sample of excerpts and split the other excerpts between the two coders. Replace num_annotators_per_item: 2 in the config above with these lines to have both coders code a 30% sample drawn within each interview:

num_annotators_per_item:
  default: 1
  overlap_sample:
    fraction: 0.3
    count: 2
    stratify_by: interview
    seed: 11
assignment_strategy: random

Keep assignment_strategy: random with two coders. Under Potato's default fixed_order assignment, a test with this sample and two coders assigned all 52 excerpts to the first coder. The second coder then received 26 excerpts outside the sample that the first coder had not yet coded, and those 26 excerpts were coded twice. With random assignment, the two coders coded 34 excerpts each, one more excerpt in total than the 15-excerpt sample and the other 37 excerpts require. These counts come from Potato 2.12.1. Potato 2.13.1 adds assignment_hold_minutes, which gives a coder who runs out of work the excerpts the other coder has not reached, rather than sharing them (Task Assignment).

Coding the excerpts

Start the server from the project folder:

potato start config.yaml -p 8000

Each coder opens http://localhost:8000, registers with a username and password, and starts at the first excerpt. A coder on another computer needs the server at an address that computer can reach, as Deploying Potato on a VPS describes. For each excerpt, the coder ticks every code that applies. The coder then clicks a code under "Highlight the words that show each code you ticked" and drags across the words in the answer. The highlight covers exactly the characters dragged across and does not snap to word boundaries. Clicking Next saves the excerpt.

The test run behind this page

The numbers on this page come from two scripted coders working through a live Potato 2.12.1 server, and the exports and the agreement report were run again on the saved coding with Potato 2.13.1, and saving each excerpt through the same requests the annotation page sends. The first coder's codes came from a careful reading of each answer, with one highlighted phrase per code. The second coder's codes were written independently and differ on uncertain answers, and the second coder highlighted whole sentences. The alpha values therefore describe two simulated coders and say nothing about the reliability of this codebook with real coders. A check in the browser confirmed that a highlight drawn by hand is saved with offsets into the same answer text the scripts used.

Krippendorff's alpha from the reliability export

When both coders have finished, run the reliability export:

python -m potato.export --config config.yaml --format reliability --output export/

The command writes export/reliability.csv and export/reliability.md. The Markdown file is a table to paste into a methods section:

| Variable | Level | Krippendorff's α | 95% CI | % agreement | Units | Coders |
|---|---|---|---|---|---|---|
| codes | masi | 0.677 | [0.58, 0.764] | 53.846 | 52 | 2 |
| quotes | positional | 0.284 | [0.225, 0.346] | 75.404 | 46 | 2 |
 
Intervals: percentile bootstrap over units, 1000 resamples.

The table has one row per question. The quotes row is new in Potato 2.13.1, and the section on highlighted passages below explains the row. Potato 2.12.1 left span questions out of the table. Potato computes Krippendorff's alpha over the set of codes each coder chose for each excerpt, with the MASI distance (reference 3). MASI gives partial credit when the two sets overlap. Two coders who chose {Funding, User community} and {Funding} get more credit than two coders with no code in common. The interval is a percentile bootstrap that resamples excerpts 1,000 times with a fixed seed. The percent agreement of 53.846 counts an excerpt as agreed only when both coders chose the same set of codes, and 28 of the 52 excerpts met that test.

A methods section can report the result in one sentence, such as "Two coders independently coded all 52 excerpts; Krippendorff's alpha with the MASI distance over the set of codes per excerpt was 0.677 (95% bootstrap interval 0.58 to 0.764)." The conventional content-analysis thresholds are 0.800 for relying on a variable and 0.667 for tentative conclusions, and Inter-Annotator Agreement explains how to read the two thresholds.

The admin dashboard shows the same coefficient while coding is in progress. Open http://localhost:8000/admin/iaa and type the admin key that Potato writes to admin_api_key.txt in the project folder. In the test run, the page reported alpha_masi 0.677 and a mean Jaccard of 0.788. Potato 2.13.1 labels a value from 0.6 to 0.8 "substantial" and a value above 0.8 "strong", and Potato 2.12.1 called every value of 0.6 or above "strong agreement". The content-analysis thresholds above set a higher bar.

Alpha for each code

Many qualitative methods sections report reliability per code, with each code scored as present or absent on each excerpt. Potato 2.13.1 does not report per-code alpha, so compute the figure yourself from the CSV export. The script below calls the same alpha function the reliability export uses, with the nominal level:

"""Krippendorff's alpha for each code, scored as present or absent per excerpt.
 
Usage: python alpha_by_code.py export/annotations.csv
"""
import csv
import sys
from collections import defaultdict
 
from potato.server_utils.iaa.alpha import krippendorff_alpha
 
chosen = defaultdict(dict)  # excerpt -> coder -> set of codes
for row in csv.DictReader(open(sys.argv[1], encoding="utf-8")):
    chosen[row["instance_id"]][row["user_id"]] = {
        col[len("codes."):] for col, value in row.items()
        if col.startswith("codes.") and value}
both = {unit: by for unit, by in chosen.items() if len(by) == 2}
codes = sorted(set().union(*(c for by in both.values() for c in by.values())))
 
print("code,alpha,either_coder,both_coders")
for code in codes:
    rows = [(coder, unit, int(code in c)) for unit, by in both.items() for coder, c in by.items()]
    either = sum(any(code in c for c in by.values()) for by in both.values())
    agreed = sum(all(code in c for c in by.values()) for by in both.values())
    print(f"{code},{krippendorff_alpha(rows, level='nominal'):.3f},{either},{agreed}")

Export the CSV and run the script:

python -m potato.export --config config.yaml --format csv --output export/
python alpha_by_code.py export/annotations.csv

The test run gave these values, sorted here by alpha:

"Public source code" falls below 0.667, because one coder applied the code whenever a participant mentioned GitHub. "More than one maintainer" and "User community" both fall below 0.800, and the two codes overlap whenever a participant describes other people keeping software alive. A code applied to five excerpts, such as "Reuse of libraries", moves a long way with one disagreement, so read the per-code values together with the two count columns.

Agreement on highlighted passages

For span questions, Potato 2.13.1's admin page reports token-level kappa, span F1 at exact and partial match, a character-level alpha named positional_alpha, and Mathet's gamma. Over the 46 excerpts that either coder highlighted, the page reported token-level kappa 0.296, exact span F1 0, partial span F1 0.837, positional_alpha 0.183, and gamma 0.575. Exact F1 is 0 because no highlight from one coder starts and ends on the same characters as a highlight from the other. The reliability export's quotes row gives 0.284, because the export pools the characters of every excerpt into one alpha, where the admin page averages alpha over excerpts. In 2.12.1 the span rows read n/a with 0 items because of a lookup error. The span agreement calculator on this site computes span F1 under other match rules, and a character-level alpha, in the browser. The calculator takes one row per highlight with item, annotator, label, start, and end, and the quotation report has each of those fields:

python -m potato.export --config config.yaml --format quotation_report --output export/

The command writes export/quotations.csv, with one row per highlight, the quote, the start and end offsets, the excerpt, and the coder.

positional_alpha, which Potato 2.12.1 named krippendorff_alpha_u, is a character-level alpha and is not Krippendorff's unitizing alpha. Potato gives each character of an answer the set of codes covering the character, runs nominal alpha over the characters, and averages over excerpts. Qualitative coders often give the same words two codes, and from 2.13.1 a character with two codes counts as a different value from a character with one of the two. Potato 2.12.1 kept only the longer highlight's code for the shared characters. Agreement for Spans explains the span measures. For the methods section, report alpha on the excerpt codes and treat the highlights as quotation evidence.

Settling the disagreements

Alpha describes the two coders' independent coding. Report alpha from the coding as it stands before anyone settles a disagreement. To build the final codes, the lead opens http://localhost:8000/adjudicate after registering with the username lead. The queue lists every excerpt where the two coders chose different sets of codes. In the test run, the queue had 24 excerpts. For each excerpt, the page shows both coders' codes and highlights, a checkbox for every code to record the final set, a confidence level, notes, and an error category from Potato's default list. A checkbox also flags the excerpt for a codebook change.

In the test run, the lead tagged 11 disagreements as coder error, 10 as gaps in the codebook, 2 as ambiguous text, and 1 as subjective disagreement. Two excerpts were flagged for codebook changes. One flag narrows "Public source code" to answers that name public code as a condition for sustainability. The other flag asks for a usability code, because participants describe software that is easy to learn and to get running, and no code covers that theme.

Export the decisions:

python -m potato.export --config config.yaml --format adjudication --output export/

The command writes export/adjudicated.csv, with one row per adjudicated excerpt, and export/adjudication_log.jsonl, with the notes, error categories, and codebook flags. adjudicated.csv covers only the 24 excerpts in the queue. A third script combines the adjudicated codes with the codes both coders agreed on:

"""One final set of codes per excerpt: the adjudicated set where the coders
disagreed, and the coders' shared set everywhere else.
 
Usage: python final_codes.py export/annotations.csv export/adjudicated.csv > final_codes.csv
"""
import csv
import json
import sys
from collections import defaultdict
 
chosen = defaultdict(list)
for row in csv.DictReader(open(sys.argv[1], encoding="utf-8")):
    chosen[row["instance_id"]].append(
        {col[len("codes."):] for col, v in row.items() if col.startswith("codes.") and v})
resolved = {row["instance_id"]: set(json.loads(row["value"]))
            for row in csv.DictReader(open(sys.argv[2], encoding="utf-8"))
            if row["schema"] == "codes"}
 
out = csv.writer(sys.stdout)
out.writerow(["excerpt", "codes", "source"])
for excerpt, sets in sorted(chosen.items()):
    if excerpt in resolved:
        out.writerow([excerpt, "; ".join(sorted(resolved[excerpt])), "adjudicated"])
    elif all(s == sets[0] for s in sets):
        out.writerow([excerpt, "; ".join(sorted(sets[0])), "agreed"])
    else:
        out.writerow([excerpt, "", "NOT RESOLVED"])

In the test run, final_codes.csv had 52 rows, 28 agreed and 24 adjudicated. After a change to the codebook, alpha from the first round no longer describes the codebook the team uses. Have both coders code a fresh set of excerpts with the revised codebook and report alpha from that round. Adjudication and Disagreement covers adjudication designs in more detail.

Exporting a REFI-QDA project file

REFI-QDA is the exchange standard that NVivo, ATLAS.ti, MAXQDA, and QualCoder read, and a .qdpx file contains a project's texts, codes, and codings. Export the coded project:

python -m potato.export --config config.yaml --format qdpx --output export/

The command writes export/Research_software_sustainability_interviews.qdpx. In the test run, the file contained:

• 52 text sources, one per excerpt
• the ten codes, with the definitions
• both coders as users
• the 154 highlights, each with the coder who made the highlight
• 167 codes applied to whole excerpts

The file contains both coders' coding, and the adjudicated codes are not in the file.

From Potato 2.13.1, the file validates against the REFI-QDA project schema. In 2.12.1 the file failed the schema check whenever a project had both highlights and whole-excerpt codes, because Potato wrote each coder's highlights and codes in turn, and a script had to reorder each source's elements.

Every highlight and every whole-excerpt code in the file names the coder who made it. In 2.12.1 the whole-excerpt codes named no coder. Importing a Project covers how potato import reads a .qdpx file. Importing the file back into Potato 2.13.1 with potato import -i Research_software_sustainability_interviews.qdpx -o reimport gave 52 texts and all 154 highlights at the same offsets, with these losses:

• The highlights arrive as suggestions for every coder. Each highlight keeps its coder's name in an annotator field, which Potato 2.12.1 dropped.
• The whole-excerpt codes arrive as a document_codes field on each item, and no question reads the field.
• The interviewer's question and the interview number stay behind, because the export writes the answer text only.
• The code definitions arrive as description on each span label, and the span question does not display description.

Other tools for team coding

NVivo's coding comparison query reports Cohen's kappa and percentage agreement for each code in each file between two users or user groups. NVivo counts agreement in characters of text. ATLAS.ti's inter-coder agreement tool reports percent agreement, the Holsti index, and Krippendorff's alpha family after the coders' projects are merged, and ATLAS.ti's manual says Cohen's kappa was left out on purpose. MAXQDA reports percentage agreement and a chance-corrected kappa at the document level, and Brennan and Prediger's kappa at the segment level. At the segment level, two codings match when the codings overlap by at least 90% by default. Dedoose's Testing Center compares a coder's code applications with an expert's using Cohen's kappa. QualCoder is a free desktop program and compares coders with Cohen's kappa. Each of these tools codes audio, video, or PDFs as well as text, and each fits a team that already works in that tool.

Potato fits a team that wants alpha with an interval in a free tool on a server the team controls, and that codes in a browser without a desktop install. Qualitative data analysis software compared lists prices, hosting, and AI features for each tool.

Limits of Potato for team coding

• Text only. Potato codes text, and the REFI-QDA importer reads text sources only. potato transcripts --transcribe --diarize turns recordings into transcripts first.
• No per-code alpha in the export. The reliability export gives one row per question, and per-code alpha needs the script above.
• One source per excerpt in the .qdpx file. A project coded by excerpt becomes 52 sources in NVivo or ATLAS.ti, one per answer, rather than six interviews.
• Both coders' raw coding in the .qdpx file. The adjudicated codes go out in adjudicated.csv only.

Where to go next

• QDA Mode, for building the codebook with one analyst before team coding
• Content Analysis, for codebooks loaded from a file and the reliability table's columns
• Inter-Annotator Agreement, for choosing and reading an agreement coefficient
• Agreement for Spans, for the measures that fit highlighted passages
• Adjudication and Disagreement, for adjudication designs
• Agreement calculator, for alpha and kappa from a CSV of codes exported by any tool
• Qualitative research in Potato, for the qualitative features in one place

References

1. Rosado de Souza, M., Haines, R., Vigo, M., & Jay, C. (2019). What Makes Research Software Sustainable? An Interview Study with Research Software Engineers. 2019 IEEE/ACM 12th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE), 135-138. https://doi.org/10.1109/CHASE.2019.00039
2. de Souza, M. R., Haines, R., Vigo, M., & Jay, C. (2019). What makes research software sustainable? Anonymized interview transcripts [Data set]. Zenodo. https://doi.org/10.5281/zenodo.1345067
3. Passonneau, R. (2006). Measuring Agreement on Set-valued Items (MASI) for Semantic and Pragmatic Annotation. Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC'06). https://aclanthology.org/L06-1392/