How to Use an LLM in Qualitative Coding and Report It

A large language model can take three roles in qualitative coding: suggesting codes to a human coder, coding a reliability sample beside the human coders, or coding the corpus first for people to review. In every role, report the model's agreement with your human coders on the coefficient you use between the coders, keep the model's codes apart from human codes, record the prompt and the model version, and confirm that the consent form covers where the interview text went.

Qualitative coding assigns codes from a codebook to excerpts of interviews, focus groups, field notes, or open-ended answers. Deductive coding applies a codebook written before coding starts, and inductive coding builds the codes from the data. A large language model (LLM) produces text from a prompt. Given a prompt holding the codebook and one excerpt, the model returns codes for the excerpt. Intercoder reliability is the agreement between coders who code the same material independently, usually reported as Cohen's kappa or Krippendorff's alpha.

The coding studies cited here all tested deductive coding against a codebook the researchers had written, and the advice assumes the same design. Wheeler (2025) notes that reflexive thematic analysis does not sit easily with AI-assisted analysis. Reflexive thematic analysis assumes that the researcher's interpretation shapes every stage. Separate guides cover when to report intercoder reliability, building a codebook, the mechanics of model pre-annotation, when to automate a labeling job, and where each QDA program sends text.

Three roles for a model

The roles differ in who makes the final call on a code and in what you report to show whether the model helped.

Published studies of LLM qualitative coding

In the studies below, agreement between models and human coders changed from code to code and with the prompt. A published figure from another codebook is therefore a weak guide to agreement on yours. Measure the model on your own reliability sample instead.

Xiao et al. (2023) coded 668 children's questions with GPT-3 at temperature 0 against a psychologists' codebook. Agreement with the expert coding reached κ = 0.61 for question complexity and κ = 0.38 for syntactic structure, against κ = 0.88 and κ = 0.90 between experts. The best prompts were laid out like a codebook, with a description and examples per code.

In a preprint, Chew et al. (2023) coded 100-document samples from four public datasets with GPT-3.5 and scored agreement with Gwet's AC1. Model-human agreement was close to human-human agreement on most codes and fell well short on codes about hashtags, @-mentions, and capitals. On hashtags, model-human agreement was 0.18 against 0.96 between humans.

In a preprint, Dunivin (2024) coded passages from New York Times articles mentioning W.E.B. Du Bois. On 111 gold-standard passages, GPT-4 reached κ ≥ 0.6 on 8 of 9 codes, and GPT-3.5 averaged κ = 0.34. Asking for a rationale with each code raised mean κ from 0.59 to 0.68.

Tai et al. (2024) ran one codebook prompt 160 times on each of three interview excerpts with ChatGPT 3.5. The codes changed from run to run. For one excerpt, the model assigned the code Persistence in 59.4% of runs.

Schroeder, Roy, and Kabbara (2025) tested the suggester role with 350 annotators on subjective tasks. With suggestions, annotators worked no faster, and the distribution of their labels shifted. Models evaluated against the assisted labels looked significantly better than models evaluated against labels given without suggestions.

Model-human agreement on the reliability sample

If your methods section reports Krippendorff's alpha between coders, compute alpha between the model and each coder on the same reliability sample, per code, and print the figures beside the human-human figure. Keep the model out of the human-human figure. One coefficient over a matrix mixing the model with the people describes neither group. Because a coder who has seen the model's codes is no longer independent of the model, the human coders code the reliability sample blind to the model's codes. Set the temperature to 0, or run the model several times per excerpt and report how often the runs agree.

High agreement alone does not show that the model could replace a coder. The alternative annotator test of Calderon, Reichart, and Dror (2025) leaves out one human coder at a time. The test then checks whether the model matches the remaining coders at least as well as the left-out coder does, within a margin. Potato runs the test for single-choice and rating schemes.

Labeling model codes in the data

A model's codes saved under a person's name look like that person's work, and any intercoder figure computed from the file then counts the model as a person. Give the model a separate coder ID, and add a column marking each row as human or model. For first-pass coding, keep the model's original code beside the reviewed code. With both codes stored, you can count how many codes reviewers changed.

Records of the prompt and the model

A reader cannot judge or rerun model coding without the prompt. Record the following details in a supplement:

• The model name and exact version, such as a dated API snapshot or the ID that ollama list prints.
• The provider or server that ran the model, and the dates of the runs.
• The full prompt, including the codebook text, with each revision dated.
• The temperature and the number of runs per excerpt.
• What happened when the model returned no valid code.

Hosted versus local models and consent

A hosted API processes participant text on the provider's servers under the provider's retention terms, and a local model keeps the text on a machine the team controls. Check the provider's terms on retention and training against the consent form and the ethics approval before a hosted run. Wheeler (2025) recommends de-identifying data first, warns that changing names alone is not always enough, and recommends a statement about generative AI processing in consent forms. The Rutgers AI consent guide suggests stating the tool's name, whether the tool is university-licensed or external, whether the vendor may train models on the data, and how long the vendor keeps the data.

Commercial QDA programs send text to different places. ATLAS.ti's AI Coding sends text to OpenAI, MAXQDA's AI Assist Premium can use a model hosted on your own computer or server, and NVivo's assistant AURA runs in NVivo AI Cloud. Air-gapped annotation covers working with no route out at all.

LLM-assisted coding in Potato 2.13.1

With ai_support, a model offers hints, keyword highlights, a rationale for each code, and suggested codes on the excerpt a coder has open, and the coder decides. The assistant works on schemes marked codebook: true and in a QDA Mode project. The prompt lists the current code names and the scheme's description, without the definitions stored in the codebook. Put the deciding criteria in the description. Setting endpoint_type to ollama or vllm with a base_url on your own server keeps excerpts there, and openai, anthropic, and gemini send each excerpt to the named provider.

For first-pass coding, pre_annotation pre-fills the model's codes from a separate predictions_file. A pre-filled code becomes the coder's answer only when the coder saves, and Potato records the seeded value in the excerpt's behavioral data. You can count unchanged codes by comparing the seeded value with the saved code. ai_support.holdout.fraction withholds suggestions and pre-filled codes from a random share of coder-excerpt pairs. The admin dashboard's AI influence tab compares the two groups on agreement with the model and on alpha within each group. The config below sets up suggestions, pre-filled codes, and the holdout with a local model:

annotation_task_name: Clinic access interviews
task_dir: .
output_annotation_dir: annotation_output/
data_files:
  - data/excerpts.json
item_properties:
  id_key: id
  text_key: text
user_config:
  allow_all_users: true
  users: []
 
# Coders apply the agreed codes and cannot add new ones
codebook_mode: fixed
 
annotation_schemes:
  - annotation_type: multiselect
    name: codes
    description: Which codes from the codebook apply to this excerpt?
    codebook: true
    labels: [access barriers, cost concerns, provider trust, administrative burden]
 
# A model on the team's own machine offers hints and suggested codes
ai_support:
  enabled: true
  endpoint_type: ollama
  ai_config:
    model: llama3.1:8b
    base_url: http://localhost:11434
    temperature: 0
    include:
      all: true
  holdout:
    fraction: 0.2
    salt: clinic-access-2026
 
# The model's first-pass codes, kept in their own file
pre_annotation:
  enabled: true
  predictions_file: data/model_codes.json
  allow_modification: true

The predictions file maps each excerpt ID to the model's codes per scheme, as in {"p01-03": {"codes": ["access barriers"]}}. Without include.all: true, no assistant buttons appear. Keep salt and fraction fixed for the study, because changing either value reshuffles the groups.

To use the model as a second coder, declare the model's coder ID under machine_annotators with the model and the run date. Where the provider exposes a dated snapshot, or ollama list prints an ID for the model, add that value as version beside model. Add this block to the config above:

machine_annotators:
  enabled: true
  annotators:
    - id: llm_coder
      kind: llm
      endpoint_type: ollama
      model: llama3.1:8b
      run_date: "2026-11-02"

/admin/iaa then keeps headline figures human-only and reports the model under separately named partitions. The csv, tsv, and parquet exports add an annotator_origin column (human or llm), and the jsonl export includes the full declaration. The reliability export pools every coder on an excerpt. To compute model-human alpha, filter the csv export on annotator_origin and load the rows into the agreement calculator. For single-choice and rating schemes, GET /admin/api/alt-test runs the alternative annotator test once three or more human coders have each coded 30 or more excerpts that the model also coded.

To record the model's codes under that ID, write a script that sends your documented prompt to the model, logs in as llm_coder, and saves through /updateinstance (API reference). The simulator's --strategy llm is no substitute. The simulator uses a generic prompt, sends the first 500 characters of each excerpt, and records a random code when the reply matches no code. Solo Mode reports raw percent agreement between one coder and the model, and ICL labeling returns one code per excerpt and scheme. Compute alpha on the exported codes before reporting results from either feature in a methods section.

Frequently asked questions

Can I use ChatGPT for qualitative coding?

You can use ChatGPT or another LLM for deductive coding against a written codebook if you measure the model's agreement with human coders on your own data and report the result. Published agreement varied by code, from κ = 0.61 to κ = 0.38 on one codebook in Xiao et al. (2023). For interview data, check the consent form and the ethics approval before using a hosted service.

How do I calculate intercoder reliability between an LLM and human coders?

Treat the model as one more coder on the reliability sample, and compute your team's coefficient between the model and each human coder. The coefficient is usually Krippendorff's alpha or Cohen's kappa. Report the model-human figures per code beside a human-human figure computed without the model. The agreement calculator computes alpha and kappa from a CSV of codes.

How should I report LLM-assisted coding in a methods section?

Report the model's role, name and version, the provider or server, the run dates, the temperature, and model-human agreement per code beside human-human agreement. State whether coders saw the model's codes and how disagreements were settled. Put the full prompt, with the codebook text, in a supplement.

Can I upload interview transcripts to ChatGPT?

Upload transcripts to a hosted service only when the consent form and the ethics approval cover sending participant text to that provider. De-identify the transcripts first, and remember that changing names alone is not always enough. A model run with Ollama or vLLM on the team's own server keeps transcripts off outside servers.

Further reading

• Qualitative research in Potato, for the qualitative features in one place
• Qualitative Data Analysis Software Compared
• Writing a Codebook an LLM Can Follow
• How to Measure Agreement Between an LLM Judge and Human Annotators
• How to Tell Review from Rubber-Stamping in Pre-Labeled Data
• Coding interview transcripts

References

1. Calderon, N., Reichart, R., & Dror, R. (2025). The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 16051-16081. https://doi.org/10.18653/v1/2025.acl-long.782
2. Chew, R., Bollenbacher, J., Wenger, M., Speer, J., & Kim, A. (2023). LLM-Assisted Content Analysis: Using Large Language Models to Support Deductive Coding. arXiv preprint 2306.14924. https://arxiv.org/abs/2306.14924
3. Dunivin, Z. O. (2024). Scalable Qualitative Coding with LLMs: Chain-of-Thought Reasoning Matches Human Performance in Some Hermeneutic Tasks. arXiv preprint 2401.15170. https://arxiv.org/abs/2401.15170
4. Schroeder, H., Roy, D., & Kabbara, J. (2025). Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Tasks. Findings of the Association for Computational Linguistics: ACL 2025, 25771-25795. https://doi.org/10.18653/v1/2025.findings-acl.1323
5. Tai, R. H., Bentley, L. R., Xia, X., Sitt, J. M., Fankhauser, S. C., Chicas-Mosier, A. M., & Monteith, B. G. (2024). An Examination of the Use of Large Language Models to Aid Analysis of Textual Data. International Journal of Qualitative Methods, 23. https://doi.org/10.1177/16094069241231168
6. Wheeler, K. (2025). How to Use Generative AI to Assist the Analysis of Qualitative Data. Sage Research Methods: Data and Research Literacy. https://doi.org/10.4135/9781036217471
7. Xiao, Z., Yuan, X., Liao, Q. V., Abdelghani, R., & Oudeyer, P.-Y. (2023). Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding. Companion Proceedings of the 28th International Conference on Intelligent User Interfaces, 75-78. https://doi.org/10.1145/3581754.3584136