# Text-as-Data Annotation for Computational Social Science

Source: https://www.potatoannotator.com/use-cases/computational-social-science

**Potato codes news articles, social media posts, speeches, and comment threads for stance, framing, toxicity, and misinformation, with several coders per item and Krippendorff's alpha reported per question. Potato records an LLM's labels under a declared model ID and reports the model's agreement with people apart from agreement among the people.**

[Computational social science](https://en.wikipedia.org/wiki/Computational_social_science) studies social behavior with computational methods, and much of the field treats text as data. [Grimmer and Stewart (2013)](https://doi.org/10.1093/pan/mps028) review automated content analysis methods for political texts. Hand-coded labels serve three purposes in this work. Labels train a classifier, labels are the reference a model's output is checked against, and labels are the measurement itself when a corpus is small enough to code in full. Because each purpose needs coders who apply a codebook the same way, the reliability of the coding is part of the result. Potato assigns items to coders, keeps each coder's answers separate, and reports agreement per question while coding runs. Questionnaires and survey experiments are covered on the [survey research](/use-cases/survey-research) page.

## Text-as-data tasks in Potato

Each row is a feature that ships with Potato 2.13.1, and the linked page has its options.

| Task | How Potato does it | Docs |
|---|---|---|
| Content analysis with a codebook | `radio` and `multiselect` questions, or a `hierarchical_multiselect` codebook loaded from a CSV, JSON, or YAML file | [Content analysis](/docs/guides/content-analysis) |
| Stance toward a target | The target shown beside the text through `instance_display`, and a `radio` question for favor, against, or neither | [Radio and multiselect](/docs/annotation-types/radio-multiselect) |
| Framing | A `multiselect` over frame categories, such as the 15 framing dimensions of the [Media Frames Corpus](https://aclanthology.org/P15-2072/) | [Radio and multiselect](/docs/annotation-types/radio-multiselect) |
| Toxicity and incivility | `likert` ratings for degree, and `span` labels for the words that make a comment toxic | [Span annotation](/docs/annotation-types/span-annotation) |
| Misinformation and claims | `span` labels for check-worthy claims, and `radio` verdicts on each claim | [Span annotation](/docs/annotation-types/span-annotation) |
| Scaling texts on a continuum | `bws` tuples, scored into one value per text by `python -m potato.bws_scoring` with counting or a Plackett-Luce model | [Best-worst scaling](/docs/annotation-types/best-worst-scaling) |
| Crowdsourced coding | Recruitment through Prolific, `attention_checks`, and `gold_standards` with an accuracy floor | [Crowdsourcing](/docs/guides/crowdsourcing-prolific-mturk) |
| LLM-assisted coding checked against people | `machine_annotators` declares the model as a coder, and agreement reports keep the model apart from the people | [LLM-assisted coding](/docs/guides/llm-assisted-qualitative-coding) |
| Reply threads and conversations | `potato convokit` turns a ConvoKit corpus into Potato items with the reply structure kept | [Threaded conversations](/docs/guides/annotating-threaded-conversations) |

## Stance, framing, and incivility in one task

A task can ask several questions about the same post. The config below shows the stance target above each post and asks for the author's stance, the frames the post uses, and how uncivil the post is, with three coders per item. The frame labels are a subset of the Media Frames Corpus dimensions, and a real codebook would define each one in the guidelines.

```yaml
annotation_task_name: "Stance and framing in news posts"
task_dir: "."
data_files:
  - data/posts.jsonl
item_properties:
  id_key: id
  text_key: text
instance_display:
  fields:
    - key: target
      type: text
      label: "Target"
    - key: text
      type: text
      label: "Post"
output_annotation_dir: annotation_output/
user_config:
  allow_all_users: true
num_annotators_per_item: 3
agreement_metrics:
  enabled: true
  min_overlap: 2
annotation_schemes:
  - annotation_type: radio
    name: stance
    description: "What position does the author take toward the target?"
    labels: [favor, against, neither]
    label_requirement:
      required: true
  - annotation_type: multiselect
    name: frames
    description: "Which frames does the post use? Select all that apply."
    labels: [economic, morality, fairness and equality, legality, security and defense, public opinion, other]
  - annotation_type: likert
    name: incivility
    description: "How uncivil is the post?"
    size: 5
    min_label: "Civil"
    max_label: "Very uncivil"
```

Each item in `data/posts.jsonl` carries an `id`, a `target`, and a `text` field. Three coders per item cost more than two. A third coder breaks most ties and steadies alpha on subjective questions such as incivility. [How many annotators do you need?](/docs/guides/how-many-annotators) works through the trade-off. After coding, the `reliability` export writes Krippendorff's alpha for each question with a bootstrap interval, at the nominal level for stance, with partial credit for overlapping frame sets, and at the ordinal level for incivility.

## Checking crowd coders and LLM coders

[Benoit et al. (2016)](https://doi.org/10.1017/S0003055416000058) reported that political text coded by many nonexpert crowd workers gave results comparable to coding by experts. Potato recruits coders through Prolific and screens each coder with `attention_checks`, and `gold_standards` measures each coder's accuracy against items with known answers during the run. [Gold standards and attention checks](/docs/guides/gold-standards-and-attention-checks) covers both checks.

[Gilardi, Alizadeh, and Kubli (2023)](https://arxiv.org/abs/2303.15056) found that ChatGPT's zero-shot accuracy exceeded crowd workers' on four of five annotation tasks on 2,382 tweets, and the tasks included stance and frame detection. Whether a model does as well on your codebook and your corpus is an empirical question for each study. Potato records a model's labels under an ID declared in `machine_annotators`, with the model name and run date. The agreement report on `/admin/iaa` then keeps the headline figures human-only and reports the model separately, and the csv, tsv, and parquet exports add an `annotator_origin` column marking each row as human or model. For single-choice and rating questions, the admin API runs the [alternative annotator test](https://aclanthology.org/2025.acl-long.782/) of Calderon, Reichart, and Dror (2025) once three or more people have each coded 30 or more items that the model also coded.

## Bringing data in and getting labels out

Data files are JSON, JSONL, CSV, or TSV with one item per row. A sample of posts or articles drawn in your own pipeline loads without conversion. A corpus in [ConvoKit](https://convokit.cornell.edu/) format comes across with one command. The command below writes one item per conversation and a starter config built from the corpus metadata:

```bash
potato convokit conversations-gone-awry-corpus -o data/awry.jsonl --emit-config config.yaml
```

Labels go out as `csv`, `jsonl`, or `parquet` with one row per coder per item, or as a Hugging Face dataset for training a classifier. The row-per-coder files keep every coder's answer for an agreement analysis. The `convokit` export writes the labels back onto the corpus's utterances and conversations.

## Working alongside Prolific, Label Studio, and spreadsheets

Many content analysis projects code in a spreadsheet, and many crowdsourced projects build the coding form in a survey tool and recruit through [Prolific](https://www.prolific.com/). Both approaches leave the assignment of items to coders, the overlap sample, and the agreement statistics to scripts written for each study. [Label Studio](https://labelstud.io/) is an open-source platform for data labeling and AI evaluation that covers text, images, audio, and time series in one interface. Potato is built around the coding study, with the number of coders per item set in the config, attention and gold checks, reliability reported per question, and a config file that reproduces the task for the methods section. Prolific recruits verified participants for academic research, and Potato runs the coding once those participants arrive.

## Setting up the study with a coding agent

[potato-skill](/docs/getting-started/quick-start-with-a-coding-agent) gives Claude Code, Codex, or Cursor the instructions for designing and building a Potato study. A plain description of the work is enough to start. A request can be this short:

> Set up stance coding for 3,000 tweets about a carbon tax, with three Prolific coders per tweet, ten gold tweets for screening, and Krippendorff's alpha reported per question.

The agent lists each assumption made along the way, builds the Potato task, and renders the task to check the result. Open the task and label a few items yourself before inviting annotators.

## Frequently asked questions

### How do I annotate tweets for stance toward a target?

Show the target beside each tweet with `instance_display`, and ask a `radio` question with favor, against, and neither, as in [SemEval-2016 Task 6](https://aclanthology.org/S16-1003/). With the target stored in each item's data, one task can cover several targets. Stance differs from sentiment, because a tweet with a negative tone can still favor the target. The guidelines should give an example of each case. The [stance detection design](/showcase/task06-stance-detection) is a starting config.

### Can ChatGPT replace crowd workers for text annotation?

ChatGPT beat crowd workers on four of five text-annotation tasks in one published comparison on tweets, and a test on your own items shows whether a model does on yours. Have people code a sample blind to the model's labels, declare the model in `machine_annotators`, and compare the model's agreement with the people against the people's agreement with each other. The admin API runs the alternative annotator test, a check of whether the model could stand in for one of the human coders.

## Further reading

- [Inter-annotator agreement](/docs/guides/inter-annotator-agreement), for which coefficient to report and how to read it
- [Aggregating crowd labels](/docs/guides/aggregating-crowd-labels), for combining several coders' answers into one label
- [Writing annotation guidelines](/docs/guides/writing-annotation-guidelines), for the codebook coders work from
- [LLM pre-annotation](/docs/guides/llm-pre-annotation), for model suggestions that coders confirm or correct
- [LLM annotators vs humans](/blog/llm-annotators-vs-humans), for when a model is a good coder
- [Content moderation annotation](/blog/content-moderation-annotation), for toxicity and policy labels
- [Survey research](/use-cases/survey-research), [qualitative research](/use-cases/qualitative-research), and [NLP and text](/use-cases/text-annotation), for neighboring fields on this site

## References

1. Grimmer and Stewart (2013). Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts. *Political Analysis* 21(3). https://doi.org/10.1093/pan/mps028
2. Card, Boydstun, Gross, Resnik, and Smith (2015). The Media Frames Corpus: Annotations of Frames Across Issues. *ACL-IJCNLP 2015*. https://doi.org/10.3115/v1/P15-2072
3. Mohammad, Kiritchenko, Sobhani, Zhu, and Cherry (2016). SemEval-2016 Task 6: Detecting Stance in Tweets. *SemEval-2016*. https://doi.org/10.18653/v1/S16-1003
4. Benoit, Conway, Lauderdale, Laver, and Mikhaylov (2016). Crowd-sourced Text Analysis: Reproducible and Agile Production of Political Data. *American Political Science Review* 110(2). https://doi.org/10.1017/S0003055416000058
5. Gilardi, Alizadeh, and Kubli (2023). ChatGPT Outperforms Crowd Workers for Text-Annotation Tasks. *Proceedings of the National Academy of Sciences* 120(30). https://doi.org/10.1073/pnas.2305016120
6. Calderon, Reichart, and Dror (2025). The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs. *ACL 2025*. https://doi.org/10.18653/v1/2025.acl-long.782