Text-as-Data Annotation for Computational Social Science

Q: How do I annotate tweets for stance toward a target?

Show the target beside each tweet with instance_display, and ask a radio question with favor, against, and neither, as in SemEval-2016 Task 6. With the target stored in each item's data, one task can cover several targets. Stance differs from sentiment, because a tweet with a negative tone can still favor the target. The guidelines should give an example of each case. The stance detection design is a starting config.

Potato codes news articles, social media posts, speeches, and comment threads for stance, framing, toxicity, and misinformation, with several coders per item and Krippendorff's alpha reported per question. Potato records an LLM's labels under a declared model ID and reports the model's agreement with people apart from agreement among the people.

Computational social science studies social behavior with computational methods, and much of the field treats text as data. Grimmer and Stewart (2013) review automated content analysis methods for political texts. Hand-coded labels serve three purposes in this work. Labels train a classifier, labels are the reference a model's output is checked against, and labels are the measurement itself when a corpus is small enough to code in full. Because each purpose needs coders who apply a codebook the same way, the reliability of the coding is part of the result. Potato assigns items to coders, keeps each coder's answers separate, and reports agreement per question while coding runs. Questionnaires and survey experiments are covered on the survey research page.

Text-as-data tasks in Potato

Each row is a feature that ships with Potato 2.13.1, and the linked page has its options.

Task	How Potato does it	Docs
Content analysis with a codebook	`radio` and `multiselect` questions, or a `hierarchical_multiselect` codebook loaded from a CSV, JSON, or YAML file	Content analysis
Stance toward a target	The target shown beside the text through `instance_display`, and a `radio` question for favor, against, or neither	Radio and multiselect
Framing	A `multiselect` over frame categories, such as the 15 framing dimensions of the Media Frames Corpus	Radio and multiselect
Toxicity and incivility	`likert` ratings for degree, and `span` labels for the words that make a comment toxic	Span annotation
Misinformation and claims	`span` labels for check-worthy claims, and `radio` verdicts on each claim	Span annotation
Scaling texts on a continuum	`bws` tuples, scored into one value per text by `python -m potato.bws_scoring` with counting or a Plackett-Luce model	Best-worst scaling
Crowdsourced coding	Recruitment through Prolific, `attention_checks`, and `gold_standards` with an accuracy floor	Crowdsourcing
LLM-assisted coding checked against people	`machine_annotators` declares the model as a coder, and agreement reports keep the model apart from the people	LLM-assisted coding
Reply threads and conversations	`potato convokit` turns a ConvoKit corpus into Potato items with the reply structure kept	Threaded conversations

Stance, framing, and incivility in one task

A task can ask several questions about the same post. The config below shows the stance target above each post and asks for the author's stance, the frames the post uses, and how uncivil the post is, with three coders per item. The frame labels are a subset of the Media Frames Corpus dimensions, and a real codebook would define each one in the guidelines.

yaml

annotation_task_name: "Stance and framing in news posts"
task_dir: "."
data_files:
  - data/posts.jsonl
item_properties:
  id_key: id
  text_key: text
instance_display:
  fields:
    - key: target
      type: text
      label: "Target"
    - key: text
      type: text
      label: "Post"
output_annotation_dir: annotation_output/
user_config:
  allow_all_users: true
num_annotators_per_item: 3
agreement_metrics:
  enabled: true
  min_overlap: 2
annotation_schemes:
  - annotation_type: radio
    name: stance
    description: "What position does the author take toward the target?"
    labels: [favor, against, neither]
    label_requirement:
      required: true
  - annotation_type: multiselect
    name: frames
    description: "Which frames does the post use? Select all that apply."
    labels: [economic, morality, fairness and equality, legality, security and defense, public opinion, other]
  - annotation_type: likert
    name: incivility
    description: "How uncivil is the post?"
    size: 5
    min_label: "Civil"
    max_label: "Very uncivil"

Each item in data/posts.jsonl carries an id, a target, and a text field. Three coders per item cost more than two. A third coder breaks most ties and steadies alpha on subjective questions such as incivility. How many annotators do you need? works through the trade-off. After coding, the reliability export writes Krippendorff's alpha for each question with a bootstrap interval, at the nominal level for stance, with partial credit for overlapping frame sets, and at the ordinal level for incivility.

Checking crowd coders and LLM coders

Benoit et al. (2016) reported that political text coded by many nonexpert crowd workers gave results comparable to coding by experts. Potato recruits coders through Prolific and screens each coder with attention_checks, and gold_standards measures each coder's accuracy against items with known answers during the run. Gold standards and attention checks covers both checks.

Gilardi, Alizadeh, and Kubli (2023) found that ChatGPT's zero-shot accuracy exceeded crowd workers' on four of five annotation tasks on 2,382 tweets, and the tasks included stance and frame detection. Whether a model does as well on your codebook and your corpus is an empirical question for each study. Potato records a model's labels under an ID declared in machine_annotators, with the model name and run date. The agreement report on /admin/iaa then keeps the headline figures human-only and reports the model separately, and the csv, tsv, and parquet exports add an annotator_origin column marking each row as human or model. For single-choice and rating questions, the admin API runs the alternative annotator test of Calderon, Reichart, and Dror (2025) once three or more people have each coded 30 or more items that the model also coded.

Bringing data in and getting labels out

Data files are JSON, JSONL, CSV, or TSV with one item per row. A sample of posts or articles drawn in your own pipeline loads without conversion. A corpus in ConvoKit format comes across with one command. The command below writes one item per conversation and a starter config built from the corpus metadata:

bash

potato convokit conversations-gone-awry-corpus -o data/awry.jsonl --emit-config config.yaml

Labels go out as csv, jsonl, or parquet with one row per coder per item, or as a Hugging Face dataset for training a classifier. The row-per-coder files keep every coder's answer for an agreement analysis. The convokit export writes the labels back onto the corpus's utterances and conversations.

Working alongside Prolific, Label Studio, and spreadsheets

Many content analysis projects code in a spreadsheet, and many crowdsourced projects build the coding form in a survey tool and recruit through Prolific. Both approaches leave the assignment of items to coders, the overlap sample, and the agreement statistics to scripts written for each study. Label Studio is an open-source platform for data labeling and AI evaluation that covers text, images, audio, and time series in one interface. Potato is built around the coding study, with the number of coders per item set in the config, attention and gold checks, reliability reported per question, and a config file that reproduces the task for the methods section. Prolific recruits verified participants for academic research, and Potato runs the coding once those participants arrive.

Setting up the study with a coding agent

potato-skill gives Claude Code, Codex, or Cursor the instructions for designing and building a Potato study. A plain description of the work is enough to start. A request can be this short:

Set up stance coding for 3,000 tweets about a carbon tax, with three Prolific coders per tweet, ten gold tweets for screening, and Krippendorff's alpha reported per question.

The agent lists each assumption made along the way, builds the Potato task, and renders the task to check the result. Open the task and label a few items yourself before inviting annotators.

Frequently asked questions

How do I annotate tweets for stance toward a target?

Show the target beside each tweet with instance_display, and ask a radio question with favor, against, and neither, as in SemEval-2016 Task 6. With the target stored in each item's data, one task can cover several targets. Stance differs from sentiment, because a tweet with a negative tone can still favor the target. The guidelines should give an example of each case. The stance detection design is a starting config.

Can ChatGPT replace crowd workers for text annotation?

ChatGPT beat crowd workers on four of five text-annotation tasks in one published comparison on tweets, and a test on your own items shows whether a model does on yours. Have people code a sample blind to the model's labels, declare the model in machine_annotators, and compare the model's agreement with the people against the people's agreement with each other. The admin API runs the alternative annotator test, a check of whether the model could stand in for one of the human coders.

References

Grimmer and Stewart (2013). Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts. Political Analysis 21(3). https://doi.org/10.1093/pan/mps028
Card, Boydstun, Gross, Resnik, and Smith (2015). The Media Frames Corpus: Annotations of Frames Across Issues. ACL-IJCNLP 2015. https://doi.org/10.3115/v1/P15-2072
Mohammad, Kiritchenko, Sobhani, Zhu, and Cherry (2016). SemEval-2016 Task 6: Detecting Stance in Tweets. SemEval-2016. https://doi.org/10.18653/v1/S16-1003
Benoit, Conway, Lauderdale, Laver, and Mikhaylov (2016). Crowd-sourced Text Analysis: Reproducible and Agile Production of Political Data. American Political Science Review 110(2). https://doi.org/10.1017/S0003055416000058
Gilardi, Alizadeh, and Kubli (2023). ChatGPT Outperforms Crowd Workers for Text-Annotation Tasks. Proceedings of the National Academy of Sciences 120(30). https://doi.org/10.1073/pnas.2305016120
Calderon, Reichart, and Dror (2025). The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs. ACL 2025. https://doi.org/10.18653/v1/2025.acl-long.782

Potato for computational social science and text as data