Build or Buy an Annotation Tool: Spreadsheets, Custom Apps and Platforms

Use a spreadsheet when one person is labelling a few hundred items with one label each. Once several people label the same items, you need them to work independently and you need an agreement number, and from then on an existing annotation tool costs less than building one. A hosted platform is worth paying for when you also want it to supply annotators, train models, or satisfy a procurement requirement.

An annotation tool shows each item to an annotator, records the label, and keeps track of who labelled what. Inter-rater reliability, usually reported as Cohen's kappa or Krippendorff's alpha, measures whether independent annotators agree, and it is what reviewers ask for when a paper reports human labels. See Inter-Annotator Agreement Explained.

When a spreadsheet is enough

A spreadsheet is the right tool when all of these hold:

• one annotator, usually you
• one label per item, from a short fixed list
• a few hundred items
• no agreement to report

Put the labels in a dropdown with data validation, so they cannot drift into Positive, positive and pos across a long afternoon.

Where spreadsheets break

• Annotators see each other's labels. In a shared sheet the second annotator can read the first annotator's column, and agreement measured that way is not agreement between independent judgments.
• Assignment is manual. Who labels which rows, and how many labels each row needs, is tracked by hand.
• A cell holds one value for the whole row. Highlighting a phrase, drawing a box, or ranking four model responses does not fit in a cell.
• Nothing records the process. There is no record of how long an annotator spent on an item, or which version of the instructions they had seen.

Building your own labelling app

Coding agents have made this common. Ask Claude Code, Codex or Cursor for a Streamlit or Flask page that shows one item at a time and saves each answer to a CSV, and you have a working tool in minutes. For one person, that is often enough.

The work arrives with the second annotator and the second study. A multi-annotator study needs logins, assignment of items to people, instructions and a training round, attention checks, agreement, a completion code for Prolific or MTurk, and export in the format the next script expects. Each of those is code someone has to write, test and debug, in a tool that serves one project.

If you were going to ask a coding agent to build the tool, ask it to design the study instead. potato-skill has the agent work through the design with you and build the result as a Potato task, which already has every item on that list. The task is a folder with a config and data, so a colleague can run it too.

Open-source or hosted

Self-hosted open-source tools, including Potato, Label Studio's community edition, Doccano and CVAT, are free to use, keep the data on machines you control, and leave you to run the server. Hosted platforms run the server for you, and some also supply a workforce, model-assisted labelling or model training, usually for a per-seat or per-item price.

Four questions narrow it down:

1. Where must the data live? If it cannot leave your institution, self-host.
2. Who are the annotators? Your own team, crowdworkers you recruit, or a vendor's workforce. Only the last needs a platform that supplies people.
3. Do you need agreement statistics in the tool? Many tools leave this to you, or reserve it for a paid tier.
4. Can you leave? Check which formats the tool imports and exports before your data is in it.

Annotation Tools Compared goes through 14 tools on these points.

Evaluating generative AI output

Human evaluation of LLM and agent output is the same decision with extra requirements. Raters should not know which model wrote a response, response order should be randomised, and you usually want agreement between human raters and, if you use one, between the humans and an LLM judge. A spreadsheet handles none of that well, and a home-built app has to implement each piece.

Potato covers pairwise comparison, rubric scoring, agent trajectories and LLM-judge calibration.

A minimal Potato task

One label per row becomes this config:

annotation_task_name: "Sentiment Analysis"
task_dir: "."
data_files:
  - "data/data.json"
item_properties:
  id_key: id
  text_key: text
output_annotation_dir: "annotation_output/"
annotation_schemes:
  - annotation_type: radio
    name: sentiment
    description: "What is the sentiment of this text?"
    labels:
      - Positive
      - Negative
      - Neutral

Start it with potato start config.yaml -p 8000 and send annotators the URL. Quick Start walks through it.

Side by side

Further reading

• Quick Start with a Coding Agent
• Annotation Tools Compared
• How Many Annotators Do You Need?
• Choosing an Open-Source Annotation Tool in 2026