Potato labels contracts, court opinions, statutes, and discovery documents with clause types, legal entities, citation links, and responsiveness and privilege calls, with two or more reviewers per document and agreement measured as the review runs. Potato is free and open source and keeps client documents on your own computer or server.
Legal annotation serves two kinds of project. First, legal NLP datasets are built from expert labels. CUAD holds over 13,000 contract annotations made with legal experts from The Atticus Project, and LexGLUE collects legal classification tasks from EU and US law. Second, electronic discovery review checks documents for relevance and privilege before production to the opposing party. Both kinds of project need annotators with legal training. Because an expert's label costs more than a crowd worker's, a disagreement between two experts is worth recording rather than averaging away. Potato assigns each document to several reviewers, keeps each reviewer's answers separate, and reports agreement per question. A pilot then shows which category definitions reviewers apply differently before the full set is labeled.
Legal annotation tasks in Potato
Each row is a feature that ships with Potato 2.13.1, and the linked page has its options.
| Task | How Potato does it | Docs |
|---|---|---|
| Clause classification | radio for one clause type per section, multiselect when a clause creates several obligations, and hierarchical_multiselect for a long provision taxonomy loaded from a file | Radio and multiselect |
| Legal named entities | span labels for parties, courts, judges, statutes, dates, and amounts | Span annotation |
| Citations and arguments | span_link connects a citation to the proposition the citation supports, or a premise to its conclusion | Span linking |
| Document review | radio questions for responsiveness and privilege on each document, with active_learning ordering the queue by a classifier's uncertainty | Active learning |
| Privilege and redaction labels | span labels mark the passages to withhold, and Potato stores the marks without changing the document | Span annotation |
| Filings and contracts as PDFs | The pdf display puts text spans, boxes, and links on the rendered page | PDF annotation |
| Legal QA evaluation | extractive_qa for answer spans, and rubric_eval or pairwise for rating a model's answer to a legal question | Rubric-based LLM evaluation |
A first-pass document review task
The config below sets up first-pass review with two reviewers from a fixed roster on each document. Each reviewer decides whether the document is responsive and privileged and marks the passages that would be withheld. Potato reports agreement on each question once documents have two reviews. The agreement figures show early whether reviewers read the privilege categories the same way.
annotation_task_name: "Document review: responsiveness and privilege"
task_dir: "."
data_files:
- data/documents.jsonl
item_properties:
id_key: id
text_key: text
output_annotation_dir: annotation_output/
user_config:
allow_all_users: false
users: [reviewer_a, reviewer_b, reviewer_c]
num_annotators_per_item: 2
agreement_metrics:
enabled: true
min_overlap: 2
annotation_schemes:
- annotation_type: radio
name: responsive
description: "Is this document responsive to the request?"
labels: [responsive, not responsive, needs attorney review]
label_requirement:
required: true
- annotation_type: radio
name: privilege
description: "Is this document privileged?"
labels: [not privileged, attorney-client, work product, partly privileged]
label_requirement:
required: true
- annotation_type: span
name: withhold
description: "Mark each passage that would be redacted before production"
labels: [PRIVILEGED_PASSAGE, PERSONAL_DATA]Setting allow_all_users: false limits the study to the usernames in users, and Potato asks for a password by default. The needs attorney review label lets a reviewer pass a hard document upward instead of guessing. Adjudication and disagreement covers settling the documents on which two reviewers disagree.
Confidential documents on your own hardware
Potato runs on a laptop, a firm's server, or a university cluster, and writes every annotation to that machine's disk under output_annotation_dir. Nothing is sent to a third party unless the config names an external service, such as a hosted AI endpoint. For model suggestions on privileged material, ai_support can call a model served on your own network through Ollama or vLLM. Air-gapped annotation covers running with no outbound connection once the model weights are on the machine. Potato does not redact documents, and a redaction label in a Potato task records a reviewer's decision for your own redaction step. Potato has no compliance certification. Whether a deployment meets a client's or a court's requirements is for your firm or institution to decide.
Bringing documents in and getting labels out
Data files are JSON, JSONL, CSV, or TSV with one item per row. Split long contracts into sections before loading, and carry the section number and title on each item. Legal document annotation explains where to cut and how to reassemble the labels into document-level results afterward. Filings that only exist as PDFs load as a file path in the item, and the pdf display renders each page in the browser.
Labels go out as jsonl with every reviewer's full answer, as csv or parquet for analysis, as conll_2003 token tags for training an entity tagger, or as a Hugging Face dataset. The reliability export writes Krippendorff's alpha for each question with a bootstrap interval, for the methods section of a dataset paper or a review protocol's quality record.
Working alongside Prodigy, Datasaur, and e-discovery platforms
Each of these tools is built for a different setting. Prodigy is a paid annotation tool sold with a lifetime license that runs on your own machines, and Prodigy's legal page shows an example case-law model trained with Prodigy. Datasaur offers LLM-assisted contract review, and Datasaur's legal page states that the deployment runs inside a firm's own infrastructure. E-discovery platforms such as Relativity provide review and production tools for litigation. Potato is built around the annotation study, with several reviewers per document, agreement reported per question, gold items with feedback, and a config file that reproduces the task. Potato suits building a labeled legal dataset or measuring how consistently reviewers apply a protocol, and an e-discovery platform suits production review on a live matter.
Setting up the study with a coding agent
potato-skill gives Claude Code, Codex, or Cursor the instructions for designing and building a Potato study. A plain description of the work is enough to start. A request can be this short:
Set up clause classification for 300 commercial leases split into sections, using our 12 clause types, two attorneys per section, and keyword highlighting for shall, must, and unless.
The agent lists each assumption made along the way, builds the Potato task, and renders the task to check the result. Open the task and label a few items yourself before inviting annotators.
Frequently asked questions
How do I measure agreement between annotators on legal documents?
Assign each document or clause to at least two annotators with num_annotators_per_item, and turn on agreement_metrics to see Krippendorff's alpha for each question on the admin dashboard. For entity spans, the /admin/iaa report gives span F1 with exact and with partial matching. The two F1 figures separate disagreement about the label from disagreement about where a span starts and stops. The reliability export writes alpha per question with a bootstrap interval. Agreement for spans explains the span measures.
Can Potato be used for document review?
Yes. Each document is one item, and reviewers answer responsiveness and privilege questions and mark the passages to withhold. With two or more reviewers per document, Potato measures agreement between the reviewers. With active_learning, a classifier trained on the labels so far puts the documents with the least certain predictions at the front of the queue. Potato does not collect, process, or produce documents for a legal matter, and Potato fits review research, protocol testing, and dataset building better than production review.
Further reading
- Writing annotation guidelines, for the protocol reviewers work from
- Gold standards and attention checks, for qualifying expert reviewers before real documents
- Expert vs. crowd annotation, for when legal training is needed
- Named entity recognition, for choosing an entity label set
- Relation and event extraction, for citation and argument links
- Document annotation, for long documents and their layout
- NLP and text and computational social science, for neighboring fields on this site
References
- Hendrycks, Burns, Chen, and Ball (2021). CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review. NeurIPS 2021. https://arxiv.org/abs/2103.06268
- Chalkidis, Jana, Hartung, Bommarito, Androutsopoulos, Katz, and Aletras (2022). LexGLUE: A Benchmark Dataset for Legal Language Understanding in English. ACL 2022. https://doi.org/10.18653/v1/2022.acl-long.297