# Legal Annotation Tool for Contracts and Document Review

Source: https://www.potatoannotator.com/use-cases/legal

**Potato labels contracts, court opinions, statutes, and discovery documents with clause types, legal entities, citation links, and responsiveness and privilege calls, with two or more reviewers per document and agreement measured as the review runs. Potato is free and open source and keeps client documents on your own computer or server.**

Legal annotation serves two kinds of project. First, legal NLP datasets are built from expert labels. [CUAD](https://arxiv.org/abs/2103.06268) holds over 13,000 contract annotations made with legal experts from The Atticus Project, and [LexGLUE](https://aclanthology.org/2022.acl-long.297/) collects legal classification tasks from EU and US law. Second, [electronic discovery](https://en.wikipedia.org/wiki/Electronic_discovery) review checks documents for relevance and privilege before production to the opposing party. Both kinds of project need annotators with legal training. Because an expert's label costs more than a crowd worker's, a disagreement between two experts is worth recording rather than averaging away. Potato assigns each document to several reviewers, keeps each reviewer's answers separate, and reports agreement per question. A pilot then shows which category definitions reviewers apply differently before the full set is labeled.

## Legal annotation tasks in Potato

Each row is a feature that ships with Potato 2.13.1, and the linked page has its options.

| Task | How Potato does it | Docs |
|---|---|---|
| Clause classification | `radio` for one clause type per section, `multiselect` when a clause creates several obligations, and `hierarchical_multiselect` for a long provision taxonomy loaded from a file | [Radio and multiselect](/docs/annotation-types/radio-multiselect) |
| Legal named entities | `span` labels for parties, courts, judges, statutes, dates, and amounts | [Span annotation](/docs/annotation-types/span-annotation) |
| Citations and arguments | `span_link` connects a citation to the proposition the citation supports, or a premise to its conclusion | [Span linking](/docs/annotation-types/span-linking) |
| Document review | `radio` questions for responsiveness and privilege on each document, with `active_learning` ordering the queue by a classifier's uncertainty | [Active learning](/docs/features/active-learning) |
| Privilege and redaction labels | `span` labels mark the passages to withhold, and Potato stores the marks without changing the document | [Span annotation](/docs/annotation-types/span-annotation) |
| Filings and contracts as PDFs | The `pdf` display puts text spans, boxes, and links on the rendered page | [PDF annotation](/docs/annotation-types/pdf-annotation) |
| Legal QA evaluation | `extractive_qa` for answer spans, and `rubric_eval` or `pairwise` for rating a model's answer to a legal question | [Rubric-based LLM evaluation](/docs/guides/rubric-based-llm-evaluation) |

## A first-pass document review task

The config below sets up first-pass review with two reviewers from a fixed roster on each document. Each reviewer decides whether the document is responsive and privileged and marks the passages that would be withheld. Potato reports agreement on each question once documents have two reviews. The agreement figures show early whether reviewers read the privilege categories the same way.

```yaml
annotation_task_name: "Document review: responsiveness and privilege"
task_dir: "."
data_files:
  - data/documents.jsonl
item_properties:
  id_key: id
  text_key: text
output_annotation_dir: annotation_output/
user_config:
  allow_all_users: false
  users: [reviewer_a, reviewer_b, reviewer_c]
num_annotators_per_item: 2
agreement_metrics:
  enabled: true
  min_overlap: 2
annotation_schemes:
  - annotation_type: radio
    name: responsive
    description: "Is this document responsive to the request?"
    labels: [responsive, not responsive, needs attorney review]
    label_requirement:
      required: true
  - annotation_type: radio
    name: privilege
    description: "Is this document privileged?"
    labels: [not privileged, attorney-client, work product, partly privileged]
    label_requirement:
      required: true
  - annotation_type: span
    name: withhold
    description: "Mark each passage that would be redacted before production"
    labels: [PRIVILEGED_PASSAGE, PERSONAL_DATA]
```

Setting `allow_all_users: false` limits the study to the usernames in `users`, and Potato asks for a password by default. The `needs attorney review` label lets a reviewer pass a hard document upward instead of guessing. [Adjudication and disagreement](/docs/guides/adjudication-and-disagreement) covers settling the documents on which two reviewers disagree.

## Confidential documents on your own hardware

Potato runs on a laptop, a firm's server, or a university cluster, and writes every annotation to that machine's disk under `output_annotation_dir`. Nothing is sent to a third party unless the config names an external service, such as a hosted AI endpoint. For model suggestions on privileged material, `ai_support` can call a model served on your own network through Ollama or vLLM. [Air-gapped annotation](/docs/guides/air-gapped-annotation) covers running with no outbound connection once the model weights are on the machine. Potato does not redact documents, and a redaction label in a Potato task records a reviewer's decision for your own redaction step. Potato has no compliance certification. Whether a deployment meets a client's or a court's requirements is for your firm or institution to decide.

## Bringing documents in and getting labels out

Data files are JSON, JSONL, CSV, or TSV with one item per row. Split long contracts into sections before loading, and carry the section number and title on each item. [Legal document annotation](/blog/legal-document-annotation) explains where to cut and how to reassemble the labels into document-level results afterward. Filings that only exist as PDFs load as a file path in the item, and the `pdf` display renders each page in the browser.

Labels go out as `jsonl` with every reviewer's full answer, as `csv` or `parquet` for analysis, as `conll_2003` token tags for training an entity tagger, or as a Hugging Face dataset. The `reliability` export writes Krippendorff's alpha for each question with a bootstrap interval, for the methods section of a dataset paper or a review protocol's quality record.

## Working alongside Prodigy, Datasaur, and e-discovery platforms

Each of these tools is built for a different setting. [Prodigy](https://prodi.gy/industries/legal-insurance) is a paid annotation tool sold with a lifetime license that runs on your own machines, and Prodigy's legal page shows an example case-law model trained with Prodigy. [Datasaur](https://datasaur.ai/industries/legal) offers LLM-assisted contract review, and Datasaur's legal page states that the deployment runs inside a firm's own infrastructure. E-discovery platforms such as [Relativity](https://www.relativity.com/ediscovery-software/relativityone/) provide review and production tools for litigation. Potato is built around the annotation study, with several reviewers per document, agreement reported per question, gold items with feedback, and a config file that reproduces the task. Potato suits building a labeled legal dataset or measuring how consistently reviewers apply a protocol, and an e-discovery platform suits production review on a live matter.

## Setting up the study with a coding agent

[potato-skill](/docs/getting-started/quick-start-with-a-coding-agent) gives Claude Code, Codex, or Cursor the instructions for designing and building a Potato study. A plain description of the work is enough to start. A request can be this short:

> Set up clause classification for 300 commercial leases split into sections, using our 12 clause types, two attorneys per section, and keyword highlighting for shall, must, and unless.

The agent lists each assumption made along the way, builds the Potato task, and renders the task to check the result. Open the task and label a few items yourself before inviting annotators.

## Frequently asked questions

### How do I measure agreement between annotators on legal documents?

Assign each document or clause to at least two annotators with `num_annotators_per_item`, and turn on `agreement_metrics` to see Krippendorff's alpha for each question on the admin dashboard. For entity spans, the `/admin/iaa` report gives span F1 with exact and with partial matching. The two F1 figures separate disagreement about the label from disagreement about where a span starts and stops. The `reliability` export writes alpha per question with a bootstrap interval. [Agreement for spans](/docs/guides/agreement-for-spans) explains the span measures.

### Can Potato be used for document review?

Yes. Each document is one item, and reviewers answer responsiveness and privilege questions and mark the passages to withhold. With two or more reviewers per document, Potato measures agreement between the reviewers. With `active_learning`, a classifier trained on the labels so far puts the documents with the least certain predictions at the front of the queue. Potato does not collect, process, or produce documents for a legal matter, and Potato fits review research, protocol testing, and dataset building better than production review.

## Further reading

- [Writing annotation guidelines](/docs/guides/writing-annotation-guidelines), for the protocol reviewers work from
- [Gold standards and attention checks](/docs/guides/gold-standards-and-attention-checks), for qualifying expert reviewers before real documents
- [Expert vs. crowd annotation](/docs/guides/expert-vs-crowd-annotation), for when legal training is needed
- [Named entity recognition](/docs/guides/named-entity-recognition), for choosing an entity label set
- [Relation and event extraction](/docs/guides/relation-and-event-extraction), for citation and argument links
- [Document annotation](/docs/guides/document-annotation), for long documents and their layout
- [NLP and text](/use-cases/text-annotation) and [computational social science](/use-cases/computational-social-science), for neighboring fields on this site

## References

1. Hendrycks, Burns, Chen, and Ball (2021). CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review. *NeurIPS 2021*. https://arxiv.org/abs/2103.06268
2. Chalkidis, Jana, Hartung, Bommarito, Androutsopoulos, Katz, and Aletras (2022). LexGLUE: A Benchmark Dataset for Legal Language Understanding in English. *ACL 2022*. https://doi.org/10.18653/v1/2022.acl-long.297