Skip to content
Guides9 min read

Legal Document Annotation Best Practices

Annotate contracts, court filings and regulatory text in Potato: section-level items, clause and entity schemes, keyword highlighting, and quality control for expert annotators.

Potato Team

Split long legal documents into section-level items before they reach the annotation tool, label clauses and entities with separate schemes, and staff the task with annotators who know the domain. The single biggest quality decision is where you cut the documents, because a clause annotated without its definitions and cross-references is a clause annotated wrong.

Legal text breaks the assumptions most annotation setups are built on. Documents run to hundreds of pages, the vocabulary is specialized enough that untrained crowd workers cannot apply the guidelines, and meaning depends on cross-references to sections that may be fifty pages away. A span boundary that is nearly right changes what a clause obliges a party to do, so the usual tolerance for approximate boundaries does not hold.

Preparing documents before Potato sees them

Segmentation is a data preparation step rather than a configuration setting. Potato annotates the items you give it, so a contract becomes annotatable by being split into sections in your own pipeline, with each section written out as one item carrying the metadata an annotator needs:

json
{
  "id": "contract_001_section_3.2",
  "text": "The Licensor grants to the Licensee a non-exclusive, non-transferable license...",
  "section_number": "3.2",
  "section_title": "License Grant",
  "document_id": "contract_001"
}

Cut on the document's own structure, usually numbered headings, rather than on a fixed token count, because a clause split down the middle cannot be labeled by anyone. Carry the section number and title on every item. An annotator deciding whether an obligation is conditional needs to know which section they are in, and a section number in the item is also what lets you reassemble the labels into document-level annotations afterward.

Declare the fields you want displayed with instance_display, which separates what annotators see from what they are asked to label:

yaml
instance_display:
  fields:
    - key: section_title
      type: text
    - key: text
      type: text

Long contracts also arrive as PDFs. The pdf and document display types render them directly, which avoids an extraction step that would strip the layout cues experienced readers rely on.

Entity and clause schemes

Legal annotation usually collects two different things about the same passage. Entity spans mark who and what the clause refers to, and a clause label classifies the passage as a whole. Keeping them in separate schemes means an annotator can mark five entities and one clause type without the two interfering.

Spans carry the entities. Each label needs a name, and a tooltip pays for itself on a task where two annotators may read "defined term" differently:

yaml
annotation_schemes:
  - annotation_type: span
    name: legal_entities
    description: "Mark the parties, defined terms, dates, amounts and obligations."
    labels:
      - name: PARTY
        color: "#FECACA"
        tooltip: "Contracting parties, including Licensor, Licensee and named companies"
      - name: DEFINED_TERM
        color: "#FDE68A"
        tooltip: "A term given a specific meaning by the definitions section"
      - name: DATE
        color: "#BBF7D0"
        tooltip: "Dates and time periods, including notice and cure periods"
      - name: MONETARY
        color: "#C4B5FD"
        tooltip: "Amounts, fees, penalties and caps"
      - name: OBLIGATION
        color: "#BFDBFE"
        tooltip: "Language creating a duty, such as shall, must or agrees to"
      - name: CONDITION
        color: "#FED7AA"
        tooltip: "Language making a duty conditional, such as if, unless or provided that"
      - name: REFERENCE
        color: "#E0E7FF"
        tooltip: "A cross-reference to another section or document"

The clause type is a single choice, so it belongs in a radio scheme. Keep the list short enough that annotators can hold it in their heads, and check the label distribution in a pilot, since rarely-used categories in a long list are where the disagreements concentrate:

yaml
  - annotation_type: radio
    name: clause_type
    description: "What type of clause is this?"
    labels:
      - Definitions
      - Grant of rights
      - Consideration and payment
      - Term and termination
      - Representations and warranties
      - Indemnification
      - Limitation of liability
      - Confidentiality
      - Intellectual property
      - Dispute resolution
      - Boilerplate

Where a clause can carry more than one obligation, a multiselect collects them all rather than forcing a choice, and a likert with a free-text companion captures risk judgments along with the reason:

yaml
  - annotation_type: multiselect
    name: obligation_type
    description: "Which kinds of obligation does this clause create?"
    labels:
      - Performance
      - Payment
      - Restriction
      - Conditional obligation
      - Warranty
 
  - annotation_type: likert
    name: risk
    description: "How much risk does this clause carry for the receiving party?"
    size: 5
    min_label: "Low risk"
    max_label: "High risk"
 
  - annotation_type: text
    name: risk_notes
    description: "Explain the risk factors you identified."
    rows: 4
    min_chars: 20

The min_chars floor on the free-text scheme is doing quality work rather than formatting work. A risk rating with no reason cannot be adjudicated later, and requiring 20 characters is enough to stop a single-word entry without turning the field into a burden.

Court filings

Court documents want a different entity set from contracts, since the useful units are the case metadata and the structure of the argument. Run them as their own task rather than stretching a contract scheme to cover them:

yaml
annotation_schemes:
  - annotation_type: span
    name: case_entities
    description: "Mark case metadata and the parties to the proceeding."
    labels:
      - name: CASE_NUMBER
        tooltip: "The docket or case identifier"
      - name: COURT
        tooltip: "Court name and jurisdiction"
      - name: JUDGE
      - name: PLAINTIFF
      - name: DEFENDANT
      - name: ATTORNEY
      - name: LEGAL_CITATION
        tooltip: "Citations to cases, statutes and regulations"
      - name: RULING
 
  - annotation_type: span
    name: argument_structure
    description: "Mark the claim, its support, and the court's conclusion."
    labels:
      - name: CLAIM
        color: "#FECACA"
      - name: PREMISE
        color: "#BBF7D0"
      - name: EVIDENCE
        color: "#BFDBFE"
      - name: REBUTTAL
        color: "#FED7AA"
      - name: CONCLUSION
        color: "#E0E7FF"

Highlighting the language that carries weight

Obligation and condition words are easy to skim past on the twentieth contract of a shift, and they are exactly the words that decide the label. Potato highlights terms from a keyword file, which is a CSV whose keyword column is required and whose label, schema and color columns are optional. A * matches any run of word characters:

text
keyword,label,schema,color
shall,OBLIGATION,legal_entities,#FEE2E2
must,OBLIGATION,legal_entities,#FEE2E2
agrees to,OBLIGATION,legal_entities,#FEE2E2
may,CONDITION,legal_entities,#DBEAFE
unless,CONDITION,legal_entities,#DBEAFE
provided that,CONDITION,legal_entities,#DBEAFE
notwithstand*,CONDITION,legal_entities,#DBEAFE

Point the config at the file, with the path relative to task_dir:

yaml
keyword_highlights_file: data/legal_terms.csv

Highlighting changes annotator behavior, which is a problem if you are measuring how well people find obligations. keyword_highlight_settings addresses it by showing only a fraction of matches and mixing in distractors, so the annotations record what annotators found rather than what the keyword list flagged:

yaml
keyword_highlight_settings:
  keyword_probability: 0.8       # show 80% of matched keywords
  random_word_probability: 0.05  # highlight ~5% of other words as distractors

Quality control for an expert task

Legal annotation inverts the usual crowdsourcing economics. Annotators are scarce and expensive, so the controls that matter are the ones that catch a misunderstanding early rather than the ones that screen a large pool.

Put the qualification up front as a training phase, where annotators work through pre-labeled examples with feedback before they touch real data. Gold standards then keep measuring accuracy during the run, and turning feedback on makes them a continuing training signal rather than a silent audit:

yaml
num_annotators_per_item: 3
 
gold_standards:
  enabled: true
  items_file: data/gold_clauses.json
  mode: mixed
  frequency: 20
  accuracy:
    min_threshold: 0.85
    evaluation_count: 10
  feedback:
    show_correct_answer: true
    show_explanation: true

The 0.85 accuracy floor is higher than a general-purpose task would use, and it is justified here by the cost of an error rather than by any property of the annotators. Three annotators per item is worth the expense for the same reason. What you do with the disagreements is the other half of the design. Route contested clauses to an attorney for adjudication rather than resolving them by majority vote, because on a legal task the majority of non-experts is not evidence.

Attention checks are less useful here than on a crowd task. A domain expert is unlikely to be clicking at random, and the failure mode you are actually guarding against is consistent misreading of a guideline, which gold items catch and attention checks do not.

A complete configuration

yaml
annotation_task_name: "Contract Clause Analysis"
task_description: "Classify contract clauses and mark the entities they refer to."
 
data_files:
  - data/contract_sections.json
 
item_properties:
  id_key: id
  text_key: text
 
instance_display:
  fields:
    - key: section_title
      type: text
    - key: text
      type: text
 
keyword_highlights_file: data/legal_terms.csv
 
keyword_highlight_settings:
  keyword_probability: 0.8       # show 80% of matched keywords
  random_word_probability: 0.05  # highlight ~5% of other words as distractors
 
annotation_schemes:
  - annotation_type: radio
    name: clause_type
    description: "What type of clause is this?"
    labels:
      - Grant of rights
      - Consideration and payment
      - Term and termination
      - Indemnification
      - Limitation of liability
      - Confidentiality
      - Other
 
  - annotation_type: span
    name: legal_entities
    description: "Mark the parties, defined terms, dates, amounts and obligations."
    labels:
      - name: PARTY
        color: "#FECACA"
      - name: DEFINED_TERM
        color: "#FDE68A"
      - name: MONETARY
        color: "#C4B5FD"
      - name: DATE
        color: "#BBF7D0"
      - name: OBLIGATION
        color: "#BFDBFE"
 
  - annotation_type: likert
    name: risk
    description: "How much risk does this clause carry for the receiving party?"
    size: 5
    min_label: "Low risk"
    max_label: "High risk"
 
  - annotation_type: text
    name: issues
    description: "Note any unusual or problematic language."
    rows: 4
    min_chars: 20
 
num_annotators_per_item: 3
 
gold_standards:
  enabled: true
  items_file: data/gold_clauses.json
  mode: mixed
  frequency: 20
  accuracy:
    min_threshold: 0.85
    evaluation_count: 10
 
agreement_metrics:
  enabled: true
  min_overlap: 2
 
output_annotation_dir: annotation_output

Writing the guidelines

Legal guidelines carry more weight than the schemes do, because the schemes only name the categories and the guidelines decide what falls into them. Four things belong in them.

  • Scope. Which document types, which jurisdictions, and which governing law. A clause type that means one thing under English law may not transfer.
  • A glossary. Define each label in the annotators' own working vocabulary, and say which ordinary words the label does not cover.
  • Boundary rules. State where a span starts and stops. Does an obligation span include the conditional clause that triggers it, or only the duty itself? Answer it once, with an example, and the agreement number will move.
  • Cross-reference policy. Say when a reference to another section is annotated and when it is ignored, since this is the most common source of silent disagreement in contract tasks.

Agreement on spans is measured differently from agreement on labels, because two annotators can pick the same entity type and disagree about where it starts. Agreement for spans covers exact and partial matching, and the /admin/iaa report gives span_f1_exact alongside span_f1_partial so you can see which of the two is failing.

Further reading