How to Annotate Documents and PDFs

Annotate contracts, reports and papers as documents rather than flattened text, keeping layout, page numbers and cross-page references intact.

Annotate a document on its rendered pages rather than on text extracted from it, because the layout carries meaning that extraction discards and because a reviewer needs to find the label on the page. Flattening a contract into a string loses the page number, the column order, the table structure, and the distinction between a footnote and body text.

Document annotation is the case where the usual text-annotation setup fails. A PDF is a description of marks on pages rather than a sequence of characters, so two different extractors will hand you two different strings from the same file. Multi-column pages interleave, headers and footers appear mid-stream, and tables become runs of whitespace. An annotator labeling that string is labeling an artifact of the extractor, and a reviewer who later opens the original cannot always tell which passage a span referred to.

Position, page number and reference structure

Three properties of a document survive on the page and not in its extracted text. Position tells a reviewer whether a clause is a heading, a footnote, or body copy, and extraction reduces all three to the same undifferentiated run of characters. The page number is how every downstream reader will cite the passage, and it exists only while the pages do. Reference structure records what a passage points at, which matters whenever a claim on one page rests on a table nine pages later.

The third property is the one that most often forces a change of tooling. Reference structure inside a document is a relation between two locations, and a tool that models a document as one flat string has nowhere to put the second location. Legal review, scientific claim verification, and financial-report analysis all turn on that relation.

Three anchor kinds

An annotation on a document is anchored in one of three ways, and the choice follows from the question rather than from preference.

Text spans anchor to selected words and suit questions about wording. A span needs the document to carry a text layer, which born-digital PDFs carry and scans lack.
Region boxes anchor to page coordinates and suit figures, tables, stamps, signatures, and anything a text layer never captured. They work on a scan with no text extraction at all.
Links join two anchors and suit reference structure. A link records a typed, optionally directed relation between two places in the document.

Mixing anchor kinds in one task is normal. A claim-verification task marks claims as text spans, tables as region boxes, and joins them with links.

Configuring it in Potato

Potato's pdf display renders the file with PDF.js and puts the annotation surface on the rendered page. The annotation_mode option selects the anchor kind, taking span, bounding_box, or link. Link mode is the one that answers the reference question, and it records anchors and links under separate schema names so the two come out of the export distinguishable.

yaml

instance_display:
  fields:
    - key: pdf
      type: pdf
      label: "Document"
      display_options:
        annotation_mode: link
        view_mode: scroll
        zoom: page-width
        enable_text_anchors: true
        enable_region_anchors: true
        anchor_schema: pdf_anchors
        link_schema: pdf_links
        anchor_labels:
          - name: claim
            color: "#dc2626"
          - name: figure
            color: "#2563eb"
        link_types:
          - name: refers_to
            directed: true
            allowed_source_labels: [claim]
            allowed_target_labels: [figure]

view_mode: scroll stacks the pages in one container so that a link between page 2 and page 9 can be drawn and seen as a single arc. The paginated alternative shows one page at a time, which reads better on a long document but hides the far end of a cross-page link while the annotator draws it.

allowed_source_labels and allowed_target_labels are what turn a guideline into something the interface enforces. Configured as above, a refers_to link can start only on a claim and end only on a figure, so an annotator cannot record the relation backwards. Guidelines that live only in a document get followed unevenly, and a constraint the tool applies costs nothing to enforce.

A Word or Markdown file uses the document display instead, which preserves heading and paragraph structure and accepts span_target, so a span scheme can point at it directly.

Scanned documents

A scan carries no text layer, so span anchors have nothing to attach to. Two options exist and they suit different corpora. Annotating by region needs no text at all and works immediately. Turning on OCR builds a text layer server-side and makes text anchors possible, at the cost of a slow pass and a Tesseract dependency.

yaml

instance_display:
  fields:
    - key: pdf
      type: pdf
      display_options:
        annotation_mode: link
        ocr: auto

auto runs OCR only when the embedded text layer comes back empty, which is the setting to use on a corpus that mixes born-digital files with scans. Potato reads the ocr option in link mode only.

OCR output is imperfect, and an annotation anchored to a misrecognized word inherits the error. On a corpus where recognition quality is poor, region anchors give a more durable record than text anchors do, because a box drawn around a passage stays correct whatever the characters underneath were read as.

Checking a label against the source

Every annotation comes back carrying the page it sits on, which is what makes the label checkable against the source. A reviewer given a page number and a region can open the original and confirm the judgment. A reviewer given a character offset into a concatenated string generally cannot, because reproducing the offset means reproducing the exact extractor and version that produced it.

Where the approach costs more than it returns is a corpus of short, single-column, born-digital documents with no reference structure, where extraction is reliable and the page number carries nothing. Plain span annotation over extracted text is simpler there and gives the same answer.