CUAD Contract Clause Extraction as Extractive QA

# CUAD: Contract Clause Extraction as Extractive QA
#
# Dataset: CUAD, the Contract Understanding Atticus Dataset v1 (The Atticus Project).
# 510 contracts from public SEC EDGAR filings, 13,101 labeled clauses, 41 clause categories.
# Dataset page: https://www.atticusprojectai.org/cuad
# Dataset DOI:  10.5281/zenodo.4595826
# Licence:      CC BY 4.0 (Zenodo record 4595826). The excerpts in sample-data.json are
#               real CUAD contract text, attributed to The Atticus Project.
#
# Paper:
#   @misc{hendrycks2021cuad,
#     doi = {10.48550/ARXIV.2103.06268},
#     url = {https://arxiv.org/abs/2103.06268},
#     author = {Hendrycks, Dan and Burns, Collin and Chen, Anya and Ball, Spencer},
#     title = {CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review},
#     publisher = {arXiv},
#     year = {2021}
#   }
# DOI: 10.48550/arXiv.2103.06268
# Published in the NeurIPS 2021 Datasets and Benchmarks Track. The proceedings
# record carries no DOI, so the arXiv DOI is the one cited.
#
# Task (paper, Section 1): "For each of 41 different labels, models must learn to
# highlight the portions of a contract most salient to that label." A contract may
# hold no such portion, as in SQuAD 2.0.
#
# Question wording (CUAD_v1.json, one template filled per category; the Details text
# is the category description from the CUAD README). The paper's Section 4.1 template
# says "this clause" where CUAD_v1.json says "this contract"; this design follows the JSON:
#   Highlight the parts (if any) of this contract related to "<Category>" that should
#   be reviewed by a lawyer. Details: <description>
#
# Categories used in this demo, with their Details text as released:
#   Governing Law              Which state/country's law governs the interpretation of the contract?
#   Anti-Assignment            Is consent or notice required of a party if the contract is assigned to a third party?
#   Termination For Convenience Can a party terminate this contract without cause (solely by giving a notice and allowing a waiting period to expire)?
#   Non-Compete                Is there a restriction on the ability of a party to compete with the counterparty or operate in a certain geography or business or technology sector?
#   Exclusivity                Is there an exclusive dealing commitment with the counterparty? (full text in sample-data.json)
#   Cap On Liability           Does the contract include a cap on liability upon the breach of a party's obligation? This includes time limitation for the counterparty to bring claims or maximum amount for recovery.
#   Audit Rights               Does a party have the right to audit the books, records, or physical locations of the counterparty to ensure compliance with the contract?
#   Insurance                  Is there a requirement for insurance that must be maintained by one party for the benefit of the counterparty?
#
# Labeling process (paper, Section 3): law students labeled the contracts after
# 70-100 hours of training, and "Each annotation was verified by three additional
# annotators."
#
# Adaptations:
#   - 8 of the 41 categories are used.
#   - Contracts are cut to short excerpts (400 to 1,150 characters) from the full
#     contract text, with whitespace runs collapsed to single spaces and non-breaking
#     spaces in the question text replaced by spaces. Answer offsets therefore differ
#     from CUAD_v1.json. Items 009 and 010 are excerpts with no answer for the asked
#     category, which Potato records through the Unanswerable button.
#   - The dataset's own answer spans are kept in each item's reference_answers field
#     for comparison, and listed in item_properties.private_keys so Potato does not
#     send them to the annotator's page. Only answers that occur inside the excerpt
#     are kept (item 005 keeps one of the dataset's two Exclusivity answers).
#   - CUAD collects every answering span per category in a contract. Potato's
#     extractive_qa scheme stores one span per question, so an excerpt with several
#     answers is a lossy fit.
#   - The two-annotator setting stands in for the paper's three-annotator verification
#     and is not the original review workflow.
#
# Training: no model_training block. No trainer in potato/training/registry.py
# consumes extractive_qa answers, so none is claimed here.

annotation_task_name: "CUAD: Contract Clause Extraction"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: id
  text_key: passage
  private_keys: [reference_answers, cuad_qa_id, clause_category, source_contract]

output_annotation_dir: "annotation_output/"
export_annotation_format: "jsonl"

annotation_schemes:
  - annotation_type: extractive_qa
    name: clause_span
    description: "Highlight the part of the contract excerpt that answers the question, or press Unanswerable if the excerpt has none"
    question_field: question
    passage_field: passage
    allow_unanswerable: true
    highlight_color: "#FFEB3B"

user_config:
  allow_all_users: true
  users: []

num_annotators_per_item: 2

annotation_instructions: |
  Each item shows a clause-review question and an excerpt from a commercial contract.
  Select the text in the excerpt that is responsive to the question. If the
  excerpt has no such text, press Unanswerable.