LegalEval - Legal Document Analysis

# LegalEval - Understanding Legal Texts
# Based on Modi et al., SemEval-2023 Task 6
# Paper: https://aclanthology.org/2023.semeval-1.318/
# Dataset: https://sites.google.com/view/legaleval/home
#
# SemEval-2023 Task 6 (LegalEval) has three sub-tasks:
#   Task A - Rhetorical Roles Labeling: assign a rhetorical role to each
#            sentence of a legal judgment (e.g., PREAMBLE, FAC, ISSUE, RATIO).
#   Task B - Legal Named Entity Recognition (L-NER): identify legal entities
#            such as courts, petitioners, respondents, judges, statutes, etc.
#   Task C - Court Judgment Prediction with Explanation (CJPE): predict the
#            binary case outcome and supply supporting explanation sentences.
#
# This showcase config demonstrates Task A (rhetorical role, radio) and
# Task B (L-NER, span) on sentence-level legal passages.

annotation_task_name: "LegalEval - Legal Document Analysis"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "text"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

port: 8000
server_name: localhost

annotation_schemes:
  - annotation_type: radio
    name: rhetorical_role
    description: "Task A - Assign the rhetorical role of this sentence in the legal judgment"
    labels:
      - "PREAMBLE"
      - "FAC"
      - "RLC"
      - "ISSUE"
      - "ARG_PETITIONER"
      - "ARG_RESPONDENT"
      - "ANALYSIS"
      - "STA"
      - "PRE_RELIED"
      - "PRE_NOT_RELIED"
      - "RATIO"
      - "RPC"
      - "NONE"
    tooltips:
      "PREAMBLE": "Metadata / header of the judgment (court, parties, judges)"
      "FAC": "Facts - events or circumstances that led to the case"
      "RLC": "Ruling by Lower Court"
      "ISSUE": "Legal issues framed by the court"
      "ARG_PETITIONER": "Arguments advanced by the petitioner/appellant"
      "ARG_RESPONDENT": "Arguments advanced by the respondent"
      "ANALYSIS": "Court's analysis / reasoning and application of law to facts"
      "STA": "Statute - relevant statutory provisions cited"
      "PRE_RELIED": "Precedent relied upon by the court"
      "PRE_NOT_RELIED": "Precedent cited but not relied upon"
      "RATIO": "Ratio of the decision - the reason for the final decision"
      "RPC": "Ruling by Present Court - final decision / order"
      "NONE": "No rhetorical role applies"

  - annotation_type: span
    name: legal_entities
    description: "Task B (L-NER) - Highlight and label all legal named entities in the text"
    labels:
      - "COURT"
      - "PETITIONER"
      - "RESPONDENT"
      - "JUDGE"
      - "LAWYER"
      - "DATE"
      - "ORG"
      - "GPE"
      - "STATUTE"
      - "PROVISION"
      - "PRECEDENT"
      - "CASE_NUMBER"
      - "WITNESS"
      - "OTHER_PERSON"

annotation_instructions: |
  You will see a sentence from an Indian court judgment.
  1. Task A (Rhetorical Role): assign the single rhetorical role that best
     describes the function of this sentence in the judgment (e.g., PREAMBLE,
     FAC, ISSUE, ANALYSIS, RATIO, RPC).
  2. Task B (L-NER): highlight every legal named entity in the sentence and
     assign the appropriate label. Entity types include COURT, PETITIONER,
     RESPONDENT, JUDGE, LAWYER, DATE, ORG, GPE, STATUTE, PROVISION, PRECEDENT,
     CASE_NUMBER, WITNESS, and OTHER_PERSON.

html_layout: |
  <div style="padding: 15px; max-width: 800px; margin: auto;">
    <div style="background: #ecfdf5; border: 1px solid #a7f3d0; border-radius: 8px; padding: 12px; margin-bottom: 12px;">
      <strong style="color: #065f46;">Case Type:</strong>
      <span style="font-size: 15px; margin-left: 8px;">{{case_type}}</span>
    </div>
    <div style="background: #f0f9ff; border: 1px solid #bae6fd; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
      <strong style="color: #0369a1;">Legal Text:</strong>
      <p style="font-size: 16px; line-height: 1.7; margin: 8px 0 0 0;">{{text}}</p>
    </div>
  </div>

allow_all_users: true
instances_per_annotator: 50
annotation_per_instance: 2
allow_skip: true
skip_reason_required: false