RAFT Few-Shot Harm Classification of NeurIPS Impact Statements

# RAFT: NeurIPS Impact Statement Risks, few-shot harm classification
#
# Paper: Neel Alex, Eli Lifland, Lewis Tunstall, Abhishek Thakur, Pegah Maham,
#   C. Jess Riedel, Emmie Hine, Carolyn Ashurst, Paul Sedille, Alexis Carlier,
#   Michael Noetel, and Andreas Stuhlmuller (2021). "RAFT: A Real-World
#   Few-Shot Text Classification Benchmark." NeurIPS 2021 Datasets and
#   Benchmarks Track. arXiv preprint; no proceedings DOI was found.
#   @misc{alex2021raft,
#     doi = {10.48550/ARXIV.2109.14076},
#     url = {https://arxiv.org/abs/2109.14076},
#     author = {Alex, Neel and Lifland, Eli and Tunstall, Lewis and Thakur, Abhishek and Maham, Pegah and Riedel, C. Jess and Hine, Emmie and Ashurst, Carolyn and Sedille, Paul and Carlier, Alexis and Noetel, Michael and Stuhlmüller, Andreas},
#     title = {RAFT: A Real-World Few-Shot Text Classification Benchmark},
#     publisher = {arXiv},
#     year = {2021}
#   }
# DOI: 10.48550/arXiv.2109.14076
# Dataset: https://huggingface.co/datasets/ought/raft (config
#   neurips_impact_statement_risks); homepage https://raft.elicit.org
#
# Licence: RAFT bundles 11 tasks under different licences. This design uses the
#   NeurIPS impact statement risks (NIS) task, which the paper's appendix A.1
#   gives as "an MIT License" for the underlying impact statement dataset
#   (Ashurst et al.) and Creative Commons Attribution 4.0 International for the
#   derivative RAFT labels. The ten sample items are real items from the NIS
#   training split, redistributed under CC BY 4.0 with attribution to the RAFT
#   authors and the paper authors named in each item's paper_link.
#
# Task: RAFT's NIS task labels each NeurIPS 2020 broader impact statement by
#   whether it mentions a harmful application of the research. Labels, quoted
#   from the dataset card (ClassLabel names):
#     "doesn't mention a harmful application"
#     "mentions a harmful application"
#   Instructions, quoted from the paper's appendix A.3 (task-specific
#   instructions): "Label the impact statement based on whether it mentions a
#   harmful application of the research done in the paper. Make sure the
#   statement is sufficient to conclude there are harmful applications of the
#   research being done, not a past risk that this research is solving."
#   RAFT's annotation record (dataset datasheet): contractors paid by Ought
#   labeled the statements, and a majority vote of three annotators was taken.
#   The paper provides 50 labeled examples per task, "not per class", which is
#   about the number of examples its authors saw people willing to label.
#
# Model training: the trainer is setfit (potato/training/registry.py), which
#   consumes radio, select, likert and multiselect schemes over text. It
#   fine-tunes a sentence-transformer on pairs of labeled statements, then fits
#   a logistic head, and needs at least 8 training items (min_instances).
#   Install with: pip install 'potato-annotation[embeddings]'. It predicts a
#   label for statements nobody has labeled and shows it as a prelabel. It does
#   not serve or deploy a model, and it is not the GPT-3 baseline from the paper.
#   Open /admin/training with the admin key below, pick setfit and the
#   harm_mention scheme, and click Train now.
#
# Adaptations:
#   - One task of RAFT's eleven, with 10 items instead of the 50 labeled and 150
#     unlabeled ones. The released labels stay in the data as released_label for
#     reference and are listed in private_keys, so annotators never receive them.
#   - The annotator-facing task is a radio scheme with the two RAFT label names;
#     the original labeling was done in a spreadsheet.
#   - Items show the paper title and impact statement. RAFT also carries a paper
#     link, which is kept in the data but not displayed.
#   - The instructions are the paper's wording, shown unchanged.

annotation_task_name: "RAFT: NeurIPS Impact Statement Harm Mention"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "impact_statement"
  private_keys: ["released_label"]

instance_display:
  fields:
    - key: paper_title
      type: text
      label: "Paper title"
    - key: impact_statement
      type: text
      label: "Impact statement"

output_annotation_dir: "annotation_output/"
export_annotation_format: "jsonl"

port: 8000
require_password: false
admin_api_key: "demo-admin-key"

annotation_schemes:
  - annotation_type: radio
    name: harm_mention
    description: "Does the impact statement mention a harmful application of the research done in the paper?"
    labels:
      - "doesn't mention a harmful application"
      - "mentions a harmful application"
    sequential_key_binding: true
    label_requirement:
      required: true

annotation_instructions: |
  Label the impact statement based on whether it mentions a harmful application
  of the research done in the paper. Make sure the statement is sufficient to
  conclude there are harmful applications of the research being done, not a
  past risk that this research is solving.

  (Instructions quoted from Alex et al., 2021, appendix A.3.)

model_training:
  enabled: true
  writeback:
    schemas: ["harm_mention"]
    include_unlabeled: true
    min_confidence: 0.6

user_config:
  allow_all_users: true
max_annotations_per_user: 10
num_annotators_per_item: 1