Aya Dataset Writing and Editing Instruction-Completion Pairs

# Aya Dataset - Writing and Editing Instruction-Completion Pairs
#
# Paper:   Singh et al., "Aya Dataset: An Open-Access Collection for Multilingual
#          Instruction Tuning", ACL 2024 (Volume 1: Long Papers), pp. 11521-11567.
#          https://aclanthology.org/2024.acl-long.620/
# DOI:     10.18653/v1/2024.acl-long.620
# Dataset: https://huggingface.co/datasets/CohereLabs/aya_dataset
# Licence: Apache-2.0 (dataset card, and stated in the paper). The ten sample
#          items are real Aya Dataset pairs, redistributed under that licence;
#          each carries its row number in the dataset's train split.
#
# BibTeX (from the ACL Anthology):
#   @inproceedings{singh-etal-2024-aya,
#     title = "Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning",
#     author = "Singh, Shivalika and Vargus, Freddie and D'souza, Daniel and
#       Karlsson, Borje F. and others (33 authors)",
#     booktitle = "Proceedings of the 62nd Annual Meeting of the Association for
#       Computational Linguistics (Volume 1: Long Papers)",
#     year = "2024", address = "Bangkok, Thailand",
#     url = "https://aclanthology.org/2024.acl-long.620/",
#     doi = "10.18653/v1/2024.acl-long.620", pages = "11521--11567"
#   }
#
# The Aya Annotation Platform (Section 2.1) offered three tasks, "following the
# find-fix-verify paradigm": "Writing new examples from scratch (original
# annotations), editing existing examples to improve the quality and
# comprehensiveness (re-annotations), and giving feedback on the quality of
# existing contributions (annotation feedback)". This design reproduces the
# re-annotation task: read a prompt-completion pair, rate it, and edit the
# prompt and the completion in the annotator's own language.
#
# Guidelines (Appendix C.2), quoted:
#   1. "No grammatical or spelling mistakes in both the prompt and completion text."
#   2. "The prompt provides clear instructions on what the task is."
#   3. "The completion answers the prompt correctly. Both the prompts and
#      completions should be in full sentences and coherent, with reasonable length."
#   4. "For original annotations, the prompts and completions should not be
#      generated from other language models."
# Rating (Appendix C.2): "annotators rated the quality of existing prompt and
# completion pairs by choosing either the thumbs-up or thumbs-down option. If the
# provided prompt and completion pair were already of good quality according to
# the criteria above, then annotators rated them with thumbs up and moved ahead
# without editing. Overall, annotators were encouraged to re-annotate the
# completions, in particular by adding more details and context to them since
# many of them were often short one-word answers."
#
# Adaptations:
#   - Only the re-annotation task is built. The annotation-feedback task needs
#     the original and the edited pair side by side and is left out.
#   - The released dataset holds each pair's final text, not the text before
#     editing. The ten items are released pairs shown as the starting point, with
#     the original dataset's "annotation_type" and train-split row recorded.
#   - The prompt and the completion use text_edit schemes (source_field: prompt
#     and source_field: completion). Each opens prefilled with the pair's text,
#     so an annotator edits the text in place, and Potato shows a word diff, the
#     word and character edit distance, and a reset button. An untouched box is
#     recorded as accepted. The paper mentions edit distance only as the
#     Levenshtein acceptance threshold (Section 2.3), not as an interface element,
#     so this display is Potato's and not a copy of the platform's screen.
#   - Points, the leaderboard and the Aya Score are not reproduced.
#
# Training: none. A text_edit answer is stored as a JSON object holding the
# edited text, the original text and the edit distances. The llm-sft trainer
# (kinds text and trajectory) reads a text scheme's stored answer as the text to
# learn, and no code under potato/training unwraps the edited text from that
# object, so this design cannot say that llm-sft learns the edited completion
# from a text_edit scheme. A plain text scheme would train, but it opens empty,
# so the edit-in-place display was kept and the model_training block was dropped.

annotation_task_name: "Aya Dataset: Writing and Editing Instruction-Completion Pairs"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: id
  text_key: prompt

output_annotation_dir: "annotation_output/"
export_annotation_format: "jsonl"

require_password: false

instance_display:
  fields:
    - key: language
      type: text
      label: "Language"
    - key: prompt
      type: text
      label: "Prompt"
    - key: completion
      type: text
      label: "Completion"

annotation_schemes:
  # Step 1: thumbs-up or thumbs-down on the pair
  - annotation_type: radio
    name: pair_rating
    description: "Rate this prompt and completion. Thumbs up means the pair already meets the guidelines (correct spelling and grammar, a prompt with a clear instruction, a correct completion in full sentences). Thumbs down means it needs editing."
    labels:
      - "Thumbs up"
      - "Thumbs down"

  # Step 2: edit the prompt, prefilled with the pair's prompt
  - annotation_type: text_edit
    name: edited_prompt
    description: "Edit the prompt if needed, in the same language, so that it gives clear instructions. Leave it unchanged to keep it."
    source_field: prompt

  # Step 3: edit the completion, prefilled with the pair's completion
  - annotation_type: text_edit
    name: edited_completion
    description: "Edit the completion so that it answers the prompt correctly, in the prompt's language, in full sentences. Add detail and context where the answer is short."
    source_field: completion

annotation_instructions: |
  You are shown a prompt and a completion from the Aya Dataset, an open instruction-tuning
  dataset written by fluent speakers of 65 languages. Work in the language of the pair.
  1. Rate the pair with a thumbs up or thumbs down. A good pair has no grammar or spelling
     mistakes, a prompt that clearly says what the task is, and a correct, coherent completion
     in full sentences of reasonable length.
  2. If the prompt needs work, edit it in the box, which starts with the pair's prompt. Leave
     it unchanged otherwise.
  3. Edit the completion, which also starts with the pair's text, into the completion you would
     want a model to give. Short one-word answers are the usual thing to improve with more
     detail and context.
  Do not paste text generated by another language model.

user_config:
  allow_all_users: true
max_annotations_per_user: 10