Natural Stories Self-Paced Reading with Comprehension Questions

# Natural Stories: self-paced reading with comprehension questions (illustrative passages)
#
# Dataset: The Natural Stories corpus, Futrell et al. (2020), Language Resources
#   and Evaluation 55, 63-77. Ten stories edited to contain rare syntactic
#   constructions, with self-paced reading times from 181 native English speakers.
# DOI: 10.1007/s10579-020-09503-7
# Dataset URL: https://github.com/languageMIT/naturalstories
# Licence: CC BY-NC-SA 4.0 (LICENSE.txt in the repository). Because of the
#   NonCommercial and ShareAlike terms, NOTHING in this design is copied from the
#   corpus. The ten passages and ten questions in sample-data.json were written for
#   this showcase in the same style (short narrative paragraphs with relative
#   clauses, clefts, topicalized objects, sentential subjects and one NP/Z
#   ambiguity). They are illustrative and are not Natural Stories text.
#
# Citation (Crossref BibTeX):
#   @article{Futrell_2020,
#     title     = {The Natural Stories corpus: a reading-time corpus of English texts containing rare syntactic constructions},
#     volume    = {55},
#     number    = {1},
#     DOI       = {10.1007/s10579-020-09503-7},
#     journal   = {Language Resources and Evaluation},
#     publisher = {Springer Science and Business Media LLC},
#     author    = {Futrell, Richard and Gibson, Edward and Tily, Harry J. and Blank, Idan and Vishnevetsky, Anastasia and Piantadosi, Steven T. and Fedorenko, Evelina},
#     year      = {2020},
#     pages     = {63--77}
#   }
#
# Protocol, quoted from section 3.3 "Self-paced reading data" of the paper:
#   "Text was presented in a dashed moving window display, with masked spaces.
#    For each word, we recorded its reading time (RT) as the amount of time taken
#    by the reader to press the button to advance to the next word."
#   "Each story was accompanied by 6 comprehension questions, where participants
#    chose the correct answer from a set of two."
#
# How this design maps the protocol:
#   Moving window ........ trial_timing.presentation.mode: moving_window. Every
#                          word starts as dashes. Space reveals the next word and
#                          re-masks the previous one. The time between presses is
#                          that word's reading time.
#   Word by word ......... no region_separator is set, so the passage is split on
#                          whitespace into one region per word.
#   Two-choice question .. a radio scheme with two labels, answered with F or J.
#
# Adaptations:
#   - Illustrative text. The passages and questions are original, so this design
#     reproduces the procedure and not the stimuli. Reading times collected here
#     are not comparable with the corpus's.
#   - The paper asked 6 questions per story of about 1000 words. Here every item
#     is a short paragraph (a story is named in the story field) with one question
#     after it, so reading times and answers are paired per paragraph.
#   - Potato's response scheme has one fixed label list, so the two answer choices
#     cannot differ per item. The labels are "First option" and "Second option"
#     (keys F and J), and the question text names the two choices as (F) and (J).
#     A per-item choice list in trial_timing.response is a feature request.
#   - The paper's display recorded line breaks at random widths and ran on Amazon
#     Mechanical Turk. This design uses Potato's own layout and its browser timer.
#   - The paper excluded a pass through a story when fewer than 5 of 6 questions
#     were correct, and excluded reading times under 100 ms or over 3000 ms.
#     Neither rule is applied here. Apply them to the export if you need them.
#
# TRAINING: none. The labels are two-choice comprehension answers and the
#   measurement is reading time per word. No trainer in potato/training/registry.py
#   predicts reading times or comprehension answers, so there is no model_training
#   block.
#
# Reading times are in the trials export (one row per item) and the regions export
# (one row per word):
#   python -m potato.export --config <this file> --format csv --output out/
#
# Run: potato start config.yaml -p 8000

annotation_task_name: "Natural Stories: self-paced reading with comprehension questions"

task_dir: "."
data_files:
  - sample-data.json

item_properties:
  id_key: id
  text_key: text

output_annotation_dir: "annotation_output/"
export_annotation_format: "jsonl"

require_password: false

user_config:
  allow_all_users: true
  users: []

assignment_strategy: fixed_order

trial_timing:
  enabled: true
  fixation_ms: 500
  iti_ms: 500
  show_response_options: true
  presentation:
    mode: moving_window
    field: text
    advance_key: space
    question_field: question
  response:
    schema: comprehension
    keys: {f: "First option", j: "Second option"}

annotation_instructions: |
  You will read a short passage one word at a time. Every word starts as dashes.
  Press the space bar to reveal the next word. Read at your natural pace and press
  space as soon as you have understood each word. After the passage, answer one
  question with F for the first option or J for the second option. The passages
  here are illustrative and were written for this showcase. They are not text from
  the Natural Stories corpus.

annotation_schemes:
  - annotation_type: radio
    name: comprehension
    description: "Answer the comprehension question about the passage you just read"
    labels: ["First option", "Second option"]
    label_requirement:
      required: true