Binarized Scoring Rule Belief Elicitation

# Belief Elicitation with the Binarized Scoring Rule (urn task, ten scenarios)
# Based on:
#   David Danz, Lise Vesterlund, and Alistair J. Wilson (2022), "Belief
#   Elicitation and Behavioral Incentive Compatibility." American Economic
#   Review 112(9):2851-2883.
#   DOI: 10.1257/aer.20201248
#   Paper (open copy): http://ungated-research.s3.amazonaws.com/pdf/1688-AER-2022-9-Belief_Elicitation_and_Behavioral_Incentive_Compatibility.pdf
#   Materials: Online Appendix C, https://www.aeaweb.org/doi/10.1257/aer.20201248.appx
#   (datasetUrl points at this appendix, which is a document and not a dataset.)
#   Replication package (openICPSR, DOI 10.3886/e157161v1): the DataCite record
#   lists no licence, so no study data is redistributed here. Every item below
#   is illustrative.
#
# Task: each scenario fills two urns with five balls ("We call the urn with
# more red balls the Red urn, and the one with more blue balls the Blue urn.").
# One urn is selected by comparing a 10-sided die roll to a number X, and the
# participant gives "your best guess that the computer has selected the Red urn
# for the scenario" as a percentage from 0 to 100. Three guesses are made per
# scenario (Guess 1 knowing only X, Guess 2 after one draw, Guess 3 after a
# second draw with replacement). Quoted from Online Appendix C.1, treatment
# "Information" (the baseline treatment).
#
# Payment rule (paper, section 2.1, Information Treatment): for a stated probability q on the Red
# urn, the chance of winning the $8 prize is 1 - (1 - q)^2 if the Red urn was
# selected and 1 - q^2 if the Blue urn was selected. Potato's incentives block
# computes the same win probability for method bsr (potato/incentives.py,
# bsr_win_probability, K = 1 for mode point), after the participant finishes.
# Nothing is drawn or shown while answering, and nothing here pays anyone.
#
# Adaptations:
#   - One belief question per item (a scenario and guess number), not a
#     sequence of three guesses on one screen. The item text restates the urns,
#     X and any draw.
#   - The paper's slider shows the winning chances for the Red and Blue urn as
#     it moves, and shows end-of-period feedback. Potato's belief_elicitation
#     scheme has no such display, so neither is reproduced. The information
#     treatments that vary this incentive information are not reproduced.
#   - Items are illustrative (urn contents, X, draws and outcomes are made up).
#     The paper's own scenarios are not reproduced.
#   - The slider is Potato's point mode with 1-point steps (the paper: "a slider
#     ranging between 0 and 100 percent (with one percentage-point
#     increments)"). Its end captions are worded for the Red and Blue urn.
#   - Pay is one random item (the paper pays one of three decisions in two of
#     ten periods) with the paper's $8 prize. Check the seed before any real use.
#   - Laboratory instructions (room, cash payment) are omitted.
#   - The participant instructions below are abridged from Online Appendix C.1
#     (Information treatment). They keep its wording for the urns, the die
#     roll and the three guesses, and drop details such as the number of
#     scenarios and the 1 to 10 range of X. The scheme's
#     instructions field quotes the C.1 passage on a percentage
#     probability from 0 to 100.
#   - Outcomes are fixed per scenario, as in the paper ("Once the computer
#     selects an urn it is fixed and stays the same for the entire scenario").
#
# Trainer: none. No trainer in potato/training/registry.py consumes
# belief_elicitation, so this design has no model_training block.

annotation_task_name: "Belief Elicitation: Binarized Scoring Rule Urn Task"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "text"
  private_keys: [outcome]

output_annotation_dir: "annotation_output/"
export_annotation_format: "jsonl"

port: 8000

annotation_schemes:
  - annotation_type: belief_elicitation
    name: red_urn_guess
    description: "How likely is it that the selected urn is the Red urn?"
    mode: point
    step: 1
    left_label: "Certainly the Blue urn"
    right_label: "Certainly the Red urn"
    instructions: "Your guess is a percentage probability from 0 to 100, with 0 indicating a 0-out-of-100 chance that the selected urn is the Red urn, and 100 indicating a 100-out-of-100 chance. The payment rule is designed so that you can secure the largest chance of winning the prize by reporting your most-accurate guess."
    label_requirement:
      required: true

incentives:
  enabled: true
  currency: USD
  # Demo seed, public in this repository. Replace it with a private string before paying anyone.
  seed: showcase-demo-seed-replace-before-paying
  pay: one_random_item
  max_bonus: 8.00
  rules:
    - scheme: red_urn_guess
      method: bsr
      prize: 8.00
      outcome_field: outcome   # 1 if the Red urn was selected, else 0; never sent to the page

annotation_instructions: |
  The experiment consists of scenarios. In each scenario the computer fills two
  urns with five balls, either red or blue. We call the urn with more red balls
  the Red urn, and the one with more blue balls the Blue urn. One of these two
  urns is selected to be used in the scenario. Your task is to guess how likely
  it is that the selected urn is the Red urn.

  The computer selects the Red or the Blue urn by rolling a fair 10-sided die
  and comparing it to a number X. If the die roll is less than or equal to X
  then the Red urn is selected. If the die roll is greater than X then the Blue
  urn is selected. The chance the computer selects the Red urn is X-in-10.

  Guess 1: Knowing only the rule X that the computer used to select an urn, you
  provide your first guess that the selected urn is the Red one.
  Guess 2: The computer fairly draws one of the five balls from the selected
  urn. After seeing the color of this ball you provide your second guess.
  Guess 3: After replacing the first-drawn ball back into the selected urn and
  mixing it, the computer fairly draws a second ball from the five. After
  seeing the color of the second ball you provide your third guess.

user_config:
  allow_all_users: true
max_annotations_per_user: 10
num_annotators_per_item: 2