Incentive Payments: Price Lists, BDM Bids, Belief Elicitation and Prolific Bonuses

New in v2.12.0

Potato can pay participants according to their answers, as experimental economics studies do. Three annotation types collect the answers:

The incentives: block says how each scheme is paid. After a participant finishes, Potato draws the random numbers, works out the payoff and writes a bonus file in Prolific's bulk format. It never pays anyone by itself.

When the draws happen

Nothing is drawn while the participant answers. The page never sees a price, a drawn row or a threshold, so none of them can influence an answer. The draws happen when you run the incentives export or call the admin route, and only for participants who have finished the study.

Each draw is seeded from the user ID, the rule name, the item or page the answer belongs to, and a private incentives.seed. Running the computation again on the same answers gives the same draws, adding an item leaves the draws for the other items unchanged, and every draw is written to bonuses.csv.

Price lists

The Holt & Laury (2002) list, with at most one switch point:

annotation_schemes:
  - annotation_type: price_list
    name: risk
    description: For each decision, choose the option you prefer.
    preset: holt_laury
    single_switch: true
    label_requirement: {required: true}

The preset has ten rows. Option A pays $2.00 with probability p and $1.60 otherwise, option B pays $3.85 with probability p and $0.10 otherwise, and p runs 0.1, 0.2, ... 1.0. scale: 10 pays ten times those amounts.

A time-preference list sets an amount now against larger amounts later:

  - annotation_type: price_list
    name: patience
    description: Which would you rather receive?
    preset: time_preference
    amount_now: 10
    amounts_later: [10, 10.5, 11, 12, 13, 15]
    delay_label: in 4 weeks

Rows can also be written by hand. a_pays and b_pays are what the incentive rule pays, either a number for a sure amount or [[probability, amount], ...] for a lottery, and without them the list can be shown but not paid:

  - annotation_type: price_list
    name: insurance
    description: Choose one option in each row.
    rows:
      - {a: "$5 for sure", b: "50% chance of $12, otherwise $0",
         a_pays: 5, b_pays: [[0.5, 12], [0.5, 0]]}
      - {a: "$6 for sure", b: "50% chance of $12, otherwise $0",
         a_pays: 6, b_pays: [[0.5, 12], [0.5, 0]]}

With single_switch: true, choosing B in a row also chooses B in every later row, and choosing A also chooses A in every earlier row. The rows filled in this way flash briefly, so the participant sees what changed. Without it, each row is independent, and the answer records how many times the participant switched. Each row is one radio group, so the arrow keys and Tab work as they do in any radio group.

BDM bids

annotation_schemes:
  - annotation_type: bdm
    name: voucher
    description: What is the most you would pay for a voucher worth $3.00?
    min: 0
    max: 5
    step: 0.05
    input: number
    instructions: >-
      You have $5.00 to spend. A price between $0.00 and $5.00 is drawn after
      the study. If your bid is at least the price, you buy the voucher and pay
      the price, not your bid; otherwise you keep the $5.00.
    label_requirement: {required: true}

max is required, and min defaults to 0. input: slider shows a slider instead of a number box, with a hollow thumb at the midpoint that fills once the participant moves it, so an untouched slider is not read as a bid. The draw range and the endowment live in the incentive rule, not in the scheme, so write the same numbers into instructions so that the participant knows the rule they are bidding under.

Belief elicitation

A probability:

annotation_schemes:
  - annotation_type: belief_elicitation
    name: forecast
    description: How likely is it that this happens?
    mode: point
    left_label: Certainly not
    right_label: Certainly
    label_requirement: {required: true}

A distribution:

  - annotation_type: belief_elicitation
    name: rainfall
    description: How much rain will fall next week?
    mode: bins
    bins: ["0-9 mm", "10-19 mm", "20-29 mm", "30 mm or more"]
    label_requirement: {required: true}

In point mode, the slider runs from 0 to 100% and starts hollow at 50%, so an untouched slider is not read as a report of 50%. In bins mode, the participant types a whole number of points for each bin, and a running total shows how many points are left to place. Until the total is exactly 100, the question reports itself incomplete.

Configuring payment

incentives:
  enabled: true
  currency: USD
  seed: "a private string, kept out of the repository"
  pay: one_random_item            # one_random_item | all | sum
  max_bonus: 6.00
  only_finished: true
  rules:
    - scheme: risk                # a price_list scheme
      method: mpl_random_row
    - scheme: voucher             # a bdm scheme
      method: bdm
      endowment: 5.00
      item_value: 3.00
      draw: {min: 0, max: 5, step: 0.25}
    - scheme: forecast            # a belief_elicitation scheme
      method: bsr
      prize: 2.00
      outcome_field: outcome

A rule can name a scheme in annotation_schemes, answered once per item, or one on a phase page such as poststudy, answered once. potato validate checks that the scheme exists and has the right type for the method.

mpl_random_row

One row of the participant's price list is drawn at random, each row equally likely, and the option they chose in that row is played. A second draw u in [0, 1) picks the lottery's outcome, with outcomes laid end to end in the order listed. A list with an unanswered row is not paid, and a list with more than one switch is paid and marked multiple switching.

bdm

A bid at or above the price buys at the price, not at the bid, so payoff = endowment - price + item value. A lower bid does not buy, and payoff = endowment. Because the price does not depend on the bid, bidding anything other than the object's value never helps the participant and sometimes costs them, which is the property that makes the procedure incentive-compatible.

bsr

The prize is paid when the quadratic loss of the report is below a threshold drawn uniformly from [0, K]. For a point report p on a binary event, loss = (1{event} - p)² with K = 1. For a bins report, loss = Σ (1{bin happened} - p_bin)² with K = 2. A participant who wants the best chance of the prize does best by reporting what they believe, whatever their attitude to risk, because the only payoffs are the prize and nothing.

The true outcome has to be supplied before you compute payments, either in the data file under outcome_field or filled in once the events have happened. Potato strips outcome_field and price_field from the item data sent to the page, so keep them out of the item text too. A participant whose drawn decision has no outcome yet is listed as pending and left out of the Prolific file until the outcome is filled in.

Auditing the draws

bonuses.csv has one row per paid-for decision, with who it was, which rule paid which answer, the seed, every random number used (as JSON in draws), the payoff, whether it counts toward the bonus, the participant's total after max_bonus, and a note that says why a decision was not paid. To check a payment by hand, read the draws and recompute the payoff from the rules above. Reproducing the draws themselves needs the same answers, the same rule names and the same incentives.seed.

Paying through Prolific

Participants need a Prolific ID, so log them in from the study URL with login: {type: url_direct, url_argument: PROLIFIC_PID} or with crowdsourcing: {provider: prolific}.

From the export:

python -m potato.export --config config.yaml --format incentives --output ./bonuses/

Upload bonuses_prolific.csv in Prolific's bulk bonus dialog. Each line is <participant_id>,<amount> with two decimals and no header. The export leaves out, with a warning each, participants owed nothing, participants without a Prolific ID, participants who have not finished, and participants whose chosen option pays later.

From the admin API, with a Prolific token configured:

# Dry run: nothing is sent to Prolific
curl -H "X-API-Key: $KEY" http://localhost:8000/admin/api/crowd/incentive_bonuses
 
# Pay: echo the dry run's prolific_total
curl -X POST -H "X-API-Key: $KEY" -H "Content-Type: application/json" \
     -d '{"pay": true, "expected_total": "12.40", "study_id": "<study id>"}' \
     http://localhost:8000/admin/api/crowd/incentive_bonuses

Paying needs "pay": true and an expected_total equal to the dry run's prolific_total. If an answer changed since the dry run, the totals differ and the request is refused with status 409.

Each paid batch is recorded in <task_dir>/incentive_payments.json. A batch that includes a participant an earlier batch paid is refused with status 409, which stops a double click or a second payment run from paying the same people twice. To pay only the participants no recorded batch has paid, add "skip_paid": true and set expected_total to the dry run's unpaid_total. To pay a recorded batch again on purpose, add "repeat": true.

Example

python potato/flask_server.py start examples/advanced/incentivized-elicitation/config.yaml -p 8000

The example asks four belief questions, one per item and paid by bsr from each item's outcome, then a post-study page with a Holt & Laury price list and a BDM bid for a voucher worth $3.00.

Further Reading

• Experiment Flow - Conditions and screen-outs
• Running a Crowdsourced Study - Prolific logins, completion codes and bonuses
• Constant Sum - Point allocations that are not paid
• Online Behavioral Experiments - A full study in Potato

For implementation details, see the source documentation.