# Incentive Payments: Price Lists, BDM Bids, Belief Elicitation and Prolific Bonuses Source: https://www.potatoannotator.com/docs/experiments/incentive-payments *New in v2.12.0* Potato can pay participants according to their answers, as experimental economics studies do. Three annotation types collect the answers: | `annotation_type` | Asks for | Paid by | |---|---|---| | `price_list` | Option A or B in each row of a multiple price list | `mpl_random_row` | | `bdm` | The most the participant would pay for something | `bdm` (Becker, DeGroot & Marschak, 1964) | | `belief_elicitation` | A probability, or a distribution of 100 points over bins | `bsr`, the binarized scoring rule (Hossain & Okui, 2013) | The `incentives:` block says how each scheme is paid. After a participant finishes, Potato draws the random numbers, works out the payoff and writes a bonus file in Prolific's bulk format. It never pays anyone by itself. ## When the draws happen Nothing is drawn while the participant answers. The page never sees a price, a drawn row or a threshold, so none of them can influence an answer. The draws happen when you run the `incentives` export or call the admin route, and only for participants who have finished the study. Each draw is seeded from the user ID, the rule name, the item or page the answer belongs to, and a private `incentives.seed`. Running the computation again on the same answers gives the same draws, adding an item leaves the draws for the other items unchanged, and every draw is written to `bonuses.csv`. ## Price lists The [Holt & Laury (2002)](https://doi.org/10.1257/000282802762024700) list, with at most one switch point: ```yaml annotation_schemes: - annotation_type: price_list name: risk description: For each decision, choose the option you prefer. preset: holt_laury single_switch: true label_requirement: {required: true} ``` The preset has ten rows. Option A pays $2.00 with probability p and $1.60 otherwise, option B pays $3.85 with probability p and $0.10 otherwise, and p runs 0.1, 0.2, ... 1.0. `scale: 10` pays ten times those amounts. A time-preference list sets an amount now against larger amounts later: ```yaml - annotation_type: price_list name: patience description: Which would you rather receive? preset: time_preference amount_now: 10 amounts_later: [10, 10.5, 11, 12, 13, 15] delay_label: in 4 weeks ``` Rows can also be written by hand. `a_pays` and `b_pays` are what the incentive rule pays, either a number for a sure amount or `[[probability, amount], ...]` for a lottery, and without them the list can be shown but not paid: ```yaml - annotation_type: price_list name: insurance description: Choose one option in each row. rows: - {a: "$5 for sure", b: "50% chance of $12, otherwise $0", a_pays: 5, b_pays: [[0.5, 12], [0.5, 0]]} - {a: "$6 for sure", b: "50% chance of $12, otherwise $0", a_pays: 6, b_pays: [[0.5, 12], [0.5, 0]]} ``` | Key | Default | Meaning | |---|---|---| | `preset` | none | `holt_laury` or `time_preference`. Give either `preset` or `rows` | | `rows` | none | At least two `{a, b, a_pays, b_pays, a_after, b_after}`. `a_after` and `b_after` name a payment delay, such as `in 4 weeks` | | `scale` | `1` | Multiplies every amount of a preset. Not allowed with `rows` | | `currency` | `$` | Symbol printed before amounts | | `single_switch` | `false` | Keeps at most one switch from A to B | | `option_labels` | `[Option A, Option B]` | Column headings | | `amount_now`, `amounts_later` | none | `time_preference` only: the sure amount now, and one later amount per row | | `delay_label`, `now_label` | `in 4 weeks`, `today` | `time_preference` only: when each option pays | With `single_switch: true`, choosing B in a row also chooses B in every later row, and choosing A also chooses A in every earlier row. The rows filled in this way flash briefly, so the participant sees what changed. Without it, each row is independent, and the answer records how many times the participant switched. Each row is one radio group, so the arrow keys and Tab work as they do in any radio group. ## BDM bids ```yaml annotation_schemes: - annotation_type: bdm name: voucher description: What is the most you would pay for a voucher worth $3.00? min: 0 max: 5 step: 0.05 input: number instructions: >- You have $5.00 to spend. A price between $0.00 and $5.00 is drawn after the study. If your bid is at least the price, you buy the voucher and pay the price, not your bid; otherwise you keep the $5.00. label_requirement: {required: true} ``` `max` is required, and `min` defaults to 0. `input: slider` shows a slider instead of a number box, with a hollow thumb at the midpoint that fills once the participant moves it, so an untouched slider is not read as a bid. The draw range and the endowment live in the incentive rule, not in the scheme, so write the same numbers into `instructions` so that the participant knows the rule they are bidding under. ## Belief elicitation A probability: ```yaml annotation_schemes: - annotation_type: belief_elicitation name: forecast description: How likely is it that this happens? mode: point left_label: Certainly not right_label: Certainly label_requirement: {required: true} ``` A distribution: ```yaml - annotation_type: belief_elicitation name: rainfall description: How much rain will fall next week? mode: bins bins: ["0-9 mm", "10-19 mm", "20-29 mm", "30 mm or more"] label_requirement: {required: true} ``` In `point` mode, the slider runs from 0 to 100% and starts hollow at 50%, so an untouched slider is not read as a report of 50%. In `bins` mode, the participant types a whole number of points for each bin, and a running total shows how many points are left to place. Until the total is exactly 100, the question reports itself incomplete. ## Configuring payment ```yaml incentives: enabled: true currency: USD seed: "a private string, kept out of the repository" pay: one_random_item # one_random_item | all | sum max_bonus: 6.00 only_finished: true rules: - scheme: risk # a price_list scheme method: mpl_random_row - scheme: voucher # a bdm scheme method: bdm endowment: 5.00 item_value: 3.00 draw: {min: 0, max: 5, step: 0.25} - scheme: forecast # a belief_elicitation scheme method: bsr prize: 2.00 outcome_field: outcome ``` | Key | Default | Meaning | |---|---|---| | `enabled` | `false` | Turns on the `incentives` export and the admin route | | `currency` | `USD` | Label written to `bonuses.csv`. Prolific pays in the study's own currency, so set this to match it | | `seed` | none | Private text mixed into every draw. Without it, anyone with the code could compute the draws before answering, and the export warns when it is missing or still the example's value | | `pay` | `one_random_item` | `one_random_item` pays one decision drawn at random from all of a participant's answered decisions. `all` pays the sum of every decision, and `sum` is another name for `all` | | `max_bonus` | none | Upper limit on one participant's bonus | | `only_finished` | `true` | Draws and pays only for participants who reached the end of the study | | `rules` | required | One entry per paid scheme, each with `scheme` (the scheme's `name`) and `method` | A rule can name a scheme in `annotation_schemes`, answered once per item, or one on a phase page such as `poststudy`, answered once. `potato validate` checks that the scheme exists and has the right type for the method. ### `mpl_random_row` One row of the participant's price list is drawn at random, each row equally likely, and the option they chose in that row is played. A second draw `u` in [0, 1) picks the lottery's outcome, with outcomes laid end to end in the order listed. A list with an unanswered row is not paid, and a list with more than one switch is paid and marked `multiple switching`. ### `bdm` | Key | Meaning | |---|---| | `endowment` | Money the participant holds for this decision, default 0. With `draw`, it must be at least `draw.max`, so a purchase never costs more than the participant was given | | `draw` | `{min, max}` for a price drawn uniformly and rounded to cents. Add `step` to draw one of `min, min + step, ..., max` | | `price_field` | Instead of `draw`, an item field holding a price fixed in advance | | `item_value`, `value_field` | What the object is worth in money to the participant who buys it, as a number or an item field. Default 0, for a physical object handed over separately | A bid at or above the price buys at the price, not at the bid, so payoff = endowment - price + item value. A lower bid does not buy, and payoff = endowment. Because the price does not depend on the bid, bidding anything other than the object's value never helps the participant and sometimes costs them, which is the property that makes the procedure incentive-compatible. ### `bsr` | Key | Meaning | |---|---| | `prize` | The amount paid on a win | | `outcome_field` | The item field that holds what happened: 1 or 0 (or true/false, yes/no) for a point report, and a bin label or 0-based bin index for a bins report | | `outcome` | Instead of `outcome_field`, one value for a question answered once, such as one on a phase page | The prize is paid when the quadratic loss of the report is below a threshold drawn uniformly from [0, K]. For a point report p on a binary event, `loss = (1{event} - p)²` with K = 1. For a bins report, `loss = Σ (1{bin happened} - p_bin)²` with K = 2. A participant who wants the best chance of the prize does best by reporting what they believe, whatever their attitude to risk, because the only payoffs are the prize and nothing. The true outcome has to be supplied before you compute payments, either in the data file under `outcome_field` or filled in once the events have happened. Potato strips `outcome_field` and `price_field` from the item data sent to the page, so keep them out of the item text too. A participant whose drawn decision has no outcome yet is listed as pending and left out of the Prolific file until the outcome is filled in. ## Auditing the draws `bonuses.csv` has one row per paid-for decision, with who it was, which rule paid which answer, the seed, every random number used (as JSON in `draws`), the payoff, whether it counts toward the bonus, the participant's total after `max_bonus`, and a `note` that says why a decision was not paid. To check a payment by hand, read the draws and recompute the payoff from the rules above. Reproducing the draws themselves needs the same answers, the same rule names and the same `incentives.seed`. ## Paying through Prolific Participants need a Prolific ID, so log them in from the study URL with `login: {type: url_direct, url_argument: PROLIFIC_PID}` or with `crowdsourcing: {provider: prolific}`. **From the export:** ```bash python -m potato.export --config config.yaml --format incentives --output ./bonuses/ ``` Upload `bonuses_prolific.csv` in Prolific's bulk bonus dialog. Each line is `,` with two decimals and no header. The export leaves out, with a warning each, participants owed nothing, participants without a Prolific ID, participants who have not finished, and participants whose chosen option pays later. **From the admin API**, with a Prolific token configured: ```bash # Dry run: nothing is sent to Prolific curl -H "X-API-Key: $KEY" http://localhost:8000/admin/api/crowd/incentive_bonuses # Pay: echo the dry run's prolific_total curl -X POST -H "X-API-Key: $KEY" -H "Content-Type: application/json" \ -d '{"pay": true, "expected_total": "12.40", "study_id": ""}' \ http://localhost:8000/admin/api/crowd/incentive_bonuses ``` Paying needs `"pay": true` and an `expected_total` equal to the dry run's `prolific_total`. If an answer changed since the dry run, the totals differ and the request is refused with status 409. Each paid batch is recorded in `/incentive_payments.json`. A batch that includes a participant an earlier batch paid is refused with status 409, which stops a double click or a second payment run from paying the same people twice. To pay only the participants no recorded batch has paid, add `"skip_paid": true` and set `expected_total` to the dry run's `unpaid_total`. To pay a recorded batch again on purpose, add `"repeat": true`. ## Example ```bash python potato/flask_server.py start examples/advanced/incentivized-elicitation/config.yaml -p 8000 ``` The example asks four belief questions, one per item and paid by `bsr` from each item's `outcome`, then a post-study page with a Holt & Laury price list and a BDM bid for a voucher worth $3.00. ## Further Reading - [Experiment Flow](/docs/experiments/experiment-flow) - Conditions and screen-outs - [Running a Crowdsourced Study](/docs/guides/crowdsourcing-prolific-mturk) - Prolific logins, completion codes and bonuses - [Constant Sum](/docs/annotation-types/constant-sum) - Point allocations that are not paid - [Online Behavioral Experiments](/docs/guides/online-behavioral-experiments) - A full study in Potato For implementation details, see the [source documentation](https://github.com/davidjurgens/potato/blob/master/docs/advanced/incentives.md).