Best-Worst Scaling

New in v2.3.0

Best-Worst Scaling (BWS), also known as MaxDiff, shows an annotator a tuple of items, usually four, and asks for the best and the worst by some criterion. Two clicks per tuple produce more information than a rating would, and because the judgment is comparative it sidesteps the problem that one annotator's 7 is another's 5.

BWS is worth the setup when:

• Direct numerical ratings suffer from scale-usage differences between annotators
• You need a reliable ranking over hundreds or thousands of items
• The quality dimension is inherently relative, such as which translation reads most fluently

Best-worst scaling interface for comparative annotation in Potato

Configuration

A BWS task has two parts. The scheme says what the annotator is asked, and the top-level bws_config block says how tuples are built and scored.

bws_config:
  tuple_size: 4              # items shown at once
  min_item_appearances: 8    # how often each item must appear across tuples
  seed: 42
  scoring:
    method: counting         # counting | bradley_terry | plackett_luce
 
annotation_schemes:
  - annotation_type: bws
    name: fluency
    description: "Select the BEST and WORST translation by fluency"
    tuple_size: 4
    best_description: "Most Fluent"
    worst_description: "Least Fluent"
    sequential_key_binding: true
    label_requirement:
      required: true

The scheme takes tuple_size, best_description, worst_description, sequential_key_binding and label_requirement, and nothing else. Anything about generating or scoring tuples belongs in bws_config.

A bws scheme without a bws_config block renders a question over an empty item list, which is the usual reason a BWS task comes up blank.

bws_config Options

Raising min_item_appearances tightens every item's score at the cost of more comparisons. It is the dial to turn when the scores come out noisy.

Data Format

Each row of the data file is one item, and Potato samples tuples from the pool.

{"id": "sys_a", "text": "Le chat s'est assis sur le tapis."}
{"id": "sys_b", "text": "Le chat a assis sur le tapis."}
{"id": "sys_c", "text": "Le chat etait assis sur le mat."}
{"id": "sys_d", "text": "Le chat se tenait sur le tapis."}

Tuple generation balances the design: every item appears about equally often, and the pairs that need comparing co-occur often enough for the scores to be comparable. Setting seed makes that assignment reproducible.

Scoring

Scoring runs after collection, from the command line:

python -m potato.bws_scoring --config config.yaml --method counting

The result is written to bws_scores.tsv in the output directory unless --output names another path. The three flags are --config, --method and --output.

Counting

The default, and the transparent one. An item's score is the share of times it was picked best minus the share of times it was picked worst, so scores run from -1.0 to +1.0. It needs no extra dependencies and it is the method to report when you want a reader to be able to recompute it by hand.

python -m potato.bws_scoring --config config.yaml --method counting --output scores.tsv

Bradley-Terry

Fits a model over the pairwise comparisons each tuple implies: the best item beats every other item in its tuple, and every other item beats the worst. It handles sparse data better than counting.

python -m potato.bws_scoring --config config.yaml --method bradley_terry

Plackett-Luce

Treats each judgment as two choices, the best from the whole tuple and then the worst from the items left, which is the MaxDiff model of how best-worst judgments are made. It fits a utility per item by maximum likelihood. Before 2.10.2 it fitted the same pairs as Bradley-Terry and returned identical scores, so rescore with 2.10.2 or later if you reported it.

python -m potato.bws_scoring --config config.yaml --method plackett_luce

Choosing a Method

bradley_terry needs pip install choix. plackett_luce uses scipy, which Potato already installs.

Several Dimensions at Once

Two BWS schemes over the same items collect two judgments per tuple, which costs one extra pair of clicks rather than a second pass over the corpus:

annotation_schemes:
  - annotation_type: bws
    name: fluency
    description: "Select the BEST and WORST by fluency"
    tuple_size: 4
    best_description: "Most Fluent"
    worst_description: "Least Fluent"
 
  - annotation_type: bws
    name: adequacy
    description: "Select the BEST and WORST by meaning preservation"
    tuple_size: 4
    best_description: "Most Accurate"
    worst_description: "Least Accurate"

What the Output Contains

Each scheme records the item chosen best and the item chosen worst:

{
  "fluency": {"best": "sys_d", "worst": "sys_c"},
  "adequacy": {"best": "sys_a", "worst": "sys_c"}
}

The scores are not in this file. They are computed afterwards by potato.bws_scoring, which is what lets you rescore a finished study with a different method without recollecting anything.

Monitoring

The admin dashboard carries a BWS tab showing annotation progress and the current item scores, so you can watch whether the ranking has settled before deciding to stop collecting.

Full Example

annotation_task_name: "MT System Ranking (BWS)"
task_dir: "."
 
data_files:
  - "data/mt_systems.jsonl"
 
item_properties:
  id_key: id
  text_key: text
 
bws_config:
  tuple_size: 4
  min_item_appearances: 8
  seed: 42
  scoring:
    method: bradley_terry
 
annotation_schemes:
  - annotation_type: bws
    name: overall_quality
    description: "Select the BEST and the WORST translation"
    tuple_size: 4
    best_description: "Best Translation"
    worst_description: "Worst Translation"
    sequential_key_binding: true
    label_requirement:
      required: true
 
output_annotation_dir: "annotation_output/"
export_annotation_format: jsonl
 
user_config:
  allow_all_users: true

Collect the annotations, then score them:

potato start config.yaml -p 8000
python -m potato.bws_scoring --config config.yaml --method bradley_terry

Further Reading

• Pairwise Comparison - the two-item case
• Likert Scales - direct rating instead of comparison
• Export Formats - getting the annotations out

For implementation details, see the source documentation.