Consensus Aggregation of Crowd Votes, Points, and Boxes

New in v2.13.0

The consensus export combines the answers of several annotators on each item into one consensus answer per item, in the way citizen-science projects such as Zooniverse combine volunteers' classifications. Each subject in a volunteer project goes to several volunteers, e.g., to count the penguins in a colony photograph or to mark the animals in a camera-trap frame. For a choice question, the export reports how many annotators gave each answer and marks each item retired or needs_review. For point marks, the export clusters the annotators' points into one point per object, and for bounding boxes the export reports one median box per object. Crowdsourcing teams outside citizen science use the same export to turn several labels per item into one label per item. Following Zooniverse usage, Potato calls the annotators who answered a question on an item that item's classifiers.

• Choice questions: radio, select, likert, and multiselect schemes get a count and a fraction for every answer, the top answer, any tie, and a status.
• Numeric questions: slider and number schemes get the count, mean, median, standard deviation, minimum, and maximum.
• Points: landmark marks in an image_annotation scheme are clustered with DBSCAN, with the radius in pixels or as a fraction of the image.
• Boxes: bbox marks in an image_annotation scheme are grouped by overlap, and each group gets a median box.
• Install: the export needs nothing beyond Potato. The clustering uses numpy, and Potato installs numpy as a core dependency.

Consensus and agreement

Potato measures how well the annotators agree, and the consensus export produces one answer per item. The reliability export and the /admin/iaa page report Krippendorff's alpha for each question across the whole study, and a team with a low alpha looks first at the guidelines or the annotator training. The consensus export works item by item. Each item gets a status that records whether the annotators on that item agreed enough to keep the answer. Check agreement before relying on the consensus, because the consensus export gives every annotator one vote whatever that annotator's accuracy. For chance-corrected agreement on spatial labels, see Measuring Agreement on Bounding Boxes.

MACE weights each annotator's votes rather than counting every vote equally. MACE estimates a competence score for each annotator on radio, likert, select, and multiselect schemes and infers a label for each item from the weighted votes. Aggregating Crowd Labels covers when a competence model beats an unweighted vote and when the spread of answers should be kept. MACE does not read point or box marks. For marks on images, use the consensus export.

Configuration

The config below sets up a penguin-count workflow with a three-way question, a multiselect question, and an image scheme in which annotators mark each adult penguin with a point and draw a box around each nest:

annotation_task_name: "Penguin colony counts"
task_dir: .
data_files: [data/colonies.jsonl]
item_properties: {id_key: id, text_key: image}
output_annotation_dir: annotation_output
num_annotators_per_item: 5
user_config:
  allow_all_users: true
 
annotation_schemes:
  - annotation_type: radio
    name: T0
    description: "Are there penguins in the image?"
    labels: ["Yes", "No", "Unsure"]
  - annotation_type: multiselect
    name: T1
    description: "What else is in the image?"
    labels: [Ice, Water, Rock]
  - annotation_type: image_annotation
    name: T2
    description: "Mark each adult penguin and draw a box round each nest"
    source_field: image
    tools: [landmark, bbox]
    labels:
      - {name: Adult, color: "#00ff00"}
      - {name: Nest, color: "#ff0000"}
 
consensus_aggregation:
  agreement_threshold: 0.8
  min_classifiers: 2
  points:
    eps: 10
    eps_unit: pixels
    min_samples: 2
  boxes:
    iou_threshold: 0.5

The scheme names T0, T1, and T2 are Zooniverse task IDs. potato import keeps the task IDs as scheme names when a project comes from Zooniverse, and a study built in Potato can use any names. Potato sends each image to five annotators (num_annotators_per_item: 5). With agreement_threshold: 0.8, four of five annotators must give the same answer before an item is retired. With min_classifiers: 2, a single annotator's answer never counts as a consensus. Two points within 10 pixels of each other can start a cluster, and two boxes belong to one nest only when the boxes overlap by an intersection over union (IoU) of at least 0.5.

To cluster with a pixel radius, the export needs each image's size. Each item names the image and the image's width and height in pixels:

{"id": "9001", "image": "/media/colony_9001.png", "image_width": 1000, "image_height": 500}

Every key in the block is optional:

The server stops at startup and prints a message naming the key when a value is out of range, a key is outside the block's list, or a schemes entry names a scheme the config does not declare. python -m potato.validate_cli config.yaml reports the same errors before a study starts.

Export command

The export reads the config and the annotations saved under output_annotation_dir:

python -m potato.export -c config.yaml -f consensus -o consensus_out

The command writes consensus.json and the CSV files for the config's scheme types into consensus_out/. Pass --option to override one setting of the block for one run without changing the config:

python -m potato.export -c config.yaml -f consensus -o consensus_out \
    --option agreement_threshold=0.9 --option eps=8 --option eps_unit=pixels

The export stops on an unknown option name and prints the list of known names. The export also stops and prints the reason when the config has no scheme of a supported type or when no annotations are saved. The admin export API at /admin/api/export offers the same export under the format name consensus to teams without shell access to the server.

Choice questions

For each item and question, the export counts the classifiers and the annotators who gave each answer. Each answer's fraction is the answer's count divided by the number of classifiers. For a single-answer question the fractions add up to 1. For multiselect, each fraction is the share of classifiers who ticked that option, and the fractions can add up to more than 1. consensus_questions.csv has one row per item and question. The rows below come from the example project described at the end of this page:

instance_id,schema,type,n_classifiers,top_answer,top_count,top_fraction,tied,status,reason
9001,T0,radio,5,Yes,4,0.8,,retired,
9001,T1,multiselect,5,Ice,4,0.8,,needs_review,no agreement on Water (ticked by more than 20% and fewer than 80% of classifiers)
9002,T0,radio,4,Yes,2,0.5,Yes;No,needs_review,"tie between Yes, No"
9002,T1,multiselect,4,Rock,4,1.0,,retired,
9003,T0,radio,1,Yes,1,1.0,,needs_review,"1 classifier, fewer than min_classifiers (2)"
9003,T1,multiselect,1,Ice,1,1.0,,needs_review,"1 classifier, fewer than min_classifiers (2)"

The status follows three rules:

• Too few classifiers: an item with fewer than min_classifiers classifiers is needs_review.
• Single-answer questions: a radio, select, or likert item is retired when the top answer's fraction is at least agreement_threshold and no other answer ties the top answer.
• Multiselect questions: an item is retired when every option is decided. An option is decided when at least agreement_threshold of the classifiers ticked the option, or at least that share left the option unticked.

On image 9001, two of five annotators ticked Water. The Water option therefore stays undecided, because neither the 40% who ticked Water nor the 60% who left Water unticked reaches the 0.8 threshold. When answers tie, top_answer names the first tied answer in the order the scheme declares the labels, and tied lists all of the tied answers with semicolons between them. The reason column says why an item needs review.

The consensus status is a report, and Potato takes no action on the status. Zooniverse retires a subject after a set number of classifications, while Potato goes on assigning items under the study's assignment rules whatever the consensus status. Use the status to choose the items to send to more annotators or to adjudication.

consensus_votes.csv has one row per item, question, and answer, with columns instance_id, schema, answer, count, fraction, n_classifiers. Every option the scheme declares gets a row, with a count of 0 when no annotator chose the option:

instance_id,schema,answer,count,fraction,n_classifiers
9001,T0,Yes,4,0.8,5
9001,T0,No,1,0.2,5
9001,T0,Unsure,0,0.0,5
9001,T1,Ice,4,0.8,5
9001,T1,Water,2,0.4,5
9001,T1,Rock,1,0.2,5

A saved multiselect record that lists options with none ticked counts as a classifier with no votes. Such a record adds one to n_classifiers and nothing to any answer's count. A record with no entries for the question does not count as a classifier.

Slider and number questions

consensus_numeric.csv has one row per item and question, with columns instance_id, schema, type, n, mean, median, sd, min, max. n is the number of annotators with a numeric answer. sd is the sample standard deviation with an n − 1 denominator. sd stays empty when only one annotator answered. Numeric questions get no status.

Point clusters

DBSCAN (density-based spatial clustering of applications with noise) groups points that lie close together and leaves isolated points out as noise. The export pools the points of one label on one item from every annotator and clusters the pooled points with DBSCAN:

• A point is a core point when the point and the other points within eps of the point number at least min_samples.
• A cluster is a connected group of core points plus the other points within eps of one of the core points. A point next to two clusters joins the cluster found first, in the order the annotations were loaded.
• Every remaining point is noise.

The rule is the one scikit-learn's DBSCAN implements and the panoptes_aggregation reducer point_reducer_dbscan calls. With eps_unit: pixels and the same eps and min_samples, the export gives the same clusters and centers as point_reducer_dbscan. panoptes_aggregation defaults to an eps of 5 pixels and a min_samples of 3, while Potato's defaults suit a handful of annotators per item. With many volunteers per image, raise min_samples.

consensus_points.csv has one row per item, scheme, label, and cluster:

instance_id,schema,label,cluster,x,y,x_px,y_px,size,n_annotators,spread,spread_px,var_x,var_y,cov_xy,annotators,label_points,label_annotators,label_noise,n_classifiers,note
9001,T2,Adult,0,0.10025,0.201,100.25,100.5,4,4,0.003897,2.331845,2.916667,4.333333,-1.166667,vol1;vol2;vol3;vol4,8,4,1,4,
9001,T2,Adult,1,0.800667,0.8,800.666667,400.0,3,3,0.004643,3.681787,16.333333,4.0,8.0,vol1;vol2;vol4,8,4,1,4,
9002,T2,Adult,,,,,,,,,,,,,,1,1,1,1,no cluster (every point is noise)

On image 9001, four annotators placed eight Adult points. Seven of the points form two clusters with one adult in each, and the eighth point is noise. The columns are:

A label whose points are all noise still gets one row with an empty cluster and a note, as on image 9002. DBSCAN does not stop one annotator's two points from landing in one cluster. Compare size with n_annotators to find such clusters.

Image size and Zooniverse imports

Potato stores every mark as a fraction of the image width and height. With eps_unit: fraction, the export therefore clusters the points on every item without the image size. On a 1000 by 500 pixel image, an eps of 0.02 equals 20 pixels across and 10 pixels down.

The export needs the image size for pixel output and for eps_unit: pixels. The export takes the size from the item's image_width and image_height fields (or width and height). Without those fields, the export reads the size from the image file itself when the file sits under the task's media directory. Remote images are not downloaded. With eps_unit: pixels, a label on an item of unknown size gets a row with the note "image size unknown; pixel eps cannot be applied", and the export prints a warning naming up to five such items.

Zooniverse subjects are images hosted on Zooniverse's servers, and the export cannot open images on those servers. From Potato 2.13.0, potato import writes image_width and image_height on each item of a project with drawing tasks. The size comes from the image dimensions Zooniverse recorded with the classifications, or from a width and height in the subject metadata. Imported drawings therefore cluster in pixels, with the same eps a team used in panoptes_aggregation. Import with --keep-annotators. With the flag, the importer writes each volunteer's classifications as that volunteer's own saved work, and the consensus export then counts each volunteer once:

potato import -i zooniverse-exports/ -o my-project --keep-annotators
python -m potato.export -c my-project/config.yaml -f consensus -o consensus_out

Volunteers who were not logged in appear as anon_<id>, and one person may then count as several classifiers. Marks on subjects with no recorded size are left out at import, and the import counts the marks left out and prints a warning. The Zooniverse import section lists the task types the importer reads.

Consensus boxes

Intersection over union (IoU) is the area two boxes share divided by the area the two boxes cover together. The export puts the boxes of one label on one item into groups. A group holds at most one box per annotator, and every pair of boxes in a group overlaps by at least iou_threshold. The export builds the groups in three steps:

1. Each remaining box starts a candidate group, and the other annotators' remaining boxes are tried in order of IoU with the first box, highest first. A box joins the group when the box's IoU with every box already in the group meets the threshold.
2. The largest candidate group is kept. A tie goes to the group with the larger summed IoU with the first box, then to the earlier box.
3. The kept group's boxes leave the pool, and the steps repeat on the remaining boxes. A box that overlaps no other box becomes a group of one.

Two boxes that each overlap a third box end up in different groups when the two boxes do not overlap each other. The consensus box is the median of the members' left, top, right, and bottom edges. IoU does not change when an image is stretched. Grouping therefore needs no image size, and the pixel columns are filled when the size is known.

consensus_boxes.csv has one row per item, scheme, label, and group:

instance_id,schema,label,group,x,y,width,height,x_px,y_px,width_px,height_px,support,n_classifiers,support_fraction,mean_iou,min_iou,annotators
9001,T2,Nest,0,0.2,0.2,0.1,0.11,200.0,100.0,100.0,55.0,3,4,0.75,0.85617,0.826087,vol1;vol2;vol3
9001,T2,Nest,1,0.6,0.6,0.05,0.1,600.0,300.0,50.0,50.0,1,4,0.25,1.0,1.0,vol4
9003,T2,Nest,0,0.01,0.04,0.03,0.08,10.0,20.0,30.0,40.0,1,1,1.0,1.0,1.0,vol1

x, y, width, and height give the consensus box as fractions of the image, and the _px columns give the same box in pixels. support is the number of annotators in the group, n_classifiers is the number of annotators who saved the image scheme on the item, and support_fraction is support divided by n_classifiers. mean_iou and min_iou compare each member box with the consensus box. On image 9001, three of four annotators drew the first nest with a mean IoU of 0.86 against the median box, and one annotator's box overlapped no other box by the threshold and forms a group of one.

The grouping is Potato's own and is not a port of the Zooniverse box reducers. panoptes_aggregation's rectangle_reducer runs DBSCAN on each box's x, y, width, and height in pixels. shape_reducer_dbscan with metric_type: IoU runs DBSCAN with 1 − IoU as the distance, and a chain of overlapping boxes can join two boxes that do not overlap each other. Box consensus from Potato and from Zooniverse can therefore differ on the same classifications.

Text and other answer types

The consensus export reads seven scheme types: radio, select, likert, multiselect, slider, number, and image_annotation. Free-text, span, and every other scheme type are left out of the consensus without a warning. To narrow the list further, set schemes. A text answer has no vote to count, and the tabular exports keep every annotator's text for reading side by side or for adjudication.

Inside an image_annotation scheme, the export aggregates points and boxes only. Polygons, polylines, ellipses, freeform shapes, masks, and keypoint sets are counted by type, and the export names the counts in a warning. consensus.json records the same counts under not_aggregated.

Output files

Values in the CSV files are rounded to six decimal places. Lists such as annotators and tied are joined with semicolons. consensus.json keeps the full values and the list of classifiers for each question.

Example

python potato/flask_server.py start examples/image/zooniverse-aggregation/config.yaml -p 8000

The example is an invented penguin-count workflow imported from Zooniverse exports, with five volunteers' saved classifications on four colony images. The CSV rows on this page come from the consensus export of the example, run from a Potato checkout with python -m potato.export -c examples/image/zooniverse-aggregation/config.yaml -f consensus -o consensus_out.

Further reading

• Aggregating Crowd Labels - Dawid-Skene, MACE, and when to keep the spread of answers
• MACE Competence Estimation - Competence-weighted labels for categorical schemes
• Measuring Agreement on Bounding Boxes - Chance-corrected agreement on spatial labels
• Export Formats - Potato's export formats and the export command
• Ecology - Potato for camera-trap and bioacoustics annotation

For implementation details, see the source documentation.