# Consensus Aggregation of Crowd Votes, Points, and Boxes

Source: https://www.potatoannotator.com/docs/features/consensus-aggregation

*New in v2.13.0*

The `consensus` export combines the answers of several annotators on each item into one consensus answer per item, in the way citizen-science projects such as [Zooniverse](https://www.zooniverse.org/) combine volunteers' classifications. Each subject in a volunteer project goes to several volunteers, e.g., to count the penguins in a colony photograph or to mark the animals in a camera-trap frame. For a choice question, the export reports how many annotators gave each answer and marks each item `retired` or `needs_review`. For point marks, the export clusters the annotators' points into one point per object, and for bounding boxes the export reports one median box per object. Crowdsourcing teams outside citizen science use the same export to turn several labels per item into one label per item. Following Zooniverse usage, Potato calls the annotators who answered a question on an item that item's classifiers.

- **Choice questions:** `radio`, `select`, `likert`, and `multiselect` schemes get a count and a fraction for every answer, the top answer, any tie, and a status.
- **Numeric questions:** `slider` and `number` schemes get the count, mean, median, standard deviation, minimum, and maximum.
- **Points:** `landmark` marks in an `image_annotation` scheme are clustered with DBSCAN, with the radius in pixels or as a fraction of the image.
- **Boxes:** `bbox` marks in an `image_annotation` scheme are grouped by overlap, and each group gets a median box.
- **Install:** the export needs nothing beyond Potato. The clustering uses numpy, and Potato installs numpy as a core dependency.

## Consensus and agreement

Potato measures how well the annotators agree, and the consensus export produces one answer per item. The `reliability` export and the `/admin/iaa` page report [Krippendorff's alpha](/docs/guides/inter-annotator-agreement) for each question across the whole study, and a team with a low alpha looks first at the guidelines or the annotator training. The consensus export works item by item. Each item gets a status that records whether the annotators on that item agreed enough to keep the answer. Check agreement before relying on the consensus, because the consensus export gives every annotator one vote whatever that annotator's accuracy. For chance-corrected agreement on spatial labels, see [Measuring Agreement on Bounding Boxes](/docs/guides/measuring-agreement-on-bounding-boxes).

[MACE](/docs/features/mace) weights each annotator's votes rather than counting every vote equally. MACE estimates a competence score for each annotator on `radio`, `likert`, `select`, and `multiselect` schemes and infers a label for each item from the weighted votes. [Aggregating Crowd Labels](/docs/guides/aggregating-crowd-labels) covers when a competence model beats an unweighted vote and when the spread of answers should be kept. MACE does not read point or box marks. For marks on images, use the consensus export.

## Configuration

The config below sets up a penguin-count workflow with a three-way question, a multiselect question, and an image scheme in which annotators mark each adult penguin with a point and draw a box around each nest:

```yaml
annotation_task_name: "Penguin colony counts"
task_dir: .
data_files: [data/colonies.jsonl]
item_properties: {id_key: id, text_key: image}
output_annotation_dir: annotation_output
num_annotators_per_item: 5
user_config:
  allow_all_users: true

annotation_schemes:
  - annotation_type: radio
    name: T0
    description: "Are there penguins in the image?"
    labels: ["Yes", "No", "Unsure"]
  - annotation_type: multiselect
    name: T1
    description: "What else is in the image?"
    labels: [Ice, Water, Rock]
  - annotation_type: image_annotation
    name: T2
    description: "Mark each adult penguin and draw a box round each nest"
    source_field: image
    tools: [landmark, bbox]
    labels:
      - {name: Adult, color: "#00ff00"}
      - {name: Nest, color: "#ff0000"}

consensus_aggregation:
  agreement_threshold: 0.8
  min_classifiers: 2
  points:
    eps: 10
    eps_unit: pixels
    min_samples: 2
  boxes:
    iou_threshold: 0.5
```

The scheme names T0, T1, and T2 are Zooniverse task IDs. `potato import` keeps the task IDs as scheme names when a project comes from Zooniverse, and a study built in Potato can use any names. Potato sends each image to five annotators (`num_annotators_per_item: 5`). With `agreement_threshold: 0.8`, four of five annotators must give the same answer before an item is retired. With `min_classifiers: 2`, a single annotator's answer never counts as a consensus. Two points within 10 pixels of each other can start a cluster, and two boxes belong to one nest only when the boxes overlap by an intersection over union (IoU) of at least 0.5.

To cluster with a pixel radius, the export needs each image's size. Each item names the image and the image's width and height in pixels:

```json
{"id": "9001", "image": "/media/colony_9001.png", "image_width": 1000, "image_height": 500}
```

Every key in the block is optional:

| Key | Default | `--option` name | Meaning |
|---|---|---|---|
| `agreement_threshold` | `0.8` | `agreement_threshold` or `threshold` | Share of classifiers the top answer needs before the item is retired. Greater than 0 and at most 1 |
| `min_classifiers` | `2` | `min_classifiers` | Fewest classifiers for a status other than `needs_review`. A whole number, at least 1 |
| `schemes` | every supported scheme | `schemes` (comma-separated) | Scheme names to aggregate. Each name must be a scheme the config declares |
| `points.eps` | `0.02` | `eps` | Cluster radius. Greater than 0, and at most 1 when `eps_unit` is `fraction` |
| `points.eps_unit` | `fraction` | `eps_unit` | `fraction` measures x as a share of the image width and y as a share of the height, and `pixels` measures in image pixels |
| `points.min_samples` | `2` | `min_samples` | Fewest points within `eps` of a point for the point to be a core point. The count includes the point itself. A whole number, at least 1 |
| `boxes.iou_threshold` | `0.5` | `iou_threshold` | Least IoU between any two boxes in a group. Greater than 0 and at most 1 |

The server stops at startup and prints a message naming the key when a value is out of range, a key is outside the block's list, or a `schemes` entry names a scheme the config does not declare. `python -m potato.validate_cli config.yaml` reports the same errors before a study starts.

## Export command

The export reads the config and the annotations saved under `output_annotation_dir`:

```bash
python -m potato.export -c config.yaml -f consensus -o consensus_out
```

The command writes `consensus.json` and the CSV files for the config's scheme types into `consensus_out/`. Pass `--option` to override one setting of the block for one run without changing the config:

```bash
python -m potato.export -c config.yaml -f consensus -o consensus_out \
    --option agreement_threshold=0.9 --option eps=8 --option eps_unit=pixels
```

The export stops on an unknown option name and prints the list of known names. The export also stops and prints the reason when the config has no scheme of a supported type or when no annotations are saved. The admin export API at `/admin/api/export` offers the same export under the format name `consensus` to teams without shell access to the server.

## Choice questions

For each item and question, the export counts the classifiers and the annotators who gave each answer. Each answer's fraction is the answer's count divided by the number of classifiers. For a single-answer question the fractions add up to 1. For `multiselect`, each fraction is the share of classifiers who ticked that option, and the fractions can add up to more than 1. `consensus_questions.csv` has one row per item and question. The rows below come from the example project described at the end of this page:

```csv
instance_id,schema,type,n_classifiers,top_answer,top_count,top_fraction,tied,status,reason
9001,T0,radio,5,Yes,4,0.8,,retired,
9001,T1,multiselect,5,Ice,4,0.8,,needs_review,no agreement on Water (ticked by more than 20% and fewer than 80% of classifiers)
9002,T0,radio,4,Yes,2,0.5,Yes;No,needs_review,"tie between Yes, No"
9002,T1,multiselect,4,Rock,4,1.0,,retired,
9003,T0,radio,1,Yes,1,1.0,,needs_review,"1 classifier, fewer than min_classifiers (2)"
9003,T1,multiselect,1,Ice,1,1.0,,needs_review,"1 classifier, fewer than min_classifiers (2)"
```

The status follows three rules:

- **Too few classifiers:** an item with fewer than `min_classifiers` classifiers is `needs_review`.
- **Single-answer questions:** a `radio`, `select`, or `likert` item is `retired` when the top answer's fraction is at least `agreement_threshold` and no other answer ties the top answer.
- **Multiselect questions:** an item is `retired` when every option is decided. An option is decided when at least `agreement_threshold` of the classifiers ticked the option, or at least that share left the option unticked.

On image 9001, two of five annotators ticked Water. The Water option therefore stays undecided, because neither the 40% who ticked Water nor the 60% who left Water unticked reaches the 0.8 threshold. When answers tie, `top_answer` names the first tied answer in the order the scheme declares the labels, and `tied` lists all of the tied answers with semicolons between them. The `reason` column says why an item needs review.

The consensus status is a report, and Potato takes no action on the status. Zooniverse retires a subject after a set number of classifications, while Potato goes on assigning items under the study's assignment rules whatever the consensus status. Use the status to choose the items to send to more annotators or to [adjudication](/docs/guides/adjudication-and-disagreement).

`consensus_votes.csv` has one row per item, question, and answer, with columns `instance_id, schema, answer, count, fraction, n_classifiers`. Every option the scheme declares gets a row, with a count of 0 when no annotator chose the option:

```csv
instance_id,schema,answer,count,fraction,n_classifiers
9001,T0,Yes,4,0.8,5
9001,T0,No,1,0.2,5
9001,T0,Unsure,0,0.0,5
9001,T1,Ice,4,0.8,5
9001,T1,Water,2,0.4,5
9001,T1,Rock,1,0.2,5
```

A saved multiselect record that lists options with none ticked counts as a classifier with no votes. Such a record adds one to `n_classifiers` and nothing to any answer's count. A record with no entries for the question does not count as a classifier.

## Slider and number questions

`consensus_numeric.csv` has one row per item and question, with columns `instance_id, schema, type, n, mean, median, sd, min, max`. `n` is the number of annotators with a numeric answer. `sd` is the sample standard deviation with an n − 1 denominator. `sd` stays empty when only one annotator answered. Numeric questions get no status.

## Point clusters

[DBSCAN](https://en.wikipedia.org/wiki/DBSCAN) (density-based spatial clustering of applications with noise) groups points that lie close together and leaves isolated points out as noise. The export pools the points of one label on one item from every annotator and clusters the pooled points with DBSCAN:

- A point is a core point when the point and the other points within `eps` of the point number at least `min_samples`.
- A cluster is a connected group of core points plus the other points within `eps` of one of the core points. A point next to two clusters joins the cluster found first, in the order the annotations were loaded.
- Every remaining point is noise.

The rule is the one scikit-learn's `DBSCAN` implements and the [`panoptes_aggregation`](https://github.com/zooniverse/aggregation-for-caesar) reducer `point_reducer_dbscan` calls. With `eps_unit: pixels` and the same `eps` and `min_samples`, the export gives the same clusters and centers as `point_reducer_dbscan`. `panoptes_aggregation` defaults to an `eps` of 5 pixels and a `min_samples` of 3, while Potato's defaults suit a handful of annotators per item. With many volunteers per image, raise `min_samples`.

`consensus_points.csv` has one row per item, scheme, label, and cluster:

```csv
instance_id,schema,label,cluster,x,y,x_px,y_px,size,n_annotators,spread,spread_px,var_x,var_y,cov_xy,annotators,label_points,label_annotators,label_noise,n_classifiers,note
9001,T2,Adult,0,0.10025,0.201,100.25,100.5,4,4,0.003897,2.331845,2.916667,4.333333,-1.166667,vol1;vol2;vol3;vol4,8,4,1,4,
9001,T2,Adult,1,0.800667,0.8,800.666667,400.0,3,3,0.004643,3.681787,16.333333,4.0,8.0,vol1;vol2;vol4,8,4,1,4,
9002,T2,Adult,,,,,,,,,,,,,,1,1,1,1,no cluster (every point is noise)
```

On image 9001, four annotators placed eight Adult points. Seven of the points form two clusters with one adult in each, and the eighth point is noise. The columns are:

| Column | Meaning |
|---|---|
| `cluster` | Cluster number within the label on the item, from 0 |
| `x`, `y` | Cluster center, the mean of the member points, as fractions of the image width and height |
| `x_px`, `y_px` | The same center in pixels, empty when the image size is unknown |
| `size` | Points in the cluster |
| `n_annotators`, `annotators` | Distinct annotators with a point in the cluster, and the annotators' names |
| `spread`, `spread_px` | Root mean squared distance of the member points from the center, as a fraction and in pixels |
| `var_x`, `var_y`, `cov_xy` | Sample variances of x and y and the covariance of x and y, in pixels when the image size is known. The values match the `clusters_var_x`, `clusters_var_y`, and `clusters_var_x_y` columns of the Zooniverse reducer |
| `label_points`, `label_annotators`, `label_noise` | Points, annotators, and noise points for the whole label on the item, repeated on each of the label's rows |
| `n_classifiers` | Annotators who saved the image scheme on the item |
| `note` | Why a label has no cluster |

A label whose points are all noise still gets one row with an empty `cluster` and a `note`, as on image 9002. DBSCAN does not stop one annotator's two points from landing in one cluster. Compare `size` with `n_annotators` to find such clusters.

## Image size and Zooniverse imports

Potato stores every mark as a fraction of the image width and height. With `eps_unit: fraction`, the export therefore clusters the points on every item without the image size. On a 1000 by 500 pixel image, an `eps` of 0.02 equals 20 pixels across and 10 pixels down.

The export needs the image size for pixel output and for `eps_unit: pixels`. The export takes the size from the item's `image_width` and `image_height` fields (or `width` and `height`). Without those fields, the export reads the size from the image file itself when the file sits under the task's media directory. Remote images are not downloaded. With `eps_unit: pixels`, a label on an item of unknown size gets a row with the note "image size unknown; pixel eps cannot be applied", and the export prints a warning naming up to five such items.

Zooniverse subjects are images hosted on Zooniverse's servers, and the export cannot open images on those servers. From Potato 2.13.0, `potato import` writes `image_width` and `image_height` on each item of a project with drawing tasks. The size comes from the image dimensions Zooniverse recorded with the classifications, or from a width and height in the subject metadata. Imported drawings therefore cluster in pixels, with the same `eps` a team used in `panoptes_aggregation`. Import with `--keep-annotators`. With the flag, the importer writes each volunteer's classifications as that volunteer's own saved work, and the consensus export then counts each volunteer once:

```bash
potato import -i zooniverse-exports/ -o my-project --keep-annotators
python -m potato.export -c my-project/config.yaml -f consensus -o consensus_out
```

Volunteers who were not logged in appear as `anon_<id>`, and one person may then count as several classifiers. Marks on subjects with no recorded size are left out at import, and the import counts the marks left out and prints a warning. The [Zooniverse import section](/docs/guides/digital-humanities#zooniverse) lists the task types the importer reads.

## Consensus boxes

[Intersection over union](https://en.wikipedia.org/wiki/Jaccard_index) (IoU) is the area two boxes share divided by the area the two boxes cover together. The export puts the boxes of one label on one item into groups. A group holds at most one box per annotator, and every pair of boxes in a group overlaps by at least `iou_threshold`. The export builds the groups in three steps:

1. Each remaining box starts a candidate group, and the other annotators' remaining boxes are tried in order of IoU with the first box, highest first. A box joins the group when the box's IoU with every box already in the group meets the threshold.
2. The largest candidate group is kept. A tie goes to the group with the larger summed IoU with the first box, then to the earlier box.
3. The kept group's boxes leave the pool, and the steps repeat on the remaining boxes. A box that overlaps no other box becomes a group of one.

Two boxes that each overlap a third box end up in different groups when the two boxes do not overlap each other. The consensus box is the median of the members' left, top, right, and bottom edges. IoU does not change when an image is stretched. Grouping therefore needs no image size, and the pixel columns are filled when the size is known.

`consensus_boxes.csv` has one row per item, scheme, label, and group:

```csv
instance_id,schema,label,group,x,y,width,height,x_px,y_px,width_px,height_px,support,n_classifiers,support_fraction,mean_iou,min_iou,annotators
9001,T2,Nest,0,0.2,0.2,0.1,0.11,200.0,100.0,100.0,55.0,3,4,0.75,0.85617,0.826087,vol1;vol2;vol3
9001,T2,Nest,1,0.6,0.6,0.05,0.1,600.0,300.0,50.0,50.0,1,4,0.25,1.0,1.0,vol4
9003,T2,Nest,0,0.01,0.04,0.03,0.08,10.0,20.0,30.0,40.0,1,1,1.0,1.0,1.0,vol1
```

`x`, `y`, `width`, and `height` give the consensus box as fractions of the image, and the `_px` columns give the same box in pixels. `support` is the number of annotators in the group, `n_classifiers` is the number of annotators who saved the image scheme on the item, and `support_fraction` is `support` divided by `n_classifiers`. `mean_iou` and `min_iou` compare each member box with the consensus box. On image 9001, three of four annotators drew the first nest with a mean IoU of 0.86 against the median box, and one annotator's box overlapped no other box by the threshold and forms a group of one.

The grouping is Potato's own and is not a port of the Zooniverse box reducers. `panoptes_aggregation`'s `rectangle_reducer` runs DBSCAN on each box's x, y, width, and height in pixels. `shape_reducer_dbscan` with `metric_type: IoU` runs DBSCAN with 1 − IoU as the distance, and a chain of overlapping boxes can join two boxes that do not overlap each other. Box consensus from Potato and from Zooniverse can therefore differ on the same classifications.

## Text and other answer types

The consensus export reads seven scheme types: `radio`, `select`, `likert`, `multiselect`, `slider`, `number`, and `image_annotation`. Free-text, span, and every other scheme type are left out of the consensus without a warning. To narrow the list further, set `schemes`. A text answer has no vote to count, and the [tabular exports](/docs/features/export-formats) keep every annotator's text for reading side by side or for adjudication.

Inside an `image_annotation` scheme, the export aggregates points and boxes only. Polygons, polylines, ellipses, freeform shapes, masks, and keypoint sets are counted by type, and the export names the counts in a warning. `consensus.json` records the same counts under `not_aggregated`.

## Output files

| File | One row per | Written when |
|---|---|---|
| `consensus_questions.csv` | item and question | the config has a `radio`, `select`, `likert`, or `multiselect` scheme |
| `consensus_votes.csv` | item, question, and answer | as above |
| `consensus_numeric.csv` | item and question | the config has a `slider` or `number` scheme |
| `consensus_points.csv` | item, scheme, label, and point cluster | the config has an `image_annotation` scheme |
| `consensus_boxes.csv` | item, scheme, label, and box group | as above |
| `consensus.json` | the results above, nested, with the settings used, the `not_aggregated` counts, and the warnings | always |

Values in the CSV files are rounded to six decimal places. Lists such as `annotators` and `tied` are joined with semicolons. `consensus.json` keeps the full values and the list of classifiers for each question.

## Example

```bash
python potato/flask_server.py start examples/image/zooniverse-aggregation/config.yaml -p 8000
```

The example is an invented penguin-count workflow imported from Zooniverse exports, with five volunteers' saved classifications on four colony images. The CSV rows on this page come from the consensus export of the example, run from a Potato checkout with `python -m potato.export -c examples/image/zooniverse-aggregation/config.yaml -f consensus -o consensus_out`.

## Further reading

- [Aggregating Crowd Labels](/docs/guides/aggregating-crowd-labels) - Dawid-Skene, MACE, and when to keep the spread of answers
- [MACE Competence Estimation](/docs/features/mace) - Competence-weighted labels for categorical schemes
- [Measuring Agreement on Bounding Boxes](/docs/guides/measuring-agreement-on-bounding-boxes) - Chance-corrected agreement on spatial labels
- [Export Formats](/docs/features/export-formats) - Potato's export formats and the export command
- [Ecology](/use-cases/ecology) - Potato for camera-trap and bioacoustics annotation

For implementation details, see the [source documentation](https://github.com/davidjurgens/potato/blob/master/docs/data-export/consensus_aggregation.md).