Agreement Drift and Re-Calibration Rounds

Potato scores intercoder agreement in successive windows of finished items, and a fall in recent coding shows up on the agreement report as a low window instead of averaging into one project-wide figure. A whole-project coefficient blends the first week's coding with the last week's, and a team whose recent work has slipped can still report an acceptable α. The Agreement over time section of /admin/iaa shows each annotation scheme's coefficient window by window and raises a re-calibration prompt when the latest window falls a set fraction below the project baseline. A re-calibration round then sends coders back through the training items and grades the answers against known correct answers. The timeline and the rounds have been in Potato since v2.9.0, and the timeline is on by default.

Drift here means a gradual change in how coders apply a codebook over the course of a study. Drift builds up as new edge cases arrive and each coder settles on a private reading of a definition. A single intercoder reliability check before coding starts cannot detect drift. In a passage that Hennessy, Bleakley, and Ellithorpe (2022) quote, Lacy and colleagues (2015) write that "the intercoder reliability check should occur with study content as it is coded." The same concern applies to an ML labeling team whose guidelines change as the label set grows. This page covers agreement as a study proceeds, while agreement on time segments covers annotators marking start and end times in audio and video.

Window boundaries and item timestamps

Agreement belongs to an item across coders rather than to one coder's answer. Potato therefore places each item in the window containing the moment the item's last coder finished. That moment is when the item first had a full set of answers to compare. Only items that have reached their annotator cap enter the timeline, and the cap has to be two or more (num_annotators_per_item: 2, for example). The whole-project agreement table draws on the same items. Each window therefore holds a share of the items behind the baseline.

The moment a coder finished an item is the last recorded change to that scheme's answer. When an item has no recorded change, Potato falls back to the time the coder left the item, then to the time the coder opened the item. Opening time is the last resort because a coder who skims a queue and answers later opened each item long before answering. Items with no timestamp at all stay off the timeline, and the section reports how many. Annotations imported from another tool are the usual case.

The items are sorted by that moment and cut into windows in one of two ways:

• Equal count (window_by: count, the default). Every window holds the same number of items. The number of windows is capped at the number of items. A five-item pilot therefore produces no empty windows.
• Equal duration (window_by: time). Every window spans the same stretch of clock time. When coding happens in bursts, for example on two weekends a month apart, the windows between the bursts are empty. An empty window has no agreement value at all. In the table on the agreement page, an empty window would read as a collapse in agreement when no coding happened, and equal count is the default for that reason.

Which coefficient each window reports

Potato re-runs the whole-project agreement computation on each window's own items. Every window and the baseline therefore use the same coefficient for a given scheme. Potato picks one headline coefficient per scheme and prefers chance-corrected measures. Percent agreement rises with class imbalance alone and can trend upward while coders agree less than chance would predict. The coefficient depends on the scheme type:

Krippendorff's α is 1 for perfect agreement and 0 for agreement at the level chance would produce. Free-text schemes such as textbox have no agreement measure and get no row on the timeline. When a window's data cannot produce the coefficient the baseline uses, the cell shows no value rather than a different measure. A drop between two different measures would mean nothing.

Timeline cells and cell notes

Each scheme gets one row. The row opens with the whole-project coefficient as the baseline, followed by one cell per window (W1, W2, and so on) with the window's coefficient and item count. A cell without a number shows a short note instead, and Potato explains each of those notes under the timeline table. A window with too few items shows the value and adds a marker. The notes and the marker mean:

When the project uses a QDA Mode codebook, a line under the tables lists the codebook revisions (r1 · r2 · r3). Potato bumps the revision on every codebook change. Creating, renaming, recoloring, moving, or deleting a code each counts as a change. Revisions recorded since v2.9.0 have an exact time. Older revisions are dated from the earliest annotation stamped with that revision. That date is a lower bound, and the revision is labeled approximate. The HTML page lists the revisions by number, and the JSON report below gives each revision's time. A script can place each codebook edit against the window boundaries to show whether a guideline change moved agreement.

Configuration

The timeline needs no configuration. Any project with two or more coders per item and agreement reporting gets the section on /admin/iaa. The admin dashboard links the page as the full agreement report. To change the defaults, add a calibration block to an existing config:

calibration:
  enabled: true
  windows: 6
  window_by: count
  drop_threshold: 0.15

calibration takes exactly these four keys:

The timeline costs one extra agreement computation per window on every load of /admin/iaa. Setting enabled: false removes that cost on a large project. The timeline then disappears from both /admin/iaa and the /admin/iaa/drift JSON report.

A re-calibration round re-runs the training phase. A project that will run rounds therefore needs training enabled and a training entry in phases. The config below is a complete two-coder study of open-ended survey answers with one code per answer:

annotation_task_name: Open-ended responses, two coders
task_dir: .
output_annotation_dir: annotation_output/
data_files: [data/responses.json]
item_properties: {id_key: id, text_key: text}
 
annotation_schemes:
  - annotation_type: radio
    name: barrier
    description: Which barrier to care does the respondent describe?
    labels: [Cost, Distance, Wait time, Trust, None]
 
num_annotators_per_item: 2
 
training:
  enabled: true
  data_file: data/training.json
  allow_retry: true
  failure_action: repeat_training
 
phases:
  order: [training, annotation]
  training: {type: training}
  annotation: {type: annotation}
 
agreement_metrics: {enabled: true}
 
calibration:
  windows: 6
  window_by: count
  drop_threshold: 0.15

num_annotators_per_item: 2 puts every excerpt in front of both coders. Two coders per excerpt give each window an agreement value. failure_action: repeat_training restarts the training items for a coder who fails a round instead of ending that coder's participation. The training file holds excerpts with the agreed code and an explanation, in the format the training phase page documents.

The re-calibration prompt

Potato compares each scheme's latest window with the baseline and raises a prompt when the relative fall reaches drop_threshold. The relative fall is (baseline − latest) / baseline. The baseline is the whole-project coefficient rather than the first window's because a newly trained team's agreement is least stable in the first window. An unusually good first window would otherwise set the bar for the whole study. With a baseline α of 0.720 and a latest window at 0.580, the fall is 19%, and the section shows:

Re-calibration suggested — barrier 19% below the project baseline (alpha_nominal 0.580 vs 0.720).

The prompt is advice to the team lead. The prompt does not notify coders and does not start a round. Potato does not raise the prompt in three cases:

• An earlier dip. Potato judges only the latest window with five or more items. A prompt about a dip the team has already recovered from would teach the team to ignore the prompt.
• An undefined latest window. When the latest window shows total agreement or undefined, Potato does not fall back to an older window.
• A baseline at or below zero. Agreement already at chance is a problem the whole-project table reports, and a percentage fall from zero is not meaningful.

Starting a calibration round through the admin API

A round resets each named coder's training progress and returns the coder to the training phase. The coder works through the same training items again. Potato grades the answers against the same correct answers and shows the explanation for each wrong answer, as in the first pass. After passing, the coder moves on to the phase that follows training in phases.order and resumes where the coder left off. A round changes neither assignments nor saved annotations. Potato shows coders no message saying a round has started. Tell the team before you start a round.

You start a round through the admin API with the admin API key. The admin pages have no button for starting a round. The admin dashboard page explains where the key comes from. Check who is eligible and how the last round went:

curl -H "X-API-Key: $KEY" http://localhost:8000/admin/calibration

The response lists the coders a round can recall, past rounds, and each recalled coder's progress in the latest round. Progress is one of passed, failed, in_progress, or unknown, with answered, total, and correct counts. Coders in the annotation phase, the post-study phase, or the done phase are eligible. Start a round for every eligible coder:

curl -X POST -H "X-API-Key: $KEY" -H "Content-Type: application/json" \
     -d '{"reason": "barrier alpha fell 19%"}' \
     http://localhost:8000/admin/calibration/start

The call returns the round's ID, the coders recalled, and any coders skipped with the reason. Pass "usernames": ["coder1", "coder2"] to recall named coders only. Omitting usernames recalls everyone eligible. Guideline drift affects the whole team. Recalling everyone is therefore usually the right choice.

A coder who is still on the consent or instructions page is listed in the response under skipped. When training is not enabled, the call returns HTTP 400 instead of recalling anyone. Without training items, recalled coders would pass straight through an empty training phase.

Plan the training settings before the first round, for two reasons. First, a coder who fails a round is handled by training.failure_action. Under the default move_to_done, failing a round ends that coder's participation in the study. Failing the first pass has the same effect. Set failure_action: repeat_training to restart the training items instead. Second, a round selects training items by ID and shows each coder the items the coder saw the first time. Training items added to the file after a coder's first pass do not reach that coder in a round. Put the hard cases into the training file before coding starts.

Where the team discusses disagreements

A round re-runs the training items because those items have known answers to grade against. The study's own disagreement items have no right answer. Potato therefore leaves those items out of a round. The discussion half of re-calibration happens elsewhere in Potato. A huddle room in Multiplayer Rooms walks the team through the items the team currently disagrees on and shows every coder's original code. In the adjudication queue, an adjudicator resolves disputed items one at a time. A qualitative team usually ends that discussion by revising code definitions in the codebook, and the next revision then appears on the timeline beside the windows that follow.

Drift report and round log

The drift report and the round log are separate from the annotation exports. No python -m potato.export format includes the drift report or the round log. Fetch the timeline as JSON:

curl -H "X-API-Key: $KEY" http://localhost:8000/admin/iaa/drift

The JSON report has these fields:

• windows lists each window's index, start and end time, item count, sparse flag, item IDs, and the full agreement report restricted to the window.
• schemas gives, per scheme, the headline metric, the baseline, and one point per window with the window's value, note, and item count.
• triggers lists every scheme that raised the prompt, with the baseline, the latest value, and the relative fall.
• markers lists codebook revisions with each revision's time and an inferred flag for approximate dates.
• n_items and n_untimed_items count the items on and off the timeline, and reason explains an empty report.

Times are Unix timestamps in seconds. The report is empty when there is nothing to compare, for example when no item has reached its annotator cap or when every timed item falls in one window. The reason field then says why.

Potato logs each round in the calibration_rounds table of <task_dir>/project.sqlite, with the round's start time, the admin who started the round (admin for an API-key call), the reason text, and the coders recalled. The round log and the windowed coefficients give a methods section the dates of each re-calibration and the agreement before and after each round. A qualitative audit trail needs both records. A coder's training answers live in user_state.json, and each round resets those answers. To keep the first-pass training scores, save a copy of the annotation output directory before a round.

Further reading

• Training phase, for the training file, passing criteria, and failure_action
• Multiplayer Rooms, for norming sessions and huddles on disputed items
• Intercoder reliability in qualitative research, for which coefficient to report and when
• Inter-annotator agreement, for κ and α across annotation types
• Agreement on time segments, for audio and video segments

For implementation details, see the source documentation.

References

• Hennessy, M., Bleakley, A., & Ellithorpe, M. E. (2022). Evaluating and tracking qualitative content coder performance using item response theory. Quality & Quantity, 57(2), 1231-1245. https://doi.org/10.1007/s11135-022-01397-7
• Lacy, S., Watson, B. R., Riffe, D., & Lovejoy, J. (2015). Issues and best practices in content analysis. Journalism & Mass Communication Quarterly, 92(4), 791-811. https://doi.org/10.1177/1077699015607338 (quoted from Hennessy et al., 2022)