Measuring How LLM Suggestions Anchor Annotators Using Blind Items

An annotator who sees an LLM's suggested label tends to give that label, so agreement between people and the model is higher when the suggestion is on screen than when it is not. Hide the suggestion on a random sample of items, measure agreement on those blind items, and use blind agreement instead of overall agreement to decide how far to trust the model.

Anchoring is the tendency for a number or answer seen first to pull a later judgment toward it. In annotation the anchor is a pre-label, a model's suggestion that the annotator accepts or corrects. Anchoring differs from rubber-stamping, where an annotator accepts suggestions without reading them. A careful annotator who reads every suggestion can still be anchored, because on an ambiguous item the suggested answer is already on the page. Timing signals catch rubber-stamping, as Agreement Cannot Catch Rubber-Stamping describes. An anchored annotator, however, can spend a normal amount of time on an item, so timing does not separate anchoring from independent judgment.

Evidence that suggestions change labels

A pre-registered experiment (Schroeder et al., 2025) collected 7,000 annotations on subjective tasks from 350 crowdworkers, across 4 conditions, 2 models and 2 datasets. Showing LLM suggestions did not make the annotators faster, but it raised their self-reported confidence. The annotators took the suggestions often enough to change the label distribution significantly compared with annotators who saw none. When the labels made with LLM assistance were used to evaluate the LLM, its reported performance rose significantly.

Agreement between a model and reviewers who saw its suggestions is therefore partly agreement between the model and itself. Any decision that rests on that number, such as handing the rest of a corpus to the model once agreement crosses 90%, is made on an inflated figure.

Blind items and the anchoring gap

Before annotation starts, choose a random share of items that the annotator labels without the suggestion. On the rest, the suggestion appears as usual. Comparing the two gives three numbers:

• blind agreement, between the model and labels given without its suggestion, which estimates how often the model is right;
• shown agreement, between the model and labels given with its suggestion on screen; and
• the anchoring gap, shown agreement minus blind agreement, which estimates how far the suggestion moves people.

The blind items have to be chosen at random and fixed for the study. Choosing the hardest items, or letting an annotator decide when to look, makes the blind sample unrepresentative, and the blind agreement then estimates nothing about the rest of the corpus. A label also counts as blind only if the annotator never saw a suggestion for that item, so a relabel after a disagreement belongs with the shown labels.

The blind share is a trade-off between speed and evidence. Every blind item is labeled without the time a suggestion saves, and too few blind items make the blind agreement noisy. Potato's defaults are a 20% share with a floor of 20 blind items. At that share a study of 20 items produces only about 4 blind items, so set the floor to what the corpus can give.

Blind suggestions in Solo Mode

Solo Mode is Potato's setup for one expert working with an LLM. The LLM labels the corpus, the expert labels alongside it, and Solo Mode hands the remaining items to the LLM once the expert's agreement with it crosses end_human_annotation_agreement. Before Potato 2.11 that agreement was measured on labels given with the suggestion visible. From 2.11, solo_mode.suggestions controls when the suggestion appears:

solo_mode:
  suggestions:
    display: blind_sample      # before_label | after_label | blind_sample
    blind_rate: 0.2
    minimum_blind_sample: 20

blind_sample hides the suggestion on a fixed 20% of items, chosen once for the study so a reload or a restart keeps the same items blind. after_label never shows the suggestion first, and when the expert and the LLM disagree, the resolution page shows the LLM's label and its reasoning. Once there are 20 blind comparisons, the handoff compares blind agreement with the threshold instead of overall agreement.

The dashboard's Blind Agreement card shows blind and shown agreement side by side, the gap between them, and how often the expert took the suggestion with the Accept button. The gap is reported once each side has at least 5 comparisons. A large positive gap means the expert agrees with the LLM more often when they can see its answer. In that case, keep more items blind or switch to after_label.

Blind calibration in the labeling engine

Potato's labeling engine, also new in 2.11, applies the same rule to every label it uses as evidence. Calibration, certification and audit requests are drawn at random with recorded inclusion probabilities, and while a person holds one, the server withholds every machine answer for that item. Pre-annotations, the inline judge and the AI assistant are all off, and the item stays blind for that person after they save it. The engine certifies machine labels only against these blind labels.

Verify requests are the opposite case. They show the machine's answer, ask the person to keep it or change it, and record the prefilled answer so you can measure how often people accept it. The engine never uses verify answers to certify, because they are anchored by construction. Setting humans.verify_visibility: blind serves verify requests without the answer too, at the cost of the time a suggestion saves.

Limits of a blind sample

A blind sample measures the model against unassisted people, which is the right reference only if unassisted people are reliable on the task. On a subjective task where careful annotators disagree with each other, blind agreement has a ceiling below 100%, and a model can sit at that ceiling. Compare the model's agreement with people to the people's agreement with each other, as LLM judge agreement describes, before you read a low blind agreement as a weak model.

A blind sample also leaves the labels that were shown with a suggestion as they are. Those labels stay anchored, and a dataset built mostly from them inherits the model's preferences. Report the blind agreement and the anchoring gap with the dataset, and set display: blind_sample before the first item is labeled, since items labeled with the suggestion visible cannot be made blind afterwards.

Further reading

• Agreement Cannot Catch Rubber-Stamping, on timing signals for annotators who accept without reading
• LLM pre-annotation, on automation bias in reviewed pre-labels
• Certifying LLM Labels with a Blind Human Sample, on certifying machine labels against a blind sample

References

1. Schroeder, H., Roy, D., and Kabbara, J. (2025). Just put a human in the loop? Investigating LLM-assisted annotation for subjective tasks. Findings of ACL 2025. https://doi.org/10.18653/v1/2025.findings-acl.1323