Human Review Queues for LLM and Agent Traces

A human review queue is a list of LLM or agent traces waiting for human labels, and Langfuse and LangSmith call the feature an annotation queue. Langfuse, LangSmith, and Arize AX build annotation queues into the tracing tool, Braintrust assigns logs to reviewers on a Review page, and Arize Phoenix lets reviewers annotate spans from the trace view. Potato is a free, self-hosted review tool that takes traces from Langfuse, a webhook, or an export, sends each trace to several reviewers, reports how well the reviewers agree, and writes the merged answer back to Langfuse or LangSmith as a score.

A trace records one run of an LLM application, and a span (an observation, in Langfuse) records one step inside the run, such as a model call or a tool call (OpenTelemetry traces). An agent transcript is the trace of a multi-step agent or a multi-turn chat. Reviewers label traces for two reasons. First, human labels are the reference that an LLM-as-a-judge is checked against before the judge scores production traffic alone (Zheng et al., 2023). Second, reviewed traces become an evaluation dataset. An evaluation dataset is the fixed set of inputs and expected outputs that each new prompt or model version is tested on. Inter-annotator agreement is how often two reviewers give the same label. Agreement is usually reported as Cohen's kappa or Krippendorff's alpha.

AI Agent Evaluation Tools Compared compares the platforms feature by feature. LLM Human Evaluation and Preference Data Tools Compared covers preference data. Annotating Agent Trajectories covers the design of a step-level error taxonomy.

Sources of traces for a review queue

A team rarely reviews every trace. Each tool therefore offers a way to pick the traces that reach a person. In Langfuse, you select traces, observations, or sessions in a table and choose Add to queue, or annotate one item from the item's detail page. LangSmith queues take runs and whole threads added by hand or by automation rules. A manual add puts up to 100 runs or threads in the queue in one action. A LangSmith thread rule adds a conversation once the conversation goes idle. Arize AX fills a queue with spans from a time range or with dataset examples.

Potato reads traces in three ways:

• Exports. python -m potato.trace_converter converts 15 trace formats into Potato's JSONL, including langchain for LangSmith run exports, langfuse, and otel for OpenTelemetry and OpenInference spans from Phoenix or Braintrust exporters.
• Webhooks. With trace_ingestion enabled, Potato accepts traces posted to /api/traces/webhook, /api/traces/langsmith, and /api/traces/otel, and adds each trace to the queue while reviewers work.
• Langfuse polling. A langfuse entry under trace_ingestion.sources pulls new traces from a Langfuse project on an interval.

Potato picks which incoming traces reach reviewers and in what order. Automation rules sample incoming traces by a field and a rate. The triage queue serves errored traces, thumbs-down feedback, and low automated scores to reviewers first.

Labels reviewers give a trace

The kind of label a reviewer gives depends on the label's later use. A pass or fail verdict gives a success rate, and a rubric score per criterion gives a judge something to match. A corrected output gives a training example. Each kind of label in the table below has a Potato scheme that records the label.

The tracing platforms record the same judgments as scores. Langfuse stores a corrected output on a trace or observation. LangSmith reviewers write assertions that become a dataset example's expected output, and Braintrust can write a free-form score to the expected field. LangSmith pairwise queues show runs from two experiments side by side, and the reviewer picks A, B, or Equal.

Review queue features by tool

The Potato team checked the licenses, plans, and features below against each project's own pages on 2026-11-02.

The Argilla repository states that the original authors have moved on and that the project publishes bug fixes and patches but no new features. The hosted platforms suit a team that already traces production traffic in one of those platforms. On a hosted platform, a reviewer opens a trace in the same tool where an engineer debugs the trace.

Humanloop's migration guide states that the Humanloop platform would be sunset on September 8, 2025, and that the platform and all data would be permanently inaccessible after that date. On August 13, 2025, TechCrunch reported that the co-founders and most of the team had joined Anthropic, and an Anthropic spokesperson said Anthropic did not acquire Humanloop's assets or intellectual property. For automated evals and human review in one hosted product, the closest hosted replacements are Langfuse, LangSmith, Braintrust, and Arize. A team that needs agreement between blind reviewers on every trace can add Potato beside any of the four platforms.

Reviewers per trace and agreement

One reviewer per trace gives a label with no check on the label, and two blind reviewers show which traces are ambiguous and whether the rubric is clear. A LangSmith queue sets a number of reviewers per run, and LangSmith reviewers cannot see each other's feedback. Braintrust hides peer scores until a reviewer submits. With Require all scores on, a Braintrust row completes once every assigned reviewer has scored the row. Phoenix lists every annotation on a span with the author. In the Langfuse forum, users reported on November 7, 2025 that two people still could not annotate the same trace independently in a queue. The workaround in the discussion is one score config per reviewer. The latest comment is dated April 22, 2026 and still describes workarounds. Langfuse score analytics computes Cohen's kappa between any two score names, two at a time.

In Potato, num_annotators_per_item sets how many reviewers see each trace. The /admin/iaa dashboard reports Cohen's kappa, Fleiss' kappa, and Krippendorff's alpha for categorical schemes and weighted kappa for ordinal schemes, and /admin/api/step_agreement reports agreement on step-level labels. An adjudicator settles disagreements on the /adjudicate page, and Adjudication and Resolving Disagreement covers when to adjudicate. Inter-Annotator Agreement Explained covers the choice of coefficient.

Labels back into the tracing tool and into eval datasets

Reviewed labels are useful once the labels reach the evaluation dataset that the next version is tested on. LangSmith reviewers add a run or a whole thread to a dataset from the queue. Langfuse adds observations to a dataset with a link to the source trace, and the Langfuse scores API returns corrected outputs as scores of type CORRECTION. Phoenix propagates span annotations to datasets, and Braintrust builds datasets from reviewed logs.

Potato returns labels in three ways:

• Scores on the original trace. With trace_ingestion.writeback, a trace that came from the Langfuse poller or the LangSmith webhook gets one score per scheme once min_annotators people have answered. A radio answer is the majority label, and a multiselect answer is the set of classes more than half the reviewers chose. The comment and metadata record the annotator count and the share of reviewer pairs who agreed. A tied vote waits for adjudication. Step-level schemes such as process_reward also score each Langfuse observation.
• Training data. The trajectory_correction exporter writes each corrected trace as an SFT record (prompt, completion) and a DPO record (prompt, chosen, rejected), as Trajectory Editing for SFT/DPO describes.
• Files and datasets. Potato exports to CSV, JSONL, Parquet, a Hugging Face dataset, and agent_eval. The agent_eval export aggregates scores per trace. Potato's datasets keep versioned evaluation sets.

A Potato review queue fed by Langfuse

With the config below, Potato polls a Langfuse project every 60 seconds and gives each trace to two reviewers. Reviewers answer three questions, and once both reviewers have answered, Potato sends each answer back to the trace as a Langfuse score named review.outcome, review.failure_mode, and review.step_correctness. The config is complete and passes python -m potato.validate_cli --strict.

annotation_task_name: Support agent trace review, two reviewers per trace
task_dir: .
output_annotation_dir: annotation_output/
data_files:
  - data/traces.jsonl
item_properties:
  id_key: id
  text_key: task_description
user_config:
  allow_all_users: true
  users: []
 
# Pull new traces from Langfuse every 60 seconds, and send each merged
# answer back to the trace as a Langfuse score once two people have judged it
trace_ingestion:
  enabled: true
  api_key: change-me
  sources:
    - type: langfuse
      api_url: https://cloud.langfuse.com
      public_key: pk-lf-...
      secret_key: sk-lf-...
      poll_interval: 60
  writeback:
    trigger: fully_annotated
    min_annotators: 2
    score_prefix: review
 
# Two reviewers per trace, so agreement can be measured on every trace
num_annotators_per_item: 2
 
instance_display:
  fields:
    - key: steps
      type: agent_trace
      label: Agent trace
 
annotation_schemes:
  - annotation_type: radio
    name: outcome
    description: Did the agent resolve the customer's request?
    labels: [pass, fail]
  - annotation_type: multiselect
    name: failure_mode
    description: Which problems does the trace show?
    labels: [wrong_tool, hallucinated_fact, ignored_instruction, unsafe_action, none]
  - annotation_type: process_reward
    name: step_correctness
    description: Mark the first step that went wrong.
    steps_key: steps
    mode: first_error

Start the server with potato start config.yaml -p 8000. Write-back reaches only traces that came from Langfuse or LangSmith, and traces loaded from data/traces.jsonl stay in Potato. For traces exported as files, convert the export first:

python -m potato.trace_converter --input langfuse_traces.json --output data/traces.jsonl --input-format langfuse

Potato uses the api_key to protect the webhook endpoints. Write-back reads the Langfuse keys from LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY when both variables are set, and otherwise from the langfuse source. curl -H "X-API-Key: $KEY" http://localhost:8000/admin/api/trace_writeback shows how many scores are pending, sent, failed, and held.

Limits of Potato as a review queue

• No production tracing. Potato reads traces from a tracing tool and does not replace the tracing tool.
• No reviewers. Invite your own team, or recruit reviewers on Prolific, as Crowdsourcing Integration describes.
• Write-back to two services. Scores go to Langfuse and LangSmith only. Text answers are never sent, and LangSmith receives trace-level feedback only. Potato's documentation states that LangSmith delivery follows LangSmith's published API and has not been run against the service.
• Polling for Langfuse only. LangSmith runs arrive through the webhook or an export.

Frequently asked questions

What is an annotation queue for agent and LLM traces?

An annotation queue is a list of traces, spans, or conversations that a team has set aside for people to label, with a fixed set of questions for each item. A reviewer opens the next item, answers the questions, and marks the item done. Langfuse, LangSmith, and Arize AX call the feature an annotation queue. Potato runs the same workflow on your own server, with several reviewers per trace and agreement reported.

Which tools support human review and labeling of LLM traces?

Langfuse, LangSmith, Braintrust, Arize Phoenix, and Arize AX attach human labels to traces inside the tracing tool. In Label Studio Enterprise, reviewers label Langfuse and Braintrust traces through templates. Potato shows reviewers traces from Langfuse, LangSmith, OpenTelemetry, and 12 other formats, measures agreement between reviewers, and sends the result back to Langfuse or LangSmith.

What are the alternatives to Humanloop for evals and human review?

The hosted alternatives to Humanloop include Langfuse, LangSmith, Braintrust, and Arize. Each of the four platforms combines tracing, automated evaluators, and human review, and Langfuse is MIT-licensed and free to self-host. The Humanloop platform was sunset on September 8, 2025. Potato is a free, self-hosted alternative for the human review part, with agreement between blind reviewers and scores written back to Langfuse or LangSmith.

Further reading

• AI Agent Evaluation Tools Compared and LLM Human Evaluation and Preference Data Tools Compared
• Potato vs LangSmith and Langfuse for Agent Evaluation
• How to Evaluate AI Agents and Annotating Agent Trajectories
• Agent evaluation in Potato and LLM evaluation in Potato
• How to Measure Agreement Between an LLM Judge and Human Annotators
• Trace score write-back, in Potato's source documentation

References

1. Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks. https://arxiv.org/abs/2306.05685