Review Langfuse Traces in Potato

Potato reads Langfuse traces in two ways, by polling a Langfuse project for finished traces or by converting a trace file with python -m potato.trace_converter --input-format langfuse. A polled trace gets the reviewers' merged answers back in Langfuse as scores on the trace and on each judged observation, and a converted trace keeps its results in Potato's exports.

Langfuse is an open-source, self-hostable AI engineering platform for debugging and evaluating LLM and agent applications, and the Langfuse repository is MIT-licensed outside its ee folders. In the Langfuse data model, a trace represents a single request or operation. An observation is one step inside a trace, such as an LLM call, a tool call, or a retrieval step, and observations nest to follow the structure of the application. A session groups the traces of one user interaction, and a score is a value attached to a trace, an observation, or a session.

In Langfuse's own annotation queues, reviewers score traces, observations, and sessions against a score config. Potato adds several blind reviewers per trace, agreement between those reviewers, and adjudication of disagreements, then returns the result to the same trace in Langfuse. Human Review Queues for LLM and Agent Traces compares the review features of Langfuse, LangSmith, Braintrust, Arize, and Potato.

Polling compared with an export file

The table compares the two paths:

Polling suits a team that wants reviewed scores next to each trace in Langfuse. An export file suits a fixed sample whose results can stay in Potato. Traces picked for a paper or for a held-out evaluation set are typical fixed samples.

Polling a Langfuse project

The config below polls Langfuse Cloud once a minute and gives each trace to two reviewers. The first poll fetches every trace that started in the past seven days. Once both reviewers have answered, Potato sends each merged answer back to the trace as a score named human.resolved, human.helpfulness, or human.step_check. The config is complete and passes python -m potato.validate_cli --strict.

annotation_task_name: Langfuse trace review
task_dir: .
output_annotation_dir: annotation_output/
data_files:
  - data/traces.jsonl
item_properties:
  id_key: id
  text_key: task_description
user_config:
  allow_all_users: true
  users: []
 
# Pull every trace that started in the last 7 days, then poll once a minute.
# Keys for write-back come from LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY.
trace_ingestion:
  enabled: true
  api_key: change-me
  sources:
    - type: langfuse
      api_url: https://cloud.langfuse.com
      public_key: pk-lf-...
      secret_key: sk-lf-...
      poll_interval: 60
      lookback_hours: 168
  writeback:
    trigger: fully_annotated
    min_annotators: 2
    score_prefix: human
    langfuse:
      score_configs: true
 
num_annotators_per_item: 2
 
instance_display:
  fields:
    - key: steps
      type: agent_trace
      label: Langfuse trace
      display_options:
        collapse_observations: true
 
annotation_schemes:
  - annotation_type: radio
    name: resolved
    description: Did the agent answer the user's question correctly?
    labels: [correct, partly_correct, wrong]
  - annotation_type: likert
    name: helpfulness
    description: How helpful was the final answer?
    size: 5
    min_label: Not helpful
    max_label: Very helpful
  - annotation_type: process_reward
    name: step_check
    description: Mark each step correct or incorrect.
    steps_key: steps
    mode: per_step

Start the server with potato start config.yaml -p 8000. Potato needs a data file at start, and any traces already in data/traces.jsonl join the polled traces in the queue. A trace from the file appears in the display above only when the trace has a steps list.

Each entry under trace_ingestion.sources with type: langfuse starts one poller. The poller reads these keys:

Traces the poller takes

The poller reads GET /api/public/v2/observations for the time window and groups the rows by trace id. A server without that endpoint gets the legacy GET /api/public/traces API instead. The Langfuse migration guide states that Langfuse Cloud serves the legacy read endpoints until November 16, 2026, and the poller needs no change for the switch. A trace counts as finished once the root observation has an end time and settle_seconds have passed. A trace whose root never ends is taken after max_trace_seconds, with metadata.complete set to false. Potato skips any trace already in the item store, so a restart adds no trace twice.

Each trace becomes one item with the id langfuse_<trace id>. The item's steps list holds the child observations in start order, and each step keeps the observation's input, output, type, name, model, and observation id. The trace name becomes task_description, and a trace with no name uses the trace input. The session id, user id, tags, input, and output go under metadata. The poller fetches every trace in the window and does not filter by name, tag, or environment. Automation rules and the triage queue decide which polled traces reviewers see first.

Converting a Langfuse export

The langfuse converter reads one trace object, or a list of trace objects, in JSON or JSONL. Each trace needs an observations list of full observation objects. The legacy GET /api/public/traces/{traceId} endpoint returns a trace in that shape, and the list endpoint GET /api/public/traces returns observation ids rather than objects. Rows from the v2 Observations API have to be grouped by traceId into one object per trace before conversion, with the observations under observations. Convert the file:

python -m potato.trace_converter \
  --input langfuse_traces.json \
  --input-format langfuse \
  --output data/traces.jsonl

The command writes one JSON line per trace, and --auto-detect recognizes the format from an observations list whose first entry has a type. The converter maps each field as follows:

The converter keeps observations in file order and drops every other observation type. Langfuse's observation types also include agent, tool, chain, retriever, evaluator, embedding, and guardrail, and a trace instrumented with those types loses those steps in conversion. The trace output, metadata, session id, and observation level are not carried over either. Poll the project instead when a trace relies on those types or fields.

For a converted file, point the display and any step-level scheme at conversation. In the polling config, swap the instance_display block and the step_check entry under annotation_schemes for these:

instance_display:
  fields:
    - key: task_description
      type: text
      label: User request
    - key: conversation
      type: agent_trace
      label: Langfuse trace
 
annotation_schemes:
  - annotation_type: process_reward
    name: step_check
    description: Click the first step that went wrong.
    steps_key: conversation
    mode: first_error

Displays and schemes for Langfuse traces

The agent_trace display shows each step as a card with a type badge, and the display reads both the steps list from the poller and the conversation list from the converter. The three-pane eval_trace display splits the same steps into reasoning, function calls, and the final answer. Set the display's key to steps or conversation to match the path the traces came in by.

Write-back sends trace-level answers from radio, likert, slider, number, select, and multiselect schemes. Step-level questions use process_reward, trajectory_eval, tool_call_review, failure_attribution, or gui_trajectory. On a polled trace, each step label that min_annotators reviewers gave becomes a score on the step's own observation. A text box records rationale for Potato's export and is never sent. For questions about a whole Langfuse session, a sessions block with enabled: true groups polled traces by metadata.session_id, and schemes with session_level: true are answered once per session on the /sessions page (session-level scoring).

The MAST failure taxonomy and τ-bench agent evaluation designs give ready-made trace-level schemes for agent failures.

Scores in Langfuse

Write-back sends one score per scheme once min_annotators reviewers have answered. With trigger: adjudicated, Potato waits for an adjudicator's decision instead, and an adjudicated value replaces the vote under either trigger. Each score is named <score_prefix>.<scheme name> and carries the value below:

A tied vote sends nothing until an adjudicator decides or another reviewer breaks the tie. The comment states how the value was reached, for example Potato: 2 of 2 annotator(s) chose correct. The metadata holds the reviewer count, the answer counts, and the share of reviewer pairs who agreed. No username is sent. With langfuse.score_configs: true, Potato creates a Langfuse score config named after each score the first time Potato sends one. Langfuse then lists the resolved categories and the 1 to 5 helpfulness range in its filters.

Write-back reads the Langfuse keys from LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY when both variables are set, and otherwise from the langfuse source. Potato gives each score a fixed id per trace and scheme. Langfuse replaces a score only when the id, the name, and the date all match (preventing duplicate scores). An answer that changes on a later day therefore adds a second score dated that day. Check delivery with the admin API key:

curl -H "X-API-Key: $KEY" http://localhost:8000/admin/api/trace_writeback

The response counts pending, sent, failed, and held scores and gives the reason each held trace is waiting.

Results in Potato's exports

Every trace, polled or converted, keeps its annotations in annotation_output/<user>/user_state.json. The agent_eval export aggregates the answers per trace, with a label distribution and majority for each single-choice scheme and a mean for each numeric one:

python -m potato.export --config config.yaml --format agent_eval --output results/

Export formats lists jsonl, parquet, huggingface, and the other formats. Session-level answers go to annotation_output/session_annotations.jsonl. The /admin/iaa dashboard reports agreement between reviewers for each scheme, and Inter-Annotator Agreement Explained covers the choice of coefficient. The datasets page builds versioned evaluation sets from polled traces with Import ingested traces. Corrected trajectories become SFT and DPO records through trajectory editing.

Limits

• Write-back covers polled traces only. A trace from a data file, converted or not, keeps its results in Potato.
• Text answers stay in Potato. Write-back sends no free-text answer, and session-level answers are not sent either.
• The poller takes the whole project. Sampling and priority come from automation rules and the triage queue after a trace arrives.

Frequently asked questions

What is Langfuse?

Langfuse is an open-source AI engineering platform that traces LLM and agent applications, evaluates the traces, and manages prompts. The repository is MIT-licensed outside its ee folders, and Langfuse runs as Langfuse Cloud or on your own server. Potato connects to Langfuse to put several human reviewers on each trace and send their merged answers back to Langfuse as scores.

What does Langfuse record for agent observability?

Langfuse records each agent run as a trace and each step inside the run as an observation nested under the trace. A step is an LLM call, a tool call, a retrieval step, or another operation inside the run. Observations take one of ten types, including generation, agent, tool, retriever, and guardrail, and sessions group the traces of one user interaction. When a trace contains an observation of a type other than span, event, or generation, Langfuse draws an agent graph from the observations' timing and nesting. Potato's poller reads those observations and turns each one into a step that reviewers can label.

Further reading

• Human Review Queues for LLM and Agent Traces and AI Agent Evaluation Tools Compared
• Agentic Annotation, for the other 14 trace converters
• How to Measure Agreement Between an LLM Judge and Human Annotators, for comparing Langfuse LLM-as-a-judge scores with reviewer labels
• Potato vs LangSmith and Langfuse for Agent Evaluation
• Trace score write-back, in Potato's source documentation