# Review Langfuse Traces in Potato

Source: https://www.potatoannotator.com/docs/agent-evaluation/langfuse

**Potato reads Langfuse traces in two ways, by polling a Langfuse project for finished traces or by converting a trace file with `python -m potato.trace_converter --input-format langfuse`. A polled trace gets the reviewers' merged answers back in Langfuse as scores on the trace and on each judged observation, and a converted trace keeps its results in Potato's exports.**

[Langfuse](https://langfuse.com/docs) is an open-source, self-hostable AI engineering platform for debugging and evaluating LLM and agent applications, and the [Langfuse repository](https://github.com/langfuse/langfuse) is MIT-licensed outside its `ee` folders. In the [Langfuse data model](https://langfuse.com/docs/observability/data-model), a trace represents a single request or operation. An observation is one step inside a trace, such as an LLM call, a tool call, or a retrieval step, and observations nest to follow the structure of the application. A session groups the traces of one user interaction, and a [score](https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk) is a value attached to a trace, an observation, or a session.

In Langfuse's own [annotation queues](https://langfuse.com/docs/evaluation/evaluation-methods/annotation-queues), reviewers score traces, observations, and sessions against a score config. Potato adds several blind reviewers per trace, agreement between those reviewers, and adjudication of disagreements, then returns the result to the same trace in Langfuse. [Human Review Queues for LLM and Agent Traces](/docs/guides/human-review-queues-for-llm-traces) compares the review features of Langfuse, LangSmith, Braintrust, Arize, and Potato.

## Polling compared with an export file

The table compares the two paths:

| | Polling (`trace_ingestion`) | Export file (`trace_converter`) |
|---|---|---|
| Traces | Every trace in the project that started inside the look-back window, then each new trace once the trace finishes | The traces in the file |
| Langfuse API | Observations API v2, with the legacy traces API as a fallback | None, since you produce the file |
| Item field the display reads | `steps`, one entry per observation | `conversation`, one turn per thought, action, or result |
| Observation types kept | All types | `GENERATION`, `SPAN`, and `EVENT` |
| Session id | Kept under `metadata.session_id` | Not kept |
| Scores sent to Langfuse | Yes | No |

Polling suits a team that wants reviewed scores next to each trace in Langfuse. An export file suits a fixed sample whose results can stay in Potato. Traces picked for a paper or for a held-out evaluation set are typical fixed samples.

## Polling a Langfuse project

The config below polls Langfuse Cloud once a minute and gives each trace to two reviewers. The first poll fetches every trace that started in the past seven days. Once both reviewers have answered, Potato sends each merged answer back to the trace as a score named `human.resolved`, `human.helpfulness`, or `human.step_check`. The config is complete and passes `python -m potato.validate_cli --strict`.

```yaml
annotation_task_name: Langfuse trace review
task_dir: .
output_annotation_dir: annotation_output/
data_files:
  - data/traces.jsonl
item_properties:
  id_key: id
  text_key: task_description
user_config:
  allow_all_users: true
  users: []

# Pull every trace that started in the last 7 days, then poll once a minute.
# Keys for write-back come from LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY.
trace_ingestion:
  enabled: true
  api_key: change-me
  sources:
    - type: langfuse
      api_url: https://cloud.langfuse.com
      public_key: pk-lf-...
      secret_key: sk-lf-...
      poll_interval: 60
      lookback_hours: 168
  writeback:
    trigger: fully_annotated
    min_annotators: 2
    score_prefix: human
    langfuse:
      score_configs: true

num_annotators_per_item: 2

instance_display:
  fields:
    - key: steps
      type: agent_trace
      label: Langfuse trace
      display_options:
        collapse_observations: true

annotation_schemes:
  - annotation_type: radio
    name: resolved
    description: Did the agent answer the user's question correctly?
    labels: [correct, partly_correct, wrong]
  - annotation_type: likert
    name: helpfulness
    description: How helpful was the final answer?
    size: 5
    min_label: Not helpful
    max_label: Very helpful
  - annotation_type: process_reward
    name: step_check
    description: Mark each step correct or incorrect.
    steps_key: steps
    mode: per_step
```

Start the server with `potato start config.yaml -p 8000`. Potato needs a data file at start, and any traces already in `data/traces.jsonl` join the polled traces in the queue. A trace from the file appears in the display above only when the trace has a `steps` list.

Each entry under `trace_ingestion.sources` with `type: langfuse` starts one poller. The poller reads these keys:

| Key | Default | Meaning |
|---|---|---|
| `api_url` | `https://cloud.langfuse.com` | The Langfuse host. A self-hosted server takes its own URL |
| `public_key`, `secret_key` | none | The project's API keys |
| `poll_interval` | `30` | Seconds between polls |
| `lookback_hours` | `24` | How far back the first poll reaches |
| `settle_seconds` | `30` | Seconds to wait after a trace finishes before taking the trace |
| `max_trace_seconds` | `3600` | Seconds after which an unfinished trace is taken anyway |

### Traces the poller takes

The poller reads `GET /api/public/v2/observations` for the time window and groups the rows by trace id. A server without that endpoint gets the legacy `GET /api/public/traces` API instead. The [Langfuse migration guide](https://langfuse.com/faq/all/deprecated-api-migration) states that Langfuse Cloud serves the legacy read endpoints until November 16, 2026, and the poller needs no change for the switch. A trace counts as finished once the root observation has an end time and `settle_seconds` have passed. A trace whose root never ends is taken after `max_trace_seconds`, with `metadata.complete` set to `false`. Potato skips any trace already in the item store, so a restart adds no trace twice.

Each trace becomes one item with the id `langfuse_<trace id>`. The item's `steps` list holds the child observations in start order, and each step keeps the observation's input, output, type, name, model, and observation id. The trace name becomes `task_description`, and a trace with no name uses the trace input. The session id, user id, tags, input, and output go under `metadata`. The poller fetches every trace in the window and does not filter by name, tag, or environment. [Automation rules](/docs/agent-evaluation/automation-rules) and the [triage queue](/docs/agent-evaluation/triage-queue) decide which polled traces reviewers see first.

## Converting a Langfuse export

The `langfuse` converter reads one trace object, or a list of trace objects, in JSON or JSONL. Each trace needs an `observations` list of full observation objects. The legacy `GET /api/public/traces/{traceId}` endpoint returns a trace in that shape, and the list endpoint `GET /api/public/traces` returns observation ids rather than objects. Rows from the v2 Observations API have to be grouped by `traceId` into one object per trace before conversion, with the observations under `observations`. Convert the file:

```bash
python -m potato.trace_converter \
  --input langfuse_traces.json \
  --input-format langfuse \
  --output data/traces.jsonl
```

The command writes one JSON line per trace, and `--auto-detect` recognizes the format from an `observations` list whose first entry has a `type`. The converter maps each field as follows:

| Langfuse field | Potato field |
|---|---|
| trace `id` | `id` |
| trace `name` | `agent_name` |
| trace `input.query`, `input.input`, `input.question`, or a string `input` | `task_description` |
| `GENERATION` observation output (`content`, `text`, or `completion`) | an `Agent (Thought)` turn in `conversation` |
| `SPAN` observation name and input | an `Agent (Action)` turn, written as `name(key=value, ...)` |
| `SPAN` observation output (`result`, `output`, or the whole object) | an `Environment` turn |
| `EVENT` observation output, or the input when no output exists | a `System (<name>)` turn |
| observation count, trace name, and models | rows in `metadata_table` |

The converter keeps observations in file order and drops every other observation type. [Langfuse's observation types](https://langfuse.com/docs/observability/features/observation-types) also include agent, tool, chain, retriever, evaluator, embedding, and guardrail, and a trace instrumented with those types loses those steps in conversion. The trace output, metadata, session id, and observation level are not carried over either. Poll the project instead when a trace relies on those types or fields.

For a converted file, point the display and any step-level scheme at `conversation`. In the polling config, swap the `instance_display` block and the `step_check` entry under `annotation_schemes` for these:

```yaml
instance_display:
  fields:
    - key: task_description
      type: text
      label: User request
    - key: conversation
      type: agent_trace
      label: Langfuse trace

annotation_schemes:
  - annotation_type: process_reward
    name: step_check
    description: Click the first step that went wrong.
    steps_key: conversation
    mode: first_error
```

## Displays and schemes for Langfuse traces

The `agent_trace` display shows each step as a card with a type badge, and the display reads both the `steps` list from the poller and the `conversation` list from the converter. The [three-pane `eval_trace` display](/docs/agent-evaluation/eval-trace) splits the same steps into reasoning, function calls, and the final answer. Set the display's `key` to `steps` or `conversation` to match the path the traces came in by.

Write-back sends trace-level answers from `radio`, `likert`, `slider`, `number`, `select`, and `multiselect` schemes. Step-level questions use [`process_reward`](/docs/features/process-reward-annotation), `trajectory_eval`, `tool_call_review`, `failure_attribution`, or `gui_trajectory`. On a polled trace, each step label that `min_annotators` reviewers gave becomes a score on the step's own observation. A text box records rationale for Potato's export and is never sent. For questions about a whole Langfuse session, a `sessions` block with `enabled: true` groups polled traces by `metadata.session_id`, and schemes with `session_level: true` are answered once per session on the `/sessions` page ([session-level scoring](https://github.com/davidjurgens/potato/blob/master/docs/agent-evaluation/session_level_scoring.md)).

The [MAST failure taxonomy](/showcase/mast-failure-taxonomy) and [τ-bench agent evaluation](/showcase/tau-bench-agent-eval) designs give ready-made trace-level schemes for agent failures.

## Scores in Langfuse

Write-back sends one score per scheme once `min_annotators` reviewers have answered. With `trigger: adjudicated`, Potato waits for an adjudicator's decision instead, and an adjudicated value replaces the vote under either trigger. Each score is named `<score_prefix>.<scheme name>` and carries the value below:

| Scheme | Score value | Langfuse data type |
|---|---|---|
| `radio`, `select`, `likert` with `labels` | The label most reviewers chose | Categorical |
| `multiselect` | The labels more than half of the reviewers chose, comma-separated | Categorical |
| `likert` without `labels` | The median point | Numeric |
| `slider`, `number` | The mean | Numeric |
| Step-level schemes | The label most reviewers gave the step, sent to the step's observation | Categorical |

A tied vote sends nothing until an adjudicator decides or another reviewer breaks the tie. The comment states how the value was reached, for example `Potato: 2 of 2 annotator(s) chose correct`. The metadata holds the reviewer count, the answer counts, and the share of reviewer pairs who agreed. No username is sent. With `langfuse.score_configs: true`, Potato creates a Langfuse score config named after each score the first time Potato sends one. Langfuse then lists the `resolved` categories and the 1 to 5 helpfulness range in its filters.

Write-back reads the Langfuse keys from `LANGFUSE_PUBLIC_KEY` and `LANGFUSE_SECRET_KEY` when both variables are set, and otherwise from the `langfuse` source. Potato gives each score a fixed id per trace and scheme. Langfuse replaces a score only when the id, the name, and the date all match ([preventing duplicate scores](https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk)). An answer that changes on a later day therefore adds a second score dated that day. Check delivery with the admin API key:

```bash
curl -H "X-API-Key: $KEY" http://localhost:8000/admin/api/trace_writeback
```

The response counts pending, sent, failed, and held scores and gives the reason each held trace is waiting.

## Results in Potato's exports

Every trace, polled or converted, keeps its annotations in `annotation_output/<user>/user_state.json`. The `agent_eval` export aggregates the answers per trace, with a label distribution and majority for each single-choice scheme and a mean for each numeric one:

```bash
python -m potato.export --config config.yaml --format agent_eval --output results/
```

[Export formats](/docs/features/export-formats) lists `jsonl`, `parquet`, `huggingface`, and the other formats. Session-level answers go to `annotation_output/session_annotations.jsonl`. The `/admin/iaa` dashboard reports agreement between reviewers for each scheme, and [Inter-Annotator Agreement Explained](/docs/guides/inter-annotator-agreement) covers the choice of coefficient. The [datasets page](/docs/agent-evaluation/datasets-and-experiments) builds versioned evaluation sets from polled traces with **Import ingested traces**. Corrected trajectories become SFT and DPO records through [trajectory editing](/docs/agent-evaluation/trajectory-correction).

## Limits

- **Write-back covers polled traces only.** A trace from a data file, converted or not, keeps its results in Potato.
- **Text answers stay in Potato.** Write-back sends no free-text answer, and session-level answers are not sent either.
- **The poller takes the whole project.** Sampling and priority come from automation rules and the triage queue after a trace arrives.

## Frequently asked questions

### What is Langfuse?

Langfuse is an open-source AI engineering platform that traces LLM and agent applications, evaluates the traces, and manages prompts. The repository is MIT-licensed outside its `ee` folders, and Langfuse runs as Langfuse Cloud or on your own server. Potato connects to Langfuse to put several human reviewers on each trace and send their merged answers back to Langfuse as scores.

### What does Langfuse record for agent observability?

Langfuse records each agent run as a trace and each step inside the run as an observation nested under the trace. A step is an LLM call, a tool call, a retrieval step, or another operation inside the run. Observations take one of ten types, including generation, agent, tool, retriever, and guardrail, and sessions group the traces of one user interaction. When a trace contains an observation of a type other than span, event, or generation, Langfuse draws an [agent graph](https://langfuse.com/docs/observability/features/agent-graphs) from the observations' timing and nesting. Potato's poller reads those observations and turns each one into a step that reviewers can label.

## Further reading

- [Human Review Queues for LLM and Agent Traces](/docs/guides/human-review-queues-for-llm-traces) and [AI Agent Evaluation Tools Compared](/docs/guides/agent-evaluation-tools-compared)
- [Agentic Annotation](/docs/features/agentic-annotation), for the other 14 trace converters
- [How to Measure Agreement Between an LLM Judge and Human Annotators](/docs/guides/llm-judge-human-agreement), for comparing Langfuse LLM-as-a-judge scores with reviewer labels
- [Potato vs LangSmith and Langfuse for Agent Evaluation](/blog/potato-vs-langsmith-agent-evaluation)
- [Trace score write-back](https://github.com/davidjurgens/potato/blob/master/docs/agent-evaluation/trace_writeback.md), in Potato's source documentation