Potato vs LangSmith and Langfuse for Agent Evaluation: A Practical Comparison

LangSmith and Langfuse are built to trace and monitor LLM applications in production. Potato is built to collect structured human judgments about what those traces show and to measure how much the judges agree. Teams that need both usually capture in an observability tool and run the human study in Potato.

The three tools overlap enough that the comparison comes up often, and they overlap at one point only. All three can attach a human judgment to a recorded agent run. What they are organized around differs, and that difference decides which one fits the question you are asking.

What each tool is built for

LangSmith instruments, traces, and debugs LLM applications, with the deepest integration into LangChain and LangGraph. Traces from an application built on those frameworks arrive without extra instrumentation, and the platform is organized around inspecting runs, tracking latency and cost, and running evaluation experiments over datasets. Its annotation queues support rubric feedback keys, a configurable number of reviewers per run, pairwise queues, and reviewers who cannot see each other's feedback.

Langfuse covers the same ground as open source, for any LLM application rather than one framework, and bundles prompt management with tracing. Scoring is available at the trace level and at the observation (span) level, and its categorical score configs let you define a fixed label set and attach several named scores to one span.

Potato is an annotation platform that was extended to agent traces. It is organized around the study rather than the application, so its primitives are who annotates what, how many people see each item, what happens when they disagree, and what the agreement statistics say about whether the labels mean anything. Agent traces are one item type among the 61 annotation types the current release registers, which is why they inherit inter-annotator agreement, training phases, adjudication, and crowdsourcing without any of that being rebuilt for agents.

None of the three is better in the abstract. The useful question is whether your bottleneck is seeing what the agent did, or trusting the human labels you are collecting about it.

Moving traces between them

Potato reads traces after the fact rather than instrumenting your code, so it registers converters for 15 trace formats:

python -m potato.trace_converter --list-formats

The list covers react, openai, anthropic, langchain, langfuse, multi_agent, mcp, otel, atif, webarena, web_agent, swebench, swe_agent_trajectory, claude_code, and aider. LangSmith run exports go through the langchain converter, since there is no separate langsmith one. CrewAI, AutoGen and LangGraph logs go through multi_agent.

Converting an export is one command in each direction of the workflow:

# LangSmith runs, exported from LangSmith, converted for annotation
python -m potato.trace_converter \
  --input langsmith_export.jsonl \
  --output data/traces.jsonl \
  --input-format langchain
 
# Langfuse traces, exported from Langfuse
python -m potato.trace_converter \
  --input langfuse_traces.json \
  --output data/traces.jsonl \
  --input-format langfuse

If your framework is not on the list, --auto-detect picks a converter from the field signatures, and a new one is a subclass of BaseTraceConverter. Detection takes the first converter that matches, so name the format explicitly once you know which one fits your files.

Traces can also arrive without an export step. Potato's trace ingestion accepts runs posted to /api/traces/* in the LangSmith run payload format, so a LangChain callback or the tracing SDK can push runs into the annotation queue as they happen:

trace_ingestion:
  enabled: true
  api_key: "your-api-key"
  notify_annotators: true

What Potato adds once the trace is in

The schemas are where an annotation platform earns its place. trajectory_eval carries the correctness label, a hierarchical error taxonomy with subtypes, severity levels with configurable weights, and a running quality score, in one schema attached to one trace:

annotation_schemes:
  - annotation_type: trajectory_eval
    name: step_evaluation
    description: "Mark each step and classify anything that went wrong"
    steps_key: conversation
    step_text_key: text
    error_types:
      - name: execution
        subtypes: [wrong_tool, wrong_args, api_error]
    severities:
      - name: minor
        weight: -1
      - name: critical
        weight: -10
    show_score: true

The severity weights are the part that matters for comparing traces. A flat count of marked steps says one agent made four mistakes and another made two, and the weighted score says whether those four were cosmetic and the two were dangerous.

Comparative evaluation runs through the pairwise schema, which has three modes. Binary mode shows two clickable tiles with an optional tie button, scale mode replaces the choice with a slider between the two items, and multi_dimension mode asks for an independent A/B verdict per named dimension such as helpfulness, accuracy, and safety.

Coding agents get their own display, because a unified diff read as a JSON string is unreviewable:

Potato's coding trace display with unified diff rendering, syntax highlighting, and file tree

The coding_trace display renders diffs with a file tree and collapses long terminal output, and the code_review schema collects line-anchored comments in configurable categories, per-file rating dimensions, and a pull-request-style verdict. The coding_eval exporter then writes process reward model labels, code review annotations, DPO preference pairs, and SWE-bench compatible scores, which is the shape that training data needs rather than the shape a dashboard needs.

Web agent traces render as screenshots with SVG action overlays and a filmstrip:

Web agent trace viewer showing screenshots with SVG action overlays and filmstrip navigation

Multi-agent runs have their own annotation surfaces rather than being flattened into one stream of steps. The registry carries schemas for an annotator-editable interaction graph, cross-agent failure attribution, handoff review, per-agent and per-team scorecards, tool contention over a shared timeline, and emergent behavior tagging. The same is true for embodied and multimodal agents, with computer-use trajectories, full-duplex voice interaction, and video temporal grounding as first-class schemas.

Measuring the judgments, not only collecting them

The difference that shows up latest in a project is what happens after two annotators disagree. Potato treats a disagreement as a measurement rather than a conflict to resolve, so the same agreement statistics, adjudication queue, training phases, and gold items that a text annotation study uses apply to agent traces without extra configuration. Potato's own tool comparison records, as of the 2026-06-24 check, that the tracing platforms are built to collect human judgments as scores and that none of them documents a reliability statistic over those judgments, and that Langfuse replaces a previous annotation on the same trace, so independent double annotation there needs a workaround.

That matters when the evaluation is the evidence. A success rate built from single unreviewed judgments has no error bar, and a reviewer who has drifted looks exactly like an agent that has improved.

Watching an agent while it runs

Potato can also run an agent inside the annotation page rather than replaying a recording. The live_agent block connects a vision model to a headless browser and streams the session to the annotator, who can pause it, send instructions, or take over manual control:

live_agent:
  endpoint_type: anthropic_vision
  ai_config:
    model: claude-sonnet-4-20250514
    api_key: ${ANTHROPIC_API_KEY}
  max_steps: 30
  allow_takeover: true
  allow_instructions: true
 
instance_display:
  fields:
    - key: agent_trace
      type: live_agent
      label: "Live Agent Session"
      display_options:
        show_overlays: true
        show_filmstrip: true

Three situations make this worth the setup over post-hoc review. Stopping a run before a destructive action is a safety measure that recorded traces cannot provide. A human takeover recorded mid-run becomes demonstration data. And watching how an agent recovers from an intervention answers a question that no completed trace contains.

Running them together

The common production setup uses both, and the division is clean because the tools want different things. Instrument the agent with LangSmith or Langfuse and let every run be traced. Sample the runs worth a human opinion, which is usually the failures, the edge cases, and a random control sample. Convert those into Potato, run the annotation study with several annotators per trace, and read the agreement report alongside the scores. Feed the error taxonomy counts back into development, where the subtype distribution says whether you have one systematic failure or a long tail.

python -m potato.trace_converter \
  --input langfuse_failed_traces.json \
  --output data/traces_to_review.jsonl \
  --input-format langfuse
 
potato start config.yaml -p 8000

Choosing

Reach for LangSmith when the application is built on LangChain or LangGraph and the need is production visibility, since traces arrive with no instrumentation work and the managed service removes the infrastructure question. Reach for Langfuse when you want that same visibility as open source across several providers, with prompt versioning in the same tool. Reach for Potato when the output you need is a defensible set of human labels, when the study needs several annotators per item and an agreement figure, when the agents are coding or multi-agent or embodied enough that the trace needs a purpose-built display, or when the data has to stay on your own infrastructure. Potato is GPL-3.0-or-later and self-hosted, with no per-seat or per-trace cost.

Statements about LangSmith and Langfuse here reflect their documentation as observed on 2026-06-24, and both ship quickly. Corrections are welcome on the GitHub repository. For the setup itself, see the agent evaluation guide, and for a worked example from raw traces to exported judgments, see Evaluating AI Agents.