Open-source LLM observability and evaluation
INFRO vs Langfuse
Langfuse is an open-source platform for tracing, evaluating and improving LLM applications: nested traces across model calls, tools and retrieval, prompt management with versions and labels, LLM-as-a-judge and human evaluation, datasets and experiments, and dashboards for quality, cost and latency. It is MIT licensed outside its enterprise folder, self-hostable on Docker, Kubernetes or Terraform, and has been part of ClickHouse since January 2026 with its roadmap unchanged. It records what your application did; it does not sit in the request path, so it does not route, fail over or refuse a request.
- Observability and evaluation are the whole job. Routing and provider access are solved, nobody is asking for a budget that refuses a request, and a gateway in the request path would be scope you do not need.
- You need prompt management. Langfuse versions prompts, deploys them by label and compares versions on cost, latency and scores. INFRO has no prompt management, and no date for it.
- You trace agents, not only model calls. Langfuse records a tree of spans across tool calls, retrieval steps and model calls. INFRO records one record per gateway request, and a span tree is something you assemble in your own collector.
- Evaluation is a product surface you will live in: datasets, experiments, annotation queues and custom scores. INFRO's eval monitors score a share of served chat answers against a rubric, which is a much narrower thing.
- You want to self-host open-source software, or start at no cost. Langfuse has a free Hobby tier and is free to run yourself. INFRO's plans are paid with a 14-day trial, and it runs in your own infrastructure only under an enterprise agreement.
- You need your traces held in a specific region today. Langfuse Cloud offers US, EU and Japan regions; INFRO's regional processing is on the roadmap.
This section is not a courtesy. If one of these describes you, pick Langfuse — a customer who adopts the wrong layer on our advice is worse for us than one we never had.
Where the two actually diverge.
- Beside the request, or in it
- Langfuse receives what your application sends it — from its SDKs, from OpenTelemetry, or from a gateway such as LiteLLM — and records it after the call. INFRO is the endpoint your application calls: it picks the provider connection, retries and fails over across your own accounts, and records the request as it serves it. The first tells you what happened. The second also decides what happens, which is the only place a budget, an allowlist or a failover rule can act.
- An alert is not a limit
- Langfuse can notify you when spend crosses a threshold. INFRO checks the request's estimated cost against the organization's, project's, member's and key's budgets before the provider is called, and refuses it with an error that names the budget when it would go over. If an unexpected bill is the risk you are managing, that difference is the whole question.
- Where the cost figure comes from
- Langfuse computes cost at ingestion from model definitions you can edit, or stores the cost you send it verbatim. INFRO estimates every request at the vendor's list rate, or from the Team plan at the rates your contracts actually carry, says unpriced rather than guessing when it has no rate, and on the Business plan checks the month's estimate against OpenAI's and Anthropic's own cost APIs and states the variance.
- They can run together
- Plenty of teams will want Langfuse's evaluation and prompt tooling and a control plane in the request path. INFRO pushes OpenTelemetry spans from the request path to a collector you name (from the Team plan), and Langfuse accepts OpenTelemetry. We have not published a tested recipe for that pairing yet, so treat it as a starting point rather than a documented integration.
Side by side.
Our column carries dashes too. A comparison where one side is all checks is one nobody believes, and rightly.
| Capability | Langfuse | INFRO |
|---|---|---|
| Sits in the request path | norecords traces sent to it | yesOpenAI-compatible endpoint |
| Routing, retries and failover | noa BYOK gateway is on its roadmap | yesacross your own provider accounts |
| Budgets that refuse a request | nospend alerts notify | yesorg, project, member, key |
| Model and provider allowlists | no | yes |
| Estimated cost per request | yesinferred from model definitions, or ingested | yesat your negotiated rates from Team |
| Reconciliation against the provider's own bill | no | partialOpenAI and Anthropic, Business plan; variance stated |
| Nested agent traces | yesspans across tools, retrieval and models | partialone record per request; OpenTelemetry export |
| Prompt management | yesversions, labels, playground | nonot built |
| Evaluations and datasets | yesLLM-as-a-judge, annotation, experiments | partialeval monitors on served chat answers |
| Self-hosting | yesMIT licensed outside ee/ | partialenterprise agreement |
| Free entry | yesHobby tier, or self-host | nopaid plans, 14-day trial |
Our reading of each product's public documentation as of October 2026. We do not quote competitors' prices — theirs change and a wrong number about someone else's business is worse than none. Think a row is wrong? Tell us and we will fix it.
Work out what the difference is worth
Estimate what your current model mix costs at your providers' list rates and the plan it needs, or have us read your real usage and tell you where we would and would not help.