Observability for traditional web services is a well-understood discipline. You instrument your code, emit spans and metrics, and watch dashboards that tell you how fast a request completed and whether it errored. Large language model (LLM) and agent systems break many of the assumptions that discipline was built on. A request to a model is not a deterministic function call that either succeeds or fails; it is a probabilistic generation whose quality varies, whose cost is measured in tokens rather than milliseconds alone, and whose behaviour can span multiple model calls, tool invocations, and retries before a user ever sees an answer. This post explains why LLM and agent observability is different, what you actually need to capture, and why the AI gateway is a natural place to capture it.
Why LLM and agent observability is different
Classic application performance monitoring (APM) grew up around synchronous, deterministic services. If you call an endpoint with the same input twice, you expect the same output, and the main things worth watching are latency, throughput, and error rates. LLM systems violate almost every part of that model, and the gaps matter.
Non-determinism
The same prompt sent to the same model can produce different completions on different calls. That means a "successful" HTTP 200 response tells you almost nothing about whether the output was correct, relevant, or safe. Observability for LLMs has to account for variation in output quality, not just whether the call completed. You cannot assert a fixed expected value the way a unit test does; instead you watch distributions, sample outputs, and track quality signals over time.
Multi-step execution
Agents rarely answer in a single model call. A typical agent loop plans, calls a tool, reads the result, calls the model again, perhaps calls another tool, and only then responds. A single user interaction can fan out into a dozen underlying operations. If you only log the final response, you lose all visibility into where time went, which tool failed, or why the agent looped. Tracing, in the distributed-systems sense of a parent span with child spans, becomes essential rather than optional, because the interesting failures live in the middle of the chain.
Token cost as a first-class signal
In a conventional service, cost is mostly a function of compute time and is tracked by your infrastructure team, not your application. With LLMs, every request consumes input and output tokens that map directly to money, and that cost varies with prompt length, context size, and model choice. A slow endpoint is a performance problem; a verbose prompt template quietly multiplied across millions of calls is a budget problem. Token usage and cost therefore belong in your telemetry right next to latency.
Tool calls and external effects
Agents call functions, query databases, hit search APIs, and invoke other services. Each of those tool calls has its own latency, its own failure modes, and its own cost. Understanding an agent's behaviour means understanding the performance of its tools, not just the performance of the model. A good observability picture attributes latency and errors to the specific model or tool responsible, so you can tell a slow retrieval step apart from a slow generation step.
What to capture
Given those differences, a useful observability layer for LLM and agent systems should capture a specific set of signals for every request, and it should let you slice them by the dimensions that matter operationally. The core signals are:
- Full request traces — the parent request and every child step (model calls, tool calls, retries) with their timing and relationships, so you can reconstruct exactly what happened in a multi-step interaction.
- Latency — not just end-to-end, but broken down by stage, so you can separate time spent generating from time spent waiting on a tool.
- Token usage — input and output tokens per call, because tokens are the unit of both performance and spend.
- Cost — the monetary cost derived from token usage and model pricing, so budgets are observable rather than reconstructed after the fact.
- Errors — failed calls, timeouts, rate-limit responses, and guardrail blocks, attributed to the component that produced them.
- Quality and model/tool performance — signals that let you compare how different models or tools behave on the same workload.
Capturing these raw signals is only half the job. The other half is being able to group them by model, by team, and by user. When cost spikes, the first question is always "whose workload, and which model?" If your telemetry cannot answer that, you are left guessing. Attribution by model, team, and user turns a wall of numbers into something you can act on: you can see that one team's experiment doubled token spend, or that a particular model is responsible for most of your latency tail. This kind of visibility is what people mean when they talk about AI gateway observability: telemetry that is specific to the way LLM and agent traffic actually behaves.
Why the gateway is the right place to capture it
An AI gateway sits between your applications and the model providers. Every request to every model flows through it. That position has a practical consequence for observability: because every request already passes through the gateway, it can record traces, latency, tokens, cost, and errors without each application having to add its own instrumentation. In practice this means effectively zero application instrumentation is required to get a full trace of what happened, since the data the gateway needs is already in the request and response it is handling.
This matters for two reasons. First, instrumentation drift is a chronic problem. When each team is responsible for adding spans and metrics to its own code, coverage is uneven; some services are well instrumented and others are dark. Capturing at the gateway makes coverage uniform by construction, because there is one chokepoint that sees everything. Second, it decouples observability from application release cycles. You do not have to wait for every team to ship an instrumentation update to start seeing their traffic; the gateway sees it the moment it routes a request.
The TrueFoundry AI Gateway is OpenTelemetry-compliant and follows exactly this model. Because every request flows through it, it captures full request traces and records cost, latency, token usage, errors, and model and tool performance, broken down by model, team, and user. Being OpenTelemetry-compliant is important in its own right: it means the traces the gateway produces use a vendor-neutral standard, so you can send them to whatever backend your organisation already runs rather than being locked into a proprietary format.
Governance signals belong in the same layer
Observability for enterprise LLM systems is not only about performance and cost. Because these systems handle prompts and completions that may contain sensitive data, and because they are increasingly subject to internal policy and external regulation, the observability layer also has to carry governance signals. Three in particular matter.
PII masking
Traces capture real prompts and responses, which can contain personally identifiable information. An observability layer that stores raw traces without protection becomes a liability. Masking personally identifiable information in traces means you keep the operational value of the trace, the timing, the token counts, the structure of the interaction, without retaining raw sensitive content where it does not belong.
Role-scoped trace access
Not everyone who can see that a request happened should be able to read its full contents. Role-scoped access to traces lets you separate the people who need aggregate performance data from the smaller group permitted to inspect the full detail of a given interaction. This keeps observability useful across an organisation without turning every dashboard into an uncontrolled window into user data.
Audit trails
In a governed environment, who changed a policy, a budget, or a guardrail is as important as what the models did. Audit trails record those administrative changes, so that when behaviour shifts you can correlate it with a configuration change rather than chasing a phantom. A budget that was quietly raised, a guardrail that was relaxed, a routing policy that was edited: each of these can explain a change in cost or behaviour, and each should be recorded.
A minimal configuration example
Turning on trace export at the gateway is a configuration task, not a code change. In the TrueFoundry AI Gateway you enable the OpenTelemetry traces exporter and point it at the OTLP endpoint of whatever backend you use. The setup lives under AI Gateway → Controls → Settings → OTEL Config: you enable the OTEL Traces Exporter, choose HTTP or gRPC, and enter the full traces endpoint. For a Grafana Cloud backend over HTTP, the endpoint looks like this:
Protocol: HTTP
Endpoint: https://otlp-gateway-prod-ap-south-1.grafana.net/otlp/v1/traces
Encoding: proto
Headers:
Authorization: Basic <base64-encoded-credentials>
For HTTP you choose an encoding of proto or json, and you can optionally add headers such as an authorization header. If you prefer gRPC, you include the scheme in the endpoint: http:// for plaintext (for example http://otel-collector:4317), and omitting the scheme implies TLS. The key point is that because the gateway is already handling every request, enabling this exporter is all it takes to start emitting standards-compliant traces; there is no per-application code to write.
Key takeaways
- LLM and agent observability differs from traditional APM because requests are non-deterministic, often multi-step, and carry token-denominated cost and tool-call behaviour that a simple success/error signal cannot describe.
- For every request you should capture full traces, latency, token usage, cost, errors, and quality, and you should be able to attribute them by model, team, and user.
- The AI gateway is a natural capture point: because every request already flows through it, it can record this telemetry with effectively zero application instrumentation, giving uniform coverage that does not depend on each team instrumenting its own code.
- Governance signals belong in the same layer: PII masking protects sensitive trace content, role-scoped access controls who can read full traces, and audit trails record who changed a policy, budget, or guardrail.
- Because the TrueFoundry gateway is OpenTelemetry-compliant, enabling trace export is a configuration step that sends standards-based traces to whatever backend your organisation already runs.
Comments (0)
Want to join the discussion?
Log In to Comment