Observability in single-service systems is relatively straightforward. A request comes in, you assign it a trace ID, you log every operation with that ID, you can reconstruct the full request path from the logs.
In multi-agent systems, observability is harder. A user request triggers an orchestrator. The orchestrator spawns three sub-agents. One sub-agent calls two tools and makes one model call. Another sub-agent calls a database and invokes a second model. All of this happens concurrently, across potentially different processes or services, with different timing and different failure modes.
Without a structured tracing system, you have logs from six different components with no way to connect them. When something fails, you know something failed. You do not know what, or where, or why.
The Trace Hierarchy for Agent Systems
Agent systems have a natural nesting structure: the workflow contains steps; each step may contain model calls, tool calls, and sub-agent invocations; sub-agents contain their own steps. Effective observability preserves this hierarchy in the trace structure.
The standard approach, borrowed from distributed tracing (OpenTelemetry), maps cleanly onto agent hierarchies:
- Trace ID — identifies the entire workflow, from initial user request to final output. Assigned at the workflow entry point and propagated to every component.
- Span ID — identifies a single unit of work within the trace. Each step, model call, and tool call gets its own span.
- Parent Span ID — the span that spawned this span. This is what creates the hierarchy: the orchestrator's span is the parent of each sub-agent's span; each sub-agent's span is the parent of its model calls and tool calls.
With this structure, you can reconstruct the complete execution tree for any workflow run: what ran, in what order, how long each step took, and where failures occurred.
What to Capture Per Span
Each span should capture enough to reconstruct what happened and diagnose what went wrong:
For model calls: model name and version, input token count, output token count, latency (time to first token, total latency), finish reason (stop / length / tool_use / error), temperature and sampling parameters. For debugging, the full prompt and completion — though this has cost and privacy implications that must be weighed.
For tool calls: tool name, input parameters, output (truncated if large), latency, success or error. The tool inputs and outputs are often the most useful debugging information — a wrong tool input explains most tool-related failures.
For sub-agent invocations: sub-agent name, task description, input context size, output size, whether the sub-agent succeeded or was retried, the trace ID of the sub-agent's own trace (for cross-trace correlation).
For orchestrator steps: step name, which sub-agents or tools were invoked, decision rationale (if the orchestrator made a routing decision), intermediate state.
Trace ID Propagation
The trace ID is only useful if it is propagated consistently to every component in the system. This is the part that breaks most often in practice.
Common failure modes:
Sub-agents that start a new trace instead of inheriting the parent trace. If a sub-agent library auto-generates a new trace ID, you lose the connection between the sub-agent's spans and the orchestrator's trace. You can see what the sub-agent did in isolation, but you cannot connect it to the user request that triggered it.
Tool calls that don't propagate the trace ID. If tool wrappers don't include the current span ID in their logging, tool call spans appear as standalone events rather than as children of the step that invoked them.
Async operations that lose the trace context. When steps run asynchronously — spawning background tasks, using queues, calling webhooks — the trace context must be explicitly serialized and passed along. Async operations that inherit context automatically from a synchronous call stack will lose it when the stack unwinds.
The fix is explicit trace context management: pass the trace ID and parent span ID explicitly through every function call, API call, and message queue publish rather than relying on implicit thread-local context propagation.
Production Sampling Strategy
Capturing full traces for every workflow in production is expensive. Token counts, model inputs and outputs, and tool call payloads add up quickly at scale.
A practical sampling strategy:
Always capture: Span start/end timestamps, span IDs and parent IDs, success/failure status, error messages, token counts, latency. This is the minimal trace that allows you to reconstruct the execution graph and identify failures.
Sample at low rate (1–5%): Full prompt and completion text, tool call payloads. This gives you enough examples for debugging and eval without logging every model call in full.
Always capture on failure: When a span fails or produces an unexpected result, capture the full context regardless of the sampling rate. Failures are rare enough that the cost is manageable, and the full context is what you need for debugging.
Connecting Traces to Evals
Traces are the foundation of production eval pipelines. An LLM-as-judge eval needs the model's input and output for a given call. A task completion eval needs the complete execution trace to verify that every required step ran and produced an acceptable output.
Storing traces in a queryable format — not just log lines — enables you to:
- Sample traces for offline eval runs by filtering on specific criteria (high latency, specific tool calls, failure paths)
- Track eval metrics over time by attaching eval results to the corresponding trace
- Build regression tests from specific failing traces — capture the trace, annotate what went wrong, add it to the eval suite
LangSmith, Langfuse, and Arize AI all provide purpose-built trace storage and query interfaces for LLM systems. OpenTelemetry-compatible backends (Jaeger, Honeycomb, Datadog APM) work for teams that want to integrate agent traces into existing observability infrastructure.
The most important decision is not which tool to use. It is to make the trace hierarchy explicit and consistent before you need it to debug a production failure.