> ## Documentation Index
> Fetch the complete documentation index at: https://trytilde.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Traces
> See every invocation as a trace, with model calls, tool calls, latency, token usage, cost, and errors, stored and viewed in Tilde.
Tilde records each agent invocation as an OpenTelemetry trace. A trace shows the model calls and tool calls the agent made, how long each took, what it cost, and where it failed.
Tilde is its own trace store. You need no separate tracing product.
The Tilde SDK sets up tracing for you. Traces appear in the agent's **Tracing** tab without any agent configuration.
## How traces reach Tilde
Click to zoom
The SDK uploads OTLP batches to the gateway, authenticated with the invocation token. Only a live invocation can upload traces, so an agent cannot write traces for work it was not given.
The gateway's embedded OpenTelemetry receiver validates each batch, and stamps it with the verified agent, run, invocation, and thread. It writes the batch to object storage before it acknowledges the upload.
Any gateway replica delivers the queued batch to ClickHouse. Delivery retries until it succeeds, and a retried span is stored once.
## Read traces in Tilde
Open the agent's **Tracing** tab.
* The **observations** grid lists model and tool calls, with filters for session, invocation, model, latency, tokens, cost, time, and any span attribute.
* The **waterfall** shows one trace's spans, nested by parent, and coloured by latency.
* The **inspector** shows a span's input, output, and metadata. Images and other media in a span open from object storage.
Tilde shows a trace only after it proves that one of its spans belongs to the agent. A trace can include spans from the gateway and from other agents the agent called.
Tilde prices each model call from its own price list, so a trace shows cost as well as tokens.
## Retention and forwarding
Tilde keeps trace history for the retention period you configure, or indefinitely if you set none. The same setting applies to logs.
To send traces to another backend as well, point Tilde at your collector's OTLP/HTTP endpoint. Forwarding runs on its own queue, so a collector outage never delays the **Tracing** tab. See [OpenTelemetry](/docs/integrations/opentelemetry).
## What a trace contains
| Captured by default | Captured only if you opt in |
| - | - |
| The invocation and its runtime calls | Prompts and completions |
| Trace context, propagated from the gateway | Tool inputs and outputs |
| Model, token usage, latency, and errors, from your framework's instrumentation | |
| Tilde SDK, sidecar, and gateway versions | |
Spans that your framework's OpenTelemetry instrumentation creates inside `run` join the same trace. Whether they include prompts and tool inputs is that instrumentation's setting. Tilde's own instrumentation never captures request bodies or authorization headers.
## Use your own OpenTelemetry provider
If your agent already configures an OpenTelemetry provider, add Tilde's span processor to it, rather than letting the SDK create a second provider.
```typescript Agent setup theme={"system"}
import { agentSpanProcessor, createAgentServer } from "@trytilde/sdk";
const provider = new NodeTracerProvider({
spanProcessors: [agentSpanProcessor, yourExistingProcessor],
});
provider.register();
createAgentServer({ tracing: "existing", run });
```
Register the provider's async context manager before you start the agent. Your spans then flow to both Tilde and your existing backend.