When an agent misbehaves, chat screenshots are a weak debug surface. You need structured traces: tool name, args hash, latency, token counts, and outcome codes in one line per step. Without that, every incident becomes folklore.
Structured traces make failures comparable across runs. They also keep debug cost down because you search logs instead of replaying entire sessions by hand.
Emit one event per tool step
Log start and end for every tool call. Include tool name, schema version, latency ms, and a stable request id. Store args as a redacted hash plus a small allowlisted field set. Never dump secrets into the log stream.
This sits next to rotating API keys in agent sandboxes. Traces should help you debug without becoming a second secret store.
Record outcomes as machine codes
Use short codes: ok, validation_error, timeout, upstream_4xx, upstream_5xx, policy_deny. Put human text in a separate message field. Codes let you count failures. Prose alone forces grepping stories.
Attach the first failing assertion id when you run evals, so CI and production speak the same language as your eval sets in apps.
Cap payload blobs before you store them
Tool results can be huge. Store size, content type, and a truncated preview. Park full bodies in object storage only when an incident flag is on. Default traces stay small so query cost stays small. See capping tool results before the next turn for the live-path twin of this rule.
Huge default logs train teams to sample randomly. Small default logs train teams to read them.
Correlate model turns with tool turns
Give each user request a trace id. Child spans cover model completion and each tool call. When a bad final answer appears, you can jump to the exact tool that poisoned context instead of rereading the whole chat.
Include model id and prompt version on the model span. Otherwise you cannot tell a model change from a tool change.
Sample successes, keep all hard failures
Keep 100 percent of validation errors, policy denials, and upstream 5xx. Sample successful ok paths if volume is high. Sampling successes protects budget. Keeping failures protects learning speed.
Watch cost the same way you watch streaming tokens without melting budget. Trace storage is part of the agent bill.
Build one query that answers who broke
Ship a saved query: failures by tool name in the last hour, grouped by outcome code. On-call should open that first. If the answer takes a custom script every time, the trace format is not done.
Structured traces will not fix the agent. They make the next fix cheaper than the last one.
FAQ
What belongs in a structured agent trace?
Per step: tool name, schema version, latency, redacted args, outcome code, and a shared trace id with the model turn.
Should I log full tool payloads?
Default to size plus a short preview. Store full bodies only when an incident flag requires them.
Why use outcome codes?
Codes aggregate cleanly. Prose-only errors force slow manual reading across incidents.
Which events must never be sampled away?
Keep all validation errors, policy denials, and upstream 5xx. Sample high-volume successes if needed.
How do traces relate to evals?
Reuse the same assertion ids and tool names so CI failures and production failures compare directly.
What is the first on-call query to build?
Failures by tool name for the last hour, grouped by outcome code.