Playbooks9 min readJune 25, 2026
AI Agent Observability: What to Log When Your Agent Ships to Prod
A production agent without observability is a slow-motion outage. Here's the minimum viable telemetry stack.
Log the context window, not just the prompt
Store the exact tokens the model saw: system, retrieved docs, tool schemas, user turn. Without this, replaying a failure is guesswork.
Trace tool calls end-to-end
Every tool call gets: input, output, latency, cost, retries. Aggregate weekly. The 3 highest-cost tools drive 80% of your bill.
Run offline evals on every deploy
A frozen set of 100–300 examples with graded expected outputs. If pass-rate drops >2% on a deploy, block it. This one check catches most regressions.
Key takeaways
- Log the full context, not just prompts.
- Trace tools with cost + latency.
- Gate deploys on eval pass-rate.
Frequently asked
Do I need a dedicated LLM observability tool?
Only after ~1M requests/mo. Before that a Postgres table + Grafana beats most vendors and keeps your data yours.