Agent steps can feel fast until one tool call burns twenty seconds and makes the whole turn look broken. Logging a latency budget for each agent step exposes slow tools before users blame the model. You set a budget, measure against it, and fix the step that misses.
Define a Budget Per Step Type
Not every step needs the same ceiling. Reads can carry tighter limits than writes. Search can allow more than a status ping. Put budgets beside the step name in config so planners and executors use the same language. Separate those roles when belts grow; see separate planner prompts from executor tool belts now.
Log Latency Beside Cost Metrics
Cost without latency hides user pain. Latency without cost hides spend. Log both per tool call and roll them up per agent turn. The cost pattern already exists in log tool-call costs per agent turn in plain metrics; extend it with duration fields.
Fail Fixtures When Budgets Break Overnight
Make chronic misses into fixtures. If a tool exceeds budget three nights running, record the failing run and attach it to the skill pack. Overnight fixtures keep regressions honest; see record failed agent runs as overnight fixtures.
Gate Write Tools That Blow the Budget
Slow write tools need extra caution because retries multiply side effects. Require human gates when a write tool repeatedly misses budget or chains retries. That control matches require human gates on high-impact agent write tools.
Review Allowlists When Latency Spikes
New tools often arrive with chat-scoped permissions and no performance story. Freeze allowlists per environment so a spike points to a known set of tools; see freeze tool allowlists per environment, not per chat.
Publish a Weekly Latency Digest
Once a week, list the five slowest steps, the budget, and the owner. Keep it boring. Boring digests get fixed. Heroic fire drills do not scale.
When a step misses budget, try caching, smaller payloads, or fewer retries before raising the ceiling. Raising budgets without a fix just hides the same pain.
Alert on Budget Drift, Not Single Spikes
One slow call can be noise. A rising p95 across a day is a signal. Alert on drift windows so on-call is not paged for every retry. Store the budget next to the alert rule so responders know the target without hunting docs.
Share the weekly digest with tool owners on the same day each week. Consistency trains the habit of fixing the slowest five before they become incidents.
Run the practice once this week, write what broke, and adjust one variable only on the next pass.
When budgets conflict with product deadlines, document the exception with an expiry date so temporary looseness does not become the new normal.
FAQ
What is a good default budget?
Start with p95 targets per step type, then tighten after two weeks of data.
Should budgets live in prompts?
No. Keep budgets in config and metrics. Prompts should not invent SLAs.
How do timeouts relate to budgets?
Timeouts are hard stops. Budgets are health targets that trigger fixes earlier.
Do MCP tools need separate budgets?
Yes. Quarantine new MCP servers and assign budgets before production write access.
What if the model is the slow part?
Split model time from tool time in logs so you do not optimize the wrong layer.
Who owns a missed budget?
The service owner of the tool, not the chat user who triggered it.