An agent hits a flaky tool, retries, hits again, and burns a quiet fortune in tokens before anyone notices. The failure looked temporary. The loop was unbounded. Cap retry budgets before agent loops melt spend and before side effects stack up in the dark.
Retry caps are product decisions, not afterthoughts. You define how many attempts, which errors qualify, and what the agent must do when the budget dies. Without that, tool-calling agents invent persistence you never priced and never logged.
Separate transport retries from semantic retries
Network blips deserve short automatic retries. Wrong arguments do not. Tag errors as transport, auth, validation, or unknown. Only transport gets a tiny auto budget. Validation failures should stop and rewrite the call, not hammer the same payload until money disappears.
If you already bound schemas tightly, validation noise drops. Pair this with bound tool schemas before agents call them so retries are not inventing fields the API never accepted.
Set a numeric budget the runtime enforces
Pick a per-tool and per-turn ceiling. Example: two transport retries, then fail the step. One semantic rewrite allowed after a validation error. Enforce in code, not in prompt prose. Prompts forget under pressure. Counters do not, and finance notices the difference.
Global turn budgets still matter. See tool call limits so a single user request cannot spawn endless tool chatter even when each individual tool looks polite and well meaning.
Make retries idempotent or refuse them
If a tool creates side effects, retries need idempotency keys or you refuse auto-retry. Charging a card twice is not resilience. Prefer make tool calls idempotent on retry patterns before you raise any budget above zero for mutating tools.
Log every retry with the error class and remaining budget. Structured traces make burn visible early, which is why teams log structured agent traces for cheaper debug work instead of reading screenshots from panicked demos.
Fail closed with a user-visible next step
When the budget ends, stop. Return a clear status and a human action: check credentials, narrow the query, or try later. Silent endless spinning trains users to distrust the agent and trains ops to chase ghosts across three dashboards.
Cap tool result size on the way back too. Huge payloads invite another confused call. Use cap tool results before the next turn so the next reasoning step stays cheap and readable.
Review spend by error class weekly
Group token and latency cost by retry reason. Auth failures need key hygiene, not higher budgets. Transport spikes need infra. Validation storms need schema work. Raising the cap is the last lever, not the first, and it should require a written reason.
Alert when a single session crosses a retry threshold. Budget discipline is a feature users feel as speed and reliability, not as a spreadsheet line that appears only after the invoice lands.
FAQ
What is a retry budget?
A hard limit on how many times a tool step may retry, enforced by the runtime with error-class rules.
Should every error retry?
No. Auto-retry transport blips. Stop or rewrite on validation and most auth failures.
Where should the cap live?
In code around the tool runner. Prompt instructions alone are not enough under failure pressure.
How do idempotency keys help?
They let safe retries avoid duplicate side effects when a response was lost after the action succeeded.
What do I show users when the budget dies?
A clear failure plus one next action, not another silent spin that looks like progress.
How do I know the budget is wrong?
Weekly cost by error class. If validation dominates, fix schemas before raising limits.