When an agent stuffs a huge tool result into the next model turn, you pay twice. First in tokens. Then in quality, because the model spends attention on noise instead of the decision you need.
Most agent stacks treat tool output as sacred. Whatever the API returned goes straight into the conversation. That works for a short JSON blob. It fails when the tool returns a 40KB HTML page, a full file tree, or a log dump. The next turn balloons, latency climbs, and the model starts summarizing the dump instead of acting.
Set a hard size budget before the result enters the message list. A practical default is a few thousand characters of structured text, or a fixed token ceiling if your SDK exposes one. Anything larger gets truncated, summarized, or stored behind a handle the model can request again.
Prefer structured truncation over blind cuts. Keep the schema keys the model relies on. Drop repeated rows, raw stack traces, and base64 payloads. If you must keep a sample, keep the first N items and a count of what was omitted. Tell the model the count so it can ask for a page or a filter.
Keep summaries deterministic. Use the same compressor for the same tool so retries do not produce different text and trigger new tool calls. A fixed template helps: tool name, status, key fields, sample, omitted count. Avoid free-form prose from a second model call unless you cache it by content hash.
Store the full payload outside the chat when you need audit or replay. Give the model an opaque id and a short description. On a follow-up request, fetch a slice or a filtered view. That keeps the turn small while still letting the agent dig in.
Watch for tools that look small on the happy path and explode on edge cases. List endpoints with no limit, search APIs that return full documents, and filesystem reads without a max bytes flag are common. Cap those at the tool boundary, not after the model already saw them.
Also cap nested results. An agent that calls five tools in one step can still blow the budget if each returns a modest blob. Budget per turn as well as per tool. If the sum exceeds the limit, keep the highest priority results and replace the rest with short stubs.
Log when truncation fires. Include the tool name, raw size, kept size, and whether the model later asked for more. That metric shows which tools need better defaults and which prompts push agents into dump-everything habits.
Teach the system prompt the rule in plain language: tool output may be shortened; ask for a narrower query when you need detail. Models follow that better than a silent cut with no explanation.
Size limits are not a quality downgrade. They are a control surface. Keep the next turn lean, keep the full data on disk, and let the agent request what it actually needs.
Put the size guard in middleware that inspects the raw tool response before it reaches the chat buffer. Compute the token count with the same tokenizer the model uses, compare to a configurable ceiling, and run truncation if needed.
When truncating a list, keep the first ten entries, add a line like “+ 27 more items omitted”, and store the full list in a key-value store keyed by a hash of the original payload. The model can request the omitted slice by name.
Add a monitoring hook that records each truncation event with tool name, original byte size, and final token count. Set an alert if the same tool exceeds the limit more than three times in an hour, so you can tighten its query or raise a quota.