Streaming tokens looks cheap until the bill arrives. Each partial chunk still counts toward input and output pricing. Retries, tool loops, and long contexts multiply the same request. Without a budget, an in-app AI feature can burn the day’s spend on one chatty user or one stuck agent.
The goal is not to stop streaming. Users like early tokens. The goal is to stream inside a hard limit you can measure and enforce on the server.
What the budget must cover
Price the whole turn, not only the visible reply. A turn includes the system prompt, history, retrieved chunks, tool results fed back into the model, and every retry.
- Max output tokens per response, enforced before the stream starts.
- Max input tokens after retrieval and history truncation.
- Max tool rounds per user action.
- A wall-clock timeout so a hung upstream cannot hold a worker.
- Per-user and per-tenant daily caps that stop new turns when crossed.
Count planned tokens, not only completed ones. If you cancel a stream early, you still paid for what left the provider. Log both requested and billed figures when the provider returns them.
Streaming without runaway spend
Open the stream only after you have reserved budget for that turn. If the reserve fails, return a plain error and do not start the request. Mid-stream, stop when the output counter hits the cap. Send a short footer that the reply was truncated, then close the connection cleanly.
Do not let the client set max tokens. The client can suggest a mode (short answer, long draft). Your server maps that mode to a fixed ceiling.
- Truncate history with a rule: keep the last N turns, or summarize older turns once and reuse the summary.
- Cap retrieval: top-k chunks with a hard character limit, not “everything that matched.”
- Cache identical system prompts and static tool schemas where the provider supports it.
- Prefer smaller models for routing and classification; reserve the large model for the final answer.
Product choices that save money
Default to short answers. Offer “expand” as a second turn with its own budget. Avoid auto-retry on timeout without a backoff and a retry counter. A silent retry loop is how a support bot becomes a finance incident.
Show users a soft meter when they approach a daily limit. Soft meters cut surprise bills and cut rage tickets. Hard stop still belongs on the server.
Ops and alerts
Alert on turns that hit the token cap, on tenants that burn more than a set share of the daily pool, and on tool loops that exceed round limits. Review those traces weekly. Often the fix is a tighter prompt or a missing allowlist, not a bigger budget.
Ship dashboards that split cost by feature flag and by model. Without that split, you will raise the global ceiling when only one screen is broken.
Streaming stays useful when the budget is real. Reserve spend before the first token. Stop at the cap. Measure the whole turn including tools and retries.
When you buy or sell an app that streams model output, ask for the per-turn token caps, daily tenant limits, and last month’s cost by feature. A product with open-ended streams and no reserve step is a cost risk. Put those numbers in the handover notes before you price the deal.