Prompt caching looks like free speed. Reuse a big system prompt, pay less on the next call, watch latency drop. The trap is silent cost growth when the cache key keeps missing or when you cache the wrong bytes.
Most providers hash a stable prefix. If your prefix wobbles, you pay full price again and again while dashboards still say caching is on. Treat cache hit rate as a first-class metric.
Stabilize the prefix before you celebrate
Put truly static instructions first. Keep timestamps, request ids, and user names out of the cached prefix. One changing character near the start can invalidate the whole block.
If you inject tool schemas, sort keys and freeze whitespace. Pretty-printing that shifts between services will thrash the cache.
Measure hits, not vibes
Log cache hit, miss, and write events per route. Track average prompt tokens billed with and without a hit. If hit rate falls under your target, stop shipping prompt tweaks until you know why.
Pair this with streaming token budget controls so a miss does not also explode output cost.
Do not cache secrets or per-user private blobs
A shared prefix should be safe to reuse across tenants. Put private documents, raw emails, and credentials in the dynamic tail or in tool results you can cap. See capping tool results for keeping those tails small.
Caching a customer transcript in a global prefix is a privacy bug dressed as an optimization.
Watch TTL and cold starts
Caches expire. Low-traffic routes look cheap in tests and expensive in production because every hour is a cold write. Batch warmups can help for hot paths. For quiet paths, design prompts that stay correct even when every call misses.
Avoid mega-prompts that only win on hits
A 20k-token policy doc feels fine when hits are high. When hits drop, every turn becomes a bill shock. Split rarely used policy into on-demand retrieval. Keep the always-on prefix lean enough that a miss is survivable.
Also review agent loops that resend the full history each hop. History growth can dwarf any cache win. Cap what re-enters the next turn.
Price the failure mode in reviews
In design review, ask what happens at 0% hit rate. If the answer is unacceptable, shrink the prefix or add a hard token budget. Caching is a discount on a stable shape, not a license for unbounded prompts. Related reading: rotating sandbox API keys so cost experiments stay scoped.
Prompt caching pays when the prefix is boring, measurable, and safe. It punishes teams that chase hit myths while the key keeps changing under them.
FAQ
What usually breaks prompt cache hits?
A changing prefix: timestamps, ids, unsorted JSON, or unstable whitespace near the start of the prompt.
Which metrics matter most?
Cache hit rate, billed prompt tokens with and without hits, and cost per route at your real traffic pattern.
Can I put user documents in the cached prefix?
No. Keep private or per-tenant content in the dynamic tail or in capped tool results.
Why do low-traffic routes stay expensive?
TTL expiry causes frequent cold writes, so quiet paths miss more often than busy ones.
Should I keep a huge policy prompt if hits are high?
Only if a 0% hit scenario is still affordable. Otherwise move rare policy to retrieval.
How do agent loops interact with caching?
Resending growing history can erase cache savings. Cap what re-enters each turn.