Score Retrieval Chunks Before the Model Sees Them
Similarity search returns neighbors, not truths. Scoring retrieval chunks before the model sees them filters weak context early. The agent still reasons. It just stops treating the first close paragraph as evidence.
Track latency so scoring stays inside tool budgets: log tool-call latency budgets for each agent step.
Define a Simple Score Card
Use three checks: topical match, freshness, and source trust. Score 1 to 5 each. Drop chunks below a floor before prompt assembly. Keep the card in config and diff changes like code: diff prompt changes in PR review like code diffs.
Avoid ornate rubrics. Reviewers ignore them.
Separate Retrieval From Generation
Retrieve, score, then generate. Mixing them hides why a bad chunk won. Pin model versions for the scorer and the generator: pin model versions in agent config like dep locks.
If you use an LLM judge, keep it smaller and cheaper than the main model.
Log Rejected Chunks With Reasons
Store query, chunk id, scores, and drop reason. Failed runs become fixtures: record failed agent runs as overnight fixtures. Patterns show stale docs or bad chunking.
Rotate keys that fetch private indexes on the same weekly hold as other agent tokens: rotate agent tool tokens on a weekly calendar hold.
Cap How Many Chunks Reach the Prompt
Even high scores can drown the model. Hard-cap at three to five. Prefer diversity across sources over five near-duplicates from one page.
New MCP doc servers stay read-only until scoring is proven: quarantine new MCP servers behind read-only profiles.
Eval Scoring With Known Bad Neighbors
Build a tiny set of queries with planted near-misses. Assert they drop. Without evals, threshold tweaks are vibes.
Re-run evals when embeddings or chunk sizes change.
Show Scores in Debug Mode Only
Operators need visibility. End users need answers. Keep score traces behind a debug flag so production prompts stay lean.
When scores and latency conflict, raise the floor rather than flooding the prompt.
When chunks disagree, prefer the higher trust source even if similarity is slightly lower. Conflict logs should note which chunk won and why. Silent preference for the first hit recreates the bug you tried to fix.
Rechunk documents that always lose on freshness. Scoring cannot save a stale paragraph that should have been archived.
Publish the drop threshold next to the agent version. Silent threshold edits make eval failures impossible to explain. Treat threshold PRs like model pin PRs.
For tables and code blocks, add a format score so prose-only chunks do not crowd out structured answers the user needs. Format mismatch is a common silent failure.
Cache scores for identical chunk ids within a run so retries do not pay double latency. Clear the cache when the index version changes.
FAQ
Is cosine similarity enough?
As a first filter yes. Add freshness and trust before generation.
Should humans tune thresholds weekly?
Tune when evals fail, not on a fixed chatter schedule.
What if all chunks score low?
Refuse or ask a clarifying question. Do not invent from weak context.
Can we score after the model answers?
Too late for grounding. Score before generation.
Do we need another model?
Heuristics plus metadata often beat a second LLM for v1.
How do we handle multilingual docs?
Score language match as its own axis or filter first.