Tool-calling agents look fine in a happy-path demo and fail in quiet ways in production. An eval harness is how you catch those failures before users do. Build it around real tool traces, not chat vibes.
A harness is a fixed set of tasks, fixtures, scorers, and a runner that replays the agent against them. You change a prompt or a tool schema, then measure what broke. Without that loop, every release is a hope.
Define tasks from production traces
Start with anonymized traces of tool chains that already happened. Pick cases that matter: billing lookups, ticket creates, file edits with side effects. Turn each into a task with a clear success check.
Avoid toy puzzles that never appear in your product. Eval sets drift from reality fast when they are invented at a whiteboard. Pair this with hard tool call limits so runaway chains cannot inflate scores.
Stub tools with deterministic fixtures
Live APIs make scores noisy. Stub tools with recorded responses and controlled error modes. Include timeouts, empty lists, permission denials, and partial JSON. The agent should recover the way your production middleware expects.
When a tool must stay live, pin versions and seed data. Flaky fixtures teach the team to ignore the harness.
Score behavior, not just the final sentence
For tool agents, the path matters. Did it call the right tool. Did it pass required fields. Did it stop after success. Did it invent an id. Score those events with assertions on the trace, then score the user-visible result.
Text-only rubrics miss silent double writes. Trace assertions catch them. This sits next to idempotent tool retries as a safety net when the model repeats a call.
Keep suites small and ranked
A hundred weak cases beat nothing, but ten sharp cases beat a hundred weak ones. Rank by severity and frequency. Gate merges on the top tier. Run the long tail nightly.
Refresh the suite when production incidents appear. Each incident should become a fixture, not a slide.
Compare against a frozen baseline
Store scores per prompt version and model id. When you change one variable, keep the rest fixed. If cost rises while quality is flat, you have a pricing problem, not a win. See also capping tool results before the next turn so eval turns stay cheap.
Wire the harness into CI early
Run the top tier on every agent config change. Fail the build on regressions in critical assertions. Publish a short report with failing task ids and the first bad tool call. Engineers fix what they can see.
An eval harness will not make agents perfect. It makes failures repeatable, ranked, and hard to ignore. That is the difference between shipping demos and shipping tools.
FAQ
What is an eval harness for tool-calling agents?
It is a fixed runner with tasks, fixtures, and scorers that replays the agent and measures tool behavior plus outcomes.
Should eval tools call live APIs?
Prefer deterministic stubs. Use live calls only when pinned and seeded, because noise teaches teams to ignore scores.
What should I score besides the final answer?
Assert on tool choice, required fields, stop conditions, and duplicate side effects in the trace.
How many cases do I need?
Start with a small ranked top tier for merge gates, then grow a longer nightly suite from real incidents.
When should the suite change?
Add a fixture after each production incident that the current suite would have missed.
How do I compare prompt changes fairly?
Freeze model id and fixtures, change one variable, and store scores per version against a baseline.