A new skill lands with a clever prompt and three tools. Nobody adds a fixture. Two weeks later a schema tweak breaks the happy path in production. Ship eval fixtures beside every new agent skill pack so changes fail in CI while the blast radius is still a pull request.
Fixtures are not a research paper. They are small recorded inputs, expected tool calls, and pass or fail checks you can run on every change.
Store fixtures next to the skill, not in a wiki
Put JSON or golden transcripts in the same directory as the skill pack. Version them with the pack the way you version prompt packs the same way you version APIs. A wiki page nobody updates is not a harness.
Name fixtures by behavior: refund_happy, refund_missing_id, refund_human_gate. Behavior names beat ticket numbers when you are debugging at speed.
Assert tool calls, not only final prose
Agents can sound confident while calling the wrong tool. Assert tool name, required args, and that forbidden tools stay silent. Build on the harness mindset from build eval harnesses for tool-calling agents.
Keep planner fixtures read-only when you separate planner prompts from executor tool belts now so plan packs cannot pass by cheating with writes.
Pin model ids inside the eval config
Evals that float on whatever default is live today lie next month. Pin model ids in eval config the same way you pin model IDs in config, never in chat history logs. Record cost per fixture run so expensive suites get trimmed on purpose.
Cap retries in eval the way you cap agent retry budgets before loops melt your spend. Flaky retries hide broken skills.
Include a human-gate fixture for write tools
If the skill can pay, delete, or post, add a fixture that expects a gate pause. Patterns from require human gates on high-impact agent write tools should show up as failing tests when someone removes the gate.
Log structured traces during eval so failures are cheap to read, following log structured agent traces for cheaper debug work.
Block merge when fixtures are missing
CI should fail if a new skill directory has no fixtures folder. Soft reminders lose to schedule pressure. Make the empty folder a hard stop, then grow coverage after the first three cases exist.
Review fixture failures weekly with the same seriousness as unit tests. Skills without green fixtures are not ready, even when the demo looks smooth.
When a fixture fails, fix the skill or update the fixture with a written reason. Silent fixture edits that only make CI green recreate the original problem. Treat golden updates like API contract changes that need a reviewer who did not author the skill.
Keep a short scoreboard: fixtures per skill, last fail age, and cost per full suite run. Skills with zero fixtures stay blocked. Skills with stale fails stay noisy until someone owns the repair.
FAQ
What is an eval fixture for an agent skill?
A small recorded input plus expected tool calls and pass or fail checks stored with the skill.
Where should fixtures live?
Beside the skill pack in version control, not in a separate wiki.
What should we assert?
Tool names, required arguments, forbidden tools, and gate pauses for high-impact writes.
Why pin model ids in eval?
Floating defaults make last month’s green suite meaningless after a silent model change.
How many fixtures does a new skill need?
Start with happy path, one missing-input case, and one gate or deny case.
Can demos replace fixtures?
No. Demos are not repeatable CI and they hide regressions.