Failed agent runs disappear into chat scrolls and overnight logs nobody checks. Treat those failures as fixtures worth keeping. Capture inputs, tool traces, and expected checks so CI replays the break instead of hoping the next demo is lucky.
Capture Failures While Context Is Fresh
When a run fails, save the prompt pack version, tool allowlist hash, model id, and redacted transcript slice. Capture it that same night. By morning, you will not reconstruct the exact flake. Pair this with how you ship eval fixtures beside every new agent skill pack.
Store Fixtures Next to the Skill
Keep failure fixtures in the skill folder rather than a private doc. Name them after the symptom: timeout-on-tool-x, schema-mismatch-y. Include a short expected assertion. Reviewers can see coverage when the skill changes. Version the pack like an API so fixtures stay aligned, as in how you version prompt packs like API contracts.
Redact Before You Commit
Remove secrets, customer names, and raw tokens. Keep structure. If the failure needs a credential shape, use fakes. Never paste live keys into fixtures or session dumps, consistent with how you keep agent secrets out of live session context packs.
Replay on a Budget
Overnight replay should respect retry caps and cost ceilings. A fixture that loops through tools forever becomes a second incident. Cap retries the way you cap agent retry budgets before loops melt your spend, and log cost per turn during replay.
Promote Flakes Into Blocking Checks
Once a failure reproduces twice, make the fixture blocking for that skill. Leaving a fixture non-blocking forever lets the bug return on the next model bump. Planner versus executor splits still matter: fixture the layer that broke, as when you separate planner prompts from executor tool belts.
Close the Loop in Standup
List new overnight fixtures in standup with owner and status: reproducing, fixed, wontfix. Delete fixtures that no longer match retired tools. A living failure library is better than a graveyard of stale JSON.
Tag fixtures with environment and model family so overnight replay does not mix sandbox failures with production-only tool quirks.
Share a weekly count of new fixtures and fixed fixtures so leadership sees learning velocity instead of only outage stories.
When a fixture needs live network access, mark it integration-only and keep a mocked twin for pull request CI. Overnight can run the heavy path; every PR should stay cheap.
Rotate fixture ownership when a skill changes hands. Orphan fixtures rot and start failing for the wrong reasons.
Archive fixed fixtures with a closed date rather than deleting immediately. A thirty-day archive helps if a regression returns after a model bump.
Keep the practice simple and repeatable. Write the rule in one sentence on a sticky note near your keyboard or bag so the habit survives busy weeks when motivation dips and shortcuts look tempting. Review that sentence once a week and tighten wording if it stopped matching how you actually work.
FAQ
What makes a failure first-class?
It is saved, versioned, replayable, and asserted in CI near the skill.
How fast should we capture it?
Same night, while model id, prompt version, and tool trace are known.
What must be redacted?
Secrets, personal data, and anything that cannot live in the repo.
Should every one-off glitch become a fixture?
Promote after repeat or clear severity. Noise fixtures hide signal.
Where do overnight runs execute?
In CI or a scheduled sandbox with cost and retry caps.
Who owns fixing a new fixture?
The skill owner named in the pack metadata, not the chat that first saw the error.