An eval set is a fixed list of inputs and expected checks for an AI feature inside your app. You run it when you change a prompt, a model, or a retrieval step, and you look at the failures before users do.
A chat box with no eval set ships on taste. The first prompt works on the founder’s examples. The next model swap quietly breaks date parsing, refund wording, or the tool the feature is allowed to call. The set is how you notice that on a pull request instead of in a support ticket.
What belongs in the set
Build the set from real product paths, not from clever prompts you wrote to impress a reviewer. Each case should look like something a user already typed.
- Happy paths copied from support tickets you already resolved correctly.
- Edge inputs: empty text, mixed languages, pasted logs, and very long messages.
- Cases that must refuse, such as a request to reveal another user’s data or to invent a price.
- Cases that must call a specific tool, and cases that must not call one.
- Answers a human marked as acceptable, stored as checks rather than one sacred paragraph.
Keep each case small enough to rerun on every prompt change. Fifty tight cases beat five hundred you never run. Store the input, the checks, and a note about which screen the case came from.
How to score a run
Do not grade with a feeling that the answer sounds right. Use checks a script can fail. A second model marking the first model correct hides the same mistakes twice.
- Required phrases or fields are present.
- Forbidden content is absent, including other tenants’ names and raw API keys.
- JSON parses and matches the schema your app already validates on the server.
- The right tool was called, with arguments inside the allowed range.
- Latency and token spend stay under the budget you set for that screen.
Store the model name, prompt hash, and date with every run. A green board with no version attached is not a record. When a case fails, keep the model output next to the check that failed so the next editor can see the miss.
Common mistakes
- Scoring only the demo script from the launch deck.
- Letting the same model that wrote the answer also mark it correct.
- Updating expected text every time the wording drifts, so the set never fails.
- Ignoring refusal cases because they are not the happy path.
- Running the set only in a notebook, never in CI.
- Copying customer text into the repo without stripping names, emails, and account ids.
Safe rollout
Put the eval command in the repo next to the feature. Fail the pull request when a required case breaks. Allow a labeled override for a prompt experiment, and expire that override. An override that lives forever is a skipped test with a nicer name.
When you add a user visible behavior, add a case the same day. Treat a missing case like a missing test for a billing path. Start with the refusals and the schema checks. Those fail in ways users notice even when the prose still looks fine.
Review failures as product bugs. If the model starts citing a policy you retired, fix the source text or the prompt, then add the case so it cannot return quietly.
What to tell partners
Tell API consumers which behaviors you test on every change: schema, refusals, and tool limits. Do not promise that every sentence stays identical across model upgrades. Promise the checks, and say so when a check starts failing in production.
Eval sets are how an AI feature stays a product instead of a prompt someone edited on a Friday. Write the cases from real tickets, score them with code, and run them before the model change reaches users.
If you sell an app whose main screen is a model call, export the eval set with the repo. Buyers discount features they cannot replay. A folder of inputs, checks, and the last run result is proof the feature has an owner, which is what a buyer is pricing.