An AI diff comment is a note a model writes on a pull request after it sees a patch, not the whole repository. The comment is useful only if a reviewer can act on it without opening a chat and re-explaining the change.
Teams that paste an entire service into the prompt get essays. Teams that send the diff, the failing test, and one house rule get comments a human can accept or dismiss in a minute. If the comment needs a meeting, it is not a review comment.
What to send the model
Keep the payload small and owned by the change. The bot should quote a line from the patch, not invent a tour of the codebase.
- The unified diff for the files in the pull request, with generated files stripped out.
- The test output that failed, trimmed to the assertion and the stack frame that points at your code.
- One house rule the team already follows, such as prefer early return, or do not add a public method without a caller.
- The ticket title, if it states the intended behavior in a sentence a stranger could check.
Do not send secrets, customer rows, or production logs. A review bot that needs a database dump to comment on a rename is the wrong tool. Strip tokens from the diff before the request leaves your network.
How to keep comments usable
Ask for a fixed shape. Three fields are enough: file and line, the problem, and a concrete change. Reject free-form essays at the parser. If the model cannot name a line in the diff, drop the comment instead of posting it as a general note.
Cap the count. Five comments on a 40 line patch is noise. One or two that name a bug, a missing test, or a broken contract is enough. Sort by a severity you defined, not by how confident the model sounds.
Show the comment as a suggestion the author can apply, not as a blocking check, until you have a month of accept rates. A red required check that the team always overrides trains people to ignore the bot.
Common mistakes
- Sending the full file history because the diff felt too small to be interesting.
- Letting the model invent a product requirement the ticket never stated.
- Posting the same comment on every revision after the author already fixed it.
- Treating a style nit and a null crash as the same severity.
- Storing the prompt and the diff in a log that support staff can search without a reason.
Safe rollout
Start on one repository and one language. Run the bot on draft pull requests first, visible to the author only. Record accept, dismiss, and edit. If fewer than a third of comments get accepted after two weeks, tighten the prompt before you widen the audience.
Pin the model version. A silent upgrade changes comment tone and the rate of false positives. Review that change the same way you review a linter rule. Keep a kill switch that stops posting without undeploying the service.
Give authors a way to mark a path as out of scope, such as generated code or a lockfile. The bot should skip those paths instead of arguing with a file nobody wrote by hand.
What to tell the team
Publish the payload rules and the comment cap in the contributing guide. Say that the bot does not approve merges and does not replace a human on auth, billing, or migration changes. Those diffs still need a person who owns the system.
AI diff comments earn their place when they shorten review. Keep the input small, keep the output shaped, and turn the bot off for paths it keeps getting wrong.
If you later sell a developer tool that includes this review bot, keep a short record of accept rate, model vendor, and which repos opted in. A buyer will ask whether the comments are a feature people keep or a toggle they disable. Those three numbers answer that faster than a demo.