AI Code Review Policy: How to Define "Good Enough" When AI Reviews Most of Your Code

Who decides what "good enough" means when AI reviews most of the code?

One named engineering leader decides, and writes it down in the repository where both people and review bots read it. The PR author stays accountable for each individual change. The AI reviewers enforce the written bar; they never get to set it.

That sounds obvious until you look at how most teams actually adopted AI review. Someone installed a PR bot, it started commenting, and the definition of "good enough" quietly became whatever the bot's default prompt considered worth mentioning. The Tech Manager Weekly community thread that prompted this page (issue #504, October 2026) described the result well: AI reviewers flagging edge cases nobody would ever hit, engineers burning time on them, and nobody quite sure whose call it was to say "ship it."

The mechanism is simple. A review model is rewarded for finding things. It has no cost model for your engineers' time, no memory of the trade-offs you made deliberately last quarter, and no sense of which code paths real users reach. Left alone, it optimizes for recall. Recall without a severity threshold is noise, and noise trains people to stop reading. Once that happens the one comment that mattered gets dismissed with the other forty.

Where should the policy be written down?

In two places: a human-readable policy file in the repo, and the operational rules mirrored into each review tool's own config. A wiki page nobody's tools read does not count.

Two tool details shape how you write the config. GitHub's documentation says Copilot code review reads only the first 4,000 characters of a custom instruction file, so the severity rules go at the top, before anything stylistic. CodeRabbit exposes a review profile setting (quiet, chill or assertive, with chill as the default) and per-path instructions via reviews.path_instructions. If your team is complaining about nitpicks, check whether someone switched the profile to assertive before you rewrite any prompts.

Which findings should block a merge?

Only verified security, data-integrity and correctness problems on paths that real inputs reach. Everything else is advice. The policy below uses four levels because that is the smallest number that separates "stop" from "fix soon" from "consider" from "ignore."

The line that does the most work is the reachability rule: an edge-case finding can only be S0 or S1 if the reviewer names the concrete input, caller or production path that triggers it. "What if this list is empty?" is an S2 note. "getInvoices() returns an empty list for accounts created before the 2024 migration, and this line divides by its length" is an S1. Asking the model for the trigger forces it to do the reasoning that separates a real bug from a theoretical one, and it gives the human reading the comment something checkable.

This mirrors a standard that predates AI review. Google's engineering practices guide tells reviewers to favor approving a change once it definitely improves the overall health of the code, even if it isn't perfect, and to prefix minor points with "Nit:" so the author knows they are optional. The AI-era version just has to be explicit enough for a machine to follow.

How does a finder/verifier split work?

One agent finds candidate issues; a second agent, with fresh context, tries to confirm each blocking finding before anyone sees it. Unconfirmed findings get downgraded to notes. This is the pattern several teams in the #504 thread converged on independently, and it works for a structural reason.

A finder tuned to catch everything will produce false positives, and the same model asked "are you sure?" about its own output tends to defend it. A verifier that only sees the finding and the relevant code has no stake in being right. Its job is narrow: reproduce the problem with a failing test, a trace or a concrete input, or say it can't. That is cheaper than a full second review and it filters exactly the class of comment that wastes engineers' time.

In practice, use a different tool or model family for the verifier where you can, so the two don't share the same blind spots. Keep the verifier's scope to S0 and S1 findings; verifying nits costs more than reading them. And log how often the verifier downgrades the finder. If the rate is high, tighten the finder's instructions instead of adding a third agent.

Why put intent notes next to the code?

Because the reviewer can't see the meeting where you decided. A large share of "unlikely edge case" comments are the model rediscovering a trade-off the team already made on purpose: no retry here because the caller retries, no validation because the input comes from our own queue, a linear scan because the list never exceeds 50 items. Without a written note, every AI pass re-raises it.

A one-line convention fixes most of this: // INTENT: <what we chose> because <why>. Revisit if <condition>. The "revisit if" clause matters. It tells both the reviewer and the next engineer when the decision stops being valid, which turns a comment into a testable assumption. For decisions bigger than a function, use a short decision record in docs/decisions/ and link it from the PR.

The other half of this is planning assumptions away before an agent writes code. When an agent starts from a vague ticket, it fills the gaps with guesses, and the reviewer then flags the guesses. Small, well-specified changes produce fewer findings because there is less ambiguity to argue about. That's also why the template caps diff size as a human-review trigger.

When must a human review AI-written or AI-reviewed code?

Whenever the blast radius is high, the AI tools disagree, or the change touches the rules themselves. The template lists the paths: authentication, payments and billing, data migrations, infrastructure, cryptography, personal data handling, and the policy, CI and reviewer configuration. That last one is easy to miss. An agent that can edit its own review instructions can lower its own bar.

Enforce these triggers with CODEOWNERS and branch protection, not with an instruction to the AI tool. Instructions are probabilistic; branch protection isn't. If you are mapping this to a broader risk model for agent-written code, the piece on how agentic systems fail at scale covers why compounding small errors is the real risk, and the AI code review analysis on aistrategy.guide covers how the bottleneck moves to the human reviewer once code volume goes up.

What do you do when two AI tools keep rewriting the same code?

Stop the automation after the second round and hand the decision to a person. The loop is predictable once two tools have different preferences: the coding agent writes it one way, the review bot suggests another, an auto-fix applies the suggestion, the next review pass from a different tool flags the new version. Each step is locally reasonable. The PR never converges.

The rule in the template is mechanical on purpose: if AI tools change the same hunk more than twice in one PR, label it review-loop, freeze automated commits, and let the PR author decide (the policy DRI if the author is unsure). Then do the two steps that stop it from happening again: write the decision as an INTENT note, and add a rule to the reviewer config. A loop that recurs means the policy is missing a rule, and the fix belongs in the policy.

The AI code review policy template (copy and paste)

Replace the angle-bracket placeholders, commit it to your repo, and mirror sections 2, 3 and 6 into your review tools' config files. It's deliberately short. A policy that fits on one screen gets read; one that needs a table of contents gets ignored.

# AI Code Review Policy  (v1.0, owner: <NAME>, reviewed: <DATE>)

## 1. Purpose
AI tools review every pull request. This policy defines what blocks a
merge, who confirms findings, and when a human must review.
"Good enough" = the change clearly improves the health of the codebase
and introduces no S0/S1 issue. It does not have to be perfect.

## 2. Severity levels
S0 BLOCKER   Security hole, secret in code, data loss or corruption,
             broken auth/permissions, breaking a public API contract,
             failing tests, license violation.        -> blocks merge
S1 MUST-FIX  Correctness bug on a reachable path, missing test for
             changed behavior on a core path, measurable perf
             regression on a hot path.                  -> blocks merge
                                                           (DRI may waive
                                                           in writing)
S2 SHOULD    Maintainability, unclear naming, plausible edge case
             with no demonstrated trigger.   -> non-blocking; ticket OK
S3 NIT       Style, alternative implementations, hypothetical
             scenarios.            -> summary only, never inline, never blocks

Reachability rule: an edge-case finding is S0/S1 only if the reviewer
names the concrete input, caller or production path that reaches it.
No named trigger = S2 at most.

## 3. Finder / verifier split
- Finder (agent A) reviews the diff and proposes findings with severity
  and evidence.
- Verifier (agent B: different tool or model, fresh context) receives
  each S0/S1 finding plus the relevant code, and must either reproduce it
  (failing test, trace or concrete input) or downgrade it to S2.
- Only verified S0/S1 findings block. Unverified ones are posted as notes.

## 4. Intent notes
- Deliberate trade-offs are recorded next to the code:
    // INTENT: <what we chose> because <why>. Revisit if <condition>.
  or in the PR description under "Deliberate choices".
- Larger decisions go in docs/decisions/ (one file per decision).
- Reviewers treat documented intent as accepted unless it causes an S0.

## 5. Human review required when
- The diff touches: auth/, payments/, billing/, migrations/, infra/,
  crypto, PII handling, or this policy/CI/reviewer config.
- Any verified S0 finding.
- Finder and verifier disagree on an S0/S1.
- Diff > <400> changed lines (split the PR instead where possible).
Enforced via CODEOWNERS + branch protection, not by the AI tool.

## 6. Loop breaker
If AI tools change the same hunk more than twice in one PR (one suggests,
another rewrites or reverts), stop auto-fix:
1. Label the PR "review-loop" and freeze automated commits.
2. The PR author decides; the policy DRI decides if the author is unsure.
3. Record the decision as an INTENT note.
4. Add a rule to the reviewer config so the loop does not recur.

## 7. Ownership
- Policy DRI: <NAME, ROLE>. Owns this file, revises it quarterly.
- Change DRI: the PR author, whoever or whatever wrote the code.
- AI tools enforce this policy. They do not change it.

## 8. What we measure (monthly)
- Share of AI comments acted on vs dismissed, by severity.
- Escaped defects that an S0/S1 rule should have caught.
- Number of "review-loop" labels.

Who owns the policy, and how often should it change?

Put one name on it. A policy owned by "the platform team" or "engineering" is owned by nobody, and it drifts the moment the review tools ship a new default. The DRI is usually the engineering manager or staff engineer accountable for the codebase. They don't have to write every rule, but they decide disputes, approve changes, and run the quarterly review. If the term is new to your team, the explainer on directly responsible individuals covers how to assign one and what changes when agents write the code.

Review the policy on a calendar and on signals. Calendar: quarterly, because the tools change that fast. Signals: the three numbers at the bottom of the template. If engineers dismiss most AI comments at a given severity, that level is miscalibrated. If production incidents trace back to something an S1 rule should have caught, the rule is too loose or the verifier is too lenient. If review-loop labels keep appearing, the config is missing a decision. The AI team metrics guide has more on choosing measures that don't turn into targets.

AI code review best practices that sit around the policy

• Keep PRs small. Both humans and models review small diffs better, and small diffs give the reachability rule less surface to argue about.
• Treat accepted-comment rate as a calibration signal. It tells you whether the bot's bar matches yours. Don't turn it into a target for anyone.
• Separate style from substance. Formatters and linters handle style deterministically. Tell the AI reviewer to skip anything a linter already enforces.
• Review the reviewer's config like code. Changes to review instructions go through a PR with a human approver, per the human-review triggers.
• Watch the debt you are not seeing. Code that passes every AI review can still be the wrong abstraction. The AI technical debt guide covers the slower-moving failure modes.

Sources

• Tech Manager Weekly #504 (19 October 2026), community discussion on AI code reviews flagging unlikely edge cases.
• Google Engineering Practices: The Standard of Code Review
• GitHub Docs: Customizing Copilot code review
• CodeRabbit configuration reference
• aistrategy.guide: AI Code Review, When the Bottleneck Moves

Frequently asked questions