Reviewing AI-Written Code: A Field Guide

We have reviewed a lot of AI-written code — three shipped open-source tools' worth (the case study has the inventory). The single most important thing we learned: reviewing AI code with human-code instincts will fail you, because agents fail differently than people do.

A tired human writes sloppy code — inconsistent naming, dead ends, visible seams. The sloppiness is a signal that says look closer. An AI agent writes immaculate code: consistent style, tidy comments, plausible structure — and the bug is sitting in the middle of it wearing a suit. The dominant failure mode is not sloppy-and-wrong. It is plausible-and-wrong, and plausible-and-wrong is precisely the failure mode human review is worst at catching, because polish is the heuristic we use to decide where not to look.

So the discipline changes. Here is what we actually do.

First question: does it solve the stated problem?

Not "is this good code" — is it the right code? Agents produce locally excellent solutions to subtly misread problems. Ask for X with an ambiguous edge and the agent won't ask back; it picks an interpretation, silently, and implements it beautifully. Before reading a single hunk, re-read the task, then check the diff's shape against it: are the files you expected touched? Are files you didn't expect touched? A diff that wanders outside the task's footprint is the first smell.

The specific things we check first

Invented APIs, flags, and config keys. The classic agent hallucination: a library function that doesn't exist, a plausible-looking flag the tool has never had, a config key from a different tool's schema. Anything that crosses a boundary you don't have automated checks on — CLI invocations, config file formats, external APIs — gets verified against the real documentation, not the agent's confidence.

Error paths. Agents write happy paths with conviction and error handling from muscle memory. Read every if err != nil and ask: is this error actually handled, or ceremonially wrapped and passed along while the function continues in a broken state? Does cleanup happen on the early return? In our firewall and routing work this is where the real review time went — the happy path was almost always right; the failure path was where lockouts live.

Concurrency. Shared state without synchronization, goroutines that outlive their request, channels that can deadlock on the path nobody tested. Agents reproduce the patterns of concurrent code convincingly while missing the reasoning that makes a specific instance safe. Any diff that adds a goroutine, a mutex, or a channel gets a slow read, every time.

Resource cleanup. Unclosed files and connections, missing defer, timers that never stop, contexts that never cancel. Individually trivial; at agent volume they accumulate into the leak you find in production three weeks later.

Tests that assert nothing

The trap that deserves its own section: test-shaped code. Agents are eager test writers, and a portion of what they write exercises the code without ever meaningfully checking it — asserting that no error occurred, that the result is non-nil, that a mock was called. Green, useless.

This matters double in an agentic workflow, because the whole loop leans on tests as the verification signal. A hollow test suite doesn't just fail to catch the current bug — it feeds the agent a false "done" signal for every future change. We review tests harder than implementation: for each test, name the bug it would catch. If you can't, it isn't a test. The strongest habit we built: occasionally break the implementation on purpose and confirm the new tests actually go red.

Security review is not optional

Anything touching authentication, session handling, input parsing, file paths, SQL, or shell-out gets a second, adversarial pass. Agents trained on public code reproduce public code's average security posture, and the average is not good: string-built commands, path joins on user input, comparisons that leak timing. Our tools run on routers and firewalls, so this pass is non-negotiable for us — but the checklist is the same for a web app. If the diff touches a trust boundary, review it as an attacker before you review it as a maintainer.

Diff-size discipline

The most effective review technique isn't a technique of reading — it's refusing to read. When an agent produces a 2,000-line diff, careful review is no longer possible; you will skim, the polish will lull you, and plausible-and-wrong will get through. We reject large diffs without reviewing them and re-cut the task smaller. This is emotionally easy once you internalize the economics: generated code is cheap. Throwing away an afternoon of agent output costs minutes to regenerate under a better-specified, smaller task. Reviewer attention is the scarce resource; spend it on diffs sized for it.

Make the agent produce review artifacts

The agent can lighten the load it creates. We have the agent end every task with:

• A summary of what changed and why — per file, in prose, in the commit message or PR description
• Risk notes — what it is least confident about, what it couldn't test, what assumptions it made

Two caveats from experience. The summary is a map, not the territory — verify the diff does what the summary says, because a mismatch between the two is itself a finding, and the summaries are occasionally aspirational. And risk notes are only useful if someone reads them; ours have flagged real problems, in the agent's own words, that we nearly skimmed past.

When to trust, when to re-verify

Review effort should follow risk, not novelty. Over months, verified conventions accumulate — the error-handling shape, the test patterns, the packaging layout — and re-litigating them in every review is wasted attention. We read those at skim speed. Full attention goes to: anything crossing a trust boundary, anything concurrent, anything touching the safety mechanisms (the auto-revert paths in our tools are read line-by-line, every diff, no exceptions), and anything in territory the agent hasn't proven itself in yet. Calibrating that dial — loose where conventions are established, tight where the blast radius is real — is, increasingly, the reviewer's actual job.

The reviewer's checklist

1. Re-read the task first — then check the diff's footprint against it
2. Verify every boundary-crossing name — APIs, flags, config keys, against real docs
3. Read the error paths, not the happy path — that's where agents coast
4. Slow-read anything concurrent — patterns aren't proofs
5. For every test, name the bug it would catch — no answer, no test
6. Adversarial pass on trust boundaries — auth, parsing, paths, SQL, shell-out
7. Reject big diffs unread — re-spec smaller; generated code is cheap, your attention isn't
8. Demand summaries and risk notes — then verify the diff matches the summary
9. Spend attention by risk — skim proven conventions, line-read safety semantics

Review is the half of agentic development that doesn't compress — and the half most teams under-invest in. If you're adopting AI coding agents and want the review discipline to match, we can help you build it.