Three Production-Grade Tools in Four Months: An Agentic Development Case Study

Most writing about AI-driven development argues from benchmarks or from vibes. We would rather argue from artifacts. Between roughly February and May 2026, our small team designed, built, and shipped three open-source infrastructure tools, with AI coding agents doing the large majority of the typing. All three are public, so every claim in this post can be checked against the repos.

This is the honest account: what the agents did, what they couldn't, and what it cost.

The three tools

nftably — a web UI for nftables that manages the firewall as its real object model: tables → chains → rules, every option a typed, explained control. Changes apply as one atomic transaction with an armed auto-revert, so a bad rule can't lock you out of the box you're editing. Announcement post.

birdy — a web UI for the BIRD routing daemon that runs on the router itself. It models peers, policies, RPKI, RTBH, and BFD in a database, renders the entire bird.conf, and applies it with the same armed auto-revert — plus a live dashboard of what every BGP session is actually doing. Announcement post.

meerkat — a console for Suricata whose home page is a list of source addresses, not a flood of alerts. It enriches every alert with ASN and geography, rolls the noise up per source, and turns each source into a decision: block, acknowledge, allowlist, or ignore. Announcement post.

Each is a single Go binary backed by SQLite, with an embedded web UI, systemd and Docker packaging, CI, docs, and a changelog. Each started from a human-written PLAN.md with milestones gated by one rule: each milestone is independently usable and cannot break anything the previous one didn't already touch. The full workflow is documented in our AI-driven development post.

What the agents were genuinely good at

The pattern across all three projects was remarkably consistent. Agents excelled at work that is voluminous, conventional, and checkable:

• CRUD and UI plumbing. Handlers, forms, list views, filters, pagination — the bulk of any management UI by line count. This is where most of the typing went, and the agents were fast and reliable at it.
• Renderers. Both nftably and birdy turn a database model into a complete config file (nft rulesets, bird.conf). Renderers are ideal agent work: pure functions with a precise spec and easily written golden tests.
• Test scaffolding. Table-driven Go tests, fixtures, fake control-socket servers for BIRD's line protocol. The agents kept coverage moving with the code instead of trailing it.
• Packaging and release engineering. Dockerfiles, docker-compose files, nfpm configs for deb/rpm, systemd units, CI workflows. Fiddly, well-documented-on-the-internet, and exactly the kind of thing agents get right on the first pass.
• Keeping the paperwork in sync. CHANGELOGs, usage docs, README updates riding along in the same diff as the feature. Agents don't find documentation beneath them, which it turns out is a genuine competitive advantage over most engineers, ourselves included.

What stayed human

Everything that could hurt someone, in the broad sense:

• The object models. What a "peer" is, what a "policy chain" is, what a "source" is in meerkat's rollup — these decisions shape everything downstream, and they were made by humans on paper before generation started.
• The safety semantics. The armed auto-revert in nftably and birdy exists because a human has been locked out of a remote box by their own firewall rule. The config-ownership rule — each tool renders and owns the entire file, and says so loudly in the README — is a human decision about honesty over convenience. Birdy will refuse to render a config it thinks is a route leak; an agent would never volunteer that constraint, because the training data is full of configs that leak.
• Deciding what not to build. Agents never push back on scope; ask and they build. Every "no" in these projects — no fleet management, no multi-router writes, not every BIRD knob exposed — came from a human.
• Review. Every diff was read. The review discipline that makes this workable is its own topic — see our field guide to reviewing AI-written code.

What surprised us

Plans diverging from what shipped is normal — and fine. Birdy's PLAN.md now opens with a note admitting exactly how reality diverged: Prometheus metrics and PeeringDB lookups shipped although the plan scoped metrics out; alerting grew far past "one webhook, no SMTP" into Slack, Discord, email, and filtered destinations; several tables in the planned schema were never built, and shipped features — BMP, IRR expansion, RTBH, BFD, graceful restart — appear nowhere in the plan. We keep the doc anyway, divergence note and all. The plan is a record of design intent, not a contract, and the delta between plan and product is where the actual engineering happened.

The bottleneck moved, permanently. With typing nearly free, our scarce resources became design time and review attention. Days were shaped less like "write the feature" and more like "specify the milestone precisely, then read diffs carefully." That trade is worth it, but it is a trade, not a free lunch.

Momentum is a risk of its own. When implementation is this cheap, the temptation is to build everything. The milestone gating and the non-goals list were what kept three tools small and finished instead of one tool sprawling and abandoned.

The honest costs

• Review time is real. Most of the recovered typing time was reinvested in reading diffs. Skipping that step is how you ship an agent's plausible-but-wrong code to a firewall.
• Beta quality is real. All three READMEs say "expect bugs," and we mean it. These tools have not had the years of hardening that routing and firewall infrastructure deserves — agentic development compressed the build, not the maturing.
• The judgment doesn't compress. The tools took months instead of — realistically — a year-plus for a team our size. What didn't get faster was knowing what to build and when it's wrong. That is still the job.

Takeaways

1. Argue from artifacts — three public repos are worth more than any benchmark; check our claims against the code
2. Agents own the volume — CRUD, renderers, tests, packaging, docs; humans own models, safety, and scope
3. Gate milestones by trust — each independently usable, none able to break the last; it's what let beta software touch real routers
4. Expect the plan to diverge — keep it anyway, and record the delta honestly
5. The savings get reinvested — typing time becomes design and review time; budget for that or don't start

If you want this workflow — the guardrails, the milestone discipline, the review practice — established in your own team, we can help.