The Edge Threat Pipeline: From Simulated Dashboard to Real Detections

Context

This site has a live threat map. For a while, it was a lie — a client-side script invented an attack every second on top of a static JSON file, with headline numbers to match. That is the industry default for these pages, and it is exactly the kind of thing we decided to stop doing.

Today the map renders real detections from intrusion detection running at the AS210622 edge. Every arc you see corresponds to an actual signature match against actual traffic. This case study is how the pipeline works, and the design decisions that a public security feed forces on you.

Constraints

• The page is public. Everything on it is available to the people probing us, so the feed must never become free reconnaissance.
• Router uplinks are metered and modest. The shipping agent runs on the edge routers themselves; the pipeline had to be light in both bandwidth and dependencies.
• One deploy, no new hosts. The collector had to live inside the existing site container, not become another service to operate.

Approach

The pipeline is deliberately short:

edge routers ── detection agent: tail alerts → enrich (ASN/country/city)
                     → batch + gzip → authenticated ship to the site
                                            │
                                      SQLite (node:sqlite)
                                            │
                          GET /api/threats → live map (polls every 5s)

A small Go agent on each router tails the IDS's event log, enriches each alert with source ASN, country, and city, then ships batches gzipped — alert JSON is repetitive and compresses roughly 10x, which matters on a metered uplink. The write path is authenticated and fail-closed: if its credentials are unconfigured, it rejects everyone rather than accepting anonymous writes. Events land in SQLite via Node's built-in driver (chosen because the Docker build runs npm ci --ignore-scripts and cannot compile native modules), on a named volume so history survives redeploys. Attack types are resolved at write time and indexed, so the read path serves 24-hour aggregates from covering indexes instead of classifying on every request.

Three decisions define the public feed:

• Destinations are sites, never IPs. The feed says "Pitesti (Port 22 — SSH)", not which customer host was probed. The event type deliberately has no destination-IP field, so the mistake is unrepresentable.
• The engine goes unnamed. Public copy says "detections from the AS210622 edge" — naming the IDS tells an attacker exactly what to evade, so signature text is scrubbed before display and the page never counts sensors.
• "Blocked" reads zero, because it is zero. The previous copy claimed mitigation that wasn't happening. The stat becomes non-zero when detections drive real firewall blocks — not before.

What broke

The classifier filed reputation-list hits as exploits. A signature ending in "…Block Listed Source" substring-matched a bare rce pattern and doubled the exploit count. The fix — reputation rules match first, short acronyms anchored with \b — is now pinned by a Vitest suite of 38 cases using real signature names, verified to fail if either property is reintroduced. The classification rules are mirrored in the Go agent, so both sides change together or not at all.

Inline blocking is quietly expensive. A filter that drops a share of legitimate traffic is an outage you inflict on yourself, so anything that blocks at the edge earns its place through evidence first. Working through that evidence is the same alert-fatigue problem that led us to build meerkat, our open-source console that rolls thousands of alerts into per-source decisions.

Secrets kept trying to enter the build. The mail integration originally constructed its client at module scope, making its API key a build-time requirement — which means a Docker build arg, which means a secret readable in image history. It now initializes on first request; the production build needs no environment at all.

Where it stands

The pipeline is in production: real events, fail-closed ingest, tested classification, 24-hour retention with indexed aggregates, and data that survives container recreation. The honest gaps are documented too — blocking is not yet wired to the firewall, and a multi-replica deployment would need the process-local cache rethought.

If your security dashboard is showing you noise — or fiction — we can help you build one that tells the truth.