Practical BGP Traffic Engineering: Local-Pref, Communities, and Prepending

AS210622 is multi-homed across three transit providers: Digi Romania (AS8708) at 10G as our primary, NAV Communications (AS6718) and ParadoxNetworks (AS52025) at 1G each. Three independent upstreams means no single provider can take us offline — but it also means we have to decide, for every prefix on the internet, which door the traffic leaves through, and try to influence which door it comes back in.

That second part is where BGP traffic engineering gets its reputation. Here is what we've learned actually moves traffic, with a small network's honest perspective: three transits give you a few real levers, not a control panel.

Outbound Is Easy: Local-Pref Decides

For traffic leaving your network, you are the decision-maker. BGP best-path selection considers local preference before almost everything else — before AS-path length, before MED, before anything your providers signal. Set it and traffic obeys.

Our policy is straightforward: the 10G primary gets the highest local-pref, the two 1G links sit below it as hot standbys that also carry traffic for the destinations where their path is genuinely better.

In BIRD terms, the shape of it:

filter transit_in_digi {
  bgp_local_pref = 200;
  bgp_large_community.add((210622, 1, 1));   # tag: learned from transit
  accept;
}

filter transit_in_backup {
  bgp_local_pref = 100;
  bgp_large_community.add((210622, 1, 1));
  accept;
}

With equal local-pref, path selection falls through to AS-path length and you get a rough, latency-agnostic load spread. With tiered local-pref, you get predictability. We chose predictability: when we're debugging a reachability issue at 2 AM, "it left via Digi unless Digi doesn't have the route" is a sentence we can reason about.

The tags matter as much as the preferences — more on that below.

Inbound Is a Negotiation, Not a Setting

Traffic entering your network follows the best-path decisions of thousands of other networks, and you don't get a vote in their routers. Everything you can do inbound is a suggestion. Roughly in order of effectiveness:

Provider Action Communities: The Sharpest Tool

Most transit providers publish a set of BGP communities you can attach to your announcements to change how they handle them: prepend on their side toward specific peers, lower the local-pref inside their network, or suppress the announcement toward a particular exchange or region entirely. This is the closest thing inbound TE has to a real control, because it changes behaviour inside the network that's actually carrying the traffic — upstream of where path length even gets compared.

Every provider's community set is different and documented (with varying enthusiasm) in their IRR record or customer portal. Read your providers' lists before reaching for any of the blunter tools below. The community you need probably already exists.

AS-Path Prepending: Diminishing Returns

Prepending — announcing 210622 210622 210622 instead of 210622 — makes a path look longer so other networks prefer the alternative. It works, but much less than people expect, because AS-path length is compared after local-pref in everyone else's routers. A network that prefers your prepended path because of its own commercial relationships (customer routes over peer routes over transit routes, always) will keep preferring it no matter how many prepends you stack.

In practice the first prepend does most of whatever moving is going to happen, the second adds a little, and beyond three you are decorating. If four prepends haven't shifted the traffic, forty won't.

MED: Mostly Decorative Across Providers

Multi-Exit Discriminator only gets compared, by default, between routes learned from the same neighbouring AS. Since our three upstreams are three different ASes, MED does approximately nothing to arbitrate between them from a remote network's point of view. Useful when you have two sessions to the same provider in different cities; otherwise, not the lever you want.

Large Communities: Tag Everything on Ingress

RFC 8092 large communities are the bookkeeping that makes all of the above manageable. Every route we accept gets tagged at ingress with where it came from — transit, and eventually customer or IX when we get there. Export policy is then written against the tags, not against prefix lists that rot:

# Export policy in one sentence:
# announce only what is tagged "ours" or "customer" — never transit.

This is how route leaks are prevented structurally instead of by vigilance. A route tagged "learned from transit" cannot be exported to another transit, because no export filter matches that tag. The classic full-table leak becomes unrepresentable rather than merely discouraged.

RTBH: The Blunt Instrument

Remotely-triggered blackholing is technically traffic engineering — it engineers specific traffic into the void. Announce a /32 with your provider's blackhole community and they drop the traffic at their edge, before it traverses your transit link. It's the tool of last resort when a DDoS exceeds what you can absorb, and we covered where it fits in the layered picture in the DDoS mitigation post. The cost is honest: you complete the attack against one address to keep everything else up.

Measure, Don't Assume

Every TE change is a hypothesis, and the internet is under no obligation to confirm it. After each change we check:

• Looking glasses — RIPE RIS, route-views, and the looking glasses of networks we care about, to see which path they actually selected.
• bgp.tools — how our announcements propagate, per upstream. (Everything we claim here is visible there; check any operator's claims, including ours.)
• Interface counters over days, not minutes — inbound shifts arrive slowly as caches, sessions, and DNS TTLs roll over.

More than once a prepend we were sure about moved almost nothing, because the networks sourcing most of our inbound traffic had local-pref policies that never consulted path length. The looking glass told us in thirty seconds what the traffic graph would have taken a day to reveal.

Lessons

1. Outbound TE is a setting; inbound TE is a negotiation — budget your effort accordingly.
2. Read your providers' community documentation first — action communities are the sharpest inbound tool and they already exist.
3. The first prepend does most of the work — beyond three, you're decorating.
4. Tag routes at ingress with large communities — export policy written against tags makes route leaks unrepresentable.
5. Verify with looking glasses, not intuition — remote best-path decisions routinely ignore your suggestion.

If you're multi-homing a network and want the policy design done right the first time, that's a thing we do — or come talk peering: peering@kicked.ro.