Shaping the SD-WAN Fabric on AWS, Part A: The Regional Hub and Keeping Traffic Symmetric

One regional-hub design — where the SD-WAN appliance pair attaches, how it survives losing a node without turning into a single-flow bottleneck, and how every return packet finds its way back to the same firewall. Built once on Transit Gateway and once on Cloud WAN, so you can see the same design in two realizations. Only the resource names change.

1The conversation that only works one way

Your SD-WAN vendor's appliance is up. BGP is established, the fabric shows the peer green, a test ping crosses cleanly — and then a real flow, a database sync from a branch office, hangs halfway through. Nothing is “down”: no tunnel flapped, no route withdrew. The fabric is simply shaped so the forward and return halves of the same conversation landed on two different appliances, and a stateful firewall has no session state for a reply it never sent.

That is the failure this article exists to design out. The regional hub — where the SD-WAN appliance pair attaches to AWS — has three properties that hold together or fall apart as a set: it survives losing an appliance, it spreads load across the pair without secretly capping every connection, and it keeps both halves of every flow glued to the same inspection node. Get one wrong and you never get a clean outage to point at — you get a ping that works, a big transfer that hangs, and a firewall log that makes no sense.

The central move of the whole series is that this hub is one design with two realizations. Built on Transit Gateway or on Cloud WAN, the picture is the same both times: an appliance pair, a Connect attachment carrying GRE and BGP, ECMP by routing, and a symmetry flag on the return path. What changes is the resource names — aws_ec2_transit_gateway_connect versus aws_networkmanager_connect_attachment — not the shape. So this is deliberately not a “which platform should I pick” bake-off; that decision (multi-Region segmentation, cost, DX support) is Part B. Part A teaches the hub itself, once.

The verdict box above is the whole design in six lines. Everything below earns those claims, in order: the map, then the naive pair and exactly why one flow stalls, then the pattern that fixes it, the GWLB variant, the bandwidth math, and the places it bites.

2Where the SD-WAN attaches — the regional hub, both realizations

Start with the map. Traffic from an on-prem branch enters AWS, reaches a pair of SD-WAN appliances (network virtual appliances, or NVAs) in a security hub VPC, is inspected, and continues to the protected workload. Each appliance forms a Connect peer: a GRE tunnel carrying a BGP session, riding on a transport attachment [f8] — GRE gives the overlay, BGP gives the fabric its routes, the transport underneath is something you already trust. On Transit Gateway that transport is a VPC attachment or a Direct Connect gateway attachment; on Cloud WAN it is a VPC attachment only [f32].

Read the figure box by box, because these names recur in every later diagram. On the far left is the branch, outside AWS; its traffic lands on a Connect peer and enters the fabric — the Transit Gateway on the left half, the Cloud WAN Core Network Edge on the right — which hands the packet to an appliance in the hub VPC for inspection, then on to the workload VPC. The one visual difference across the seam is the fabric box itself; everything else — branch, appliance pair, Connect peers, flow — is shared, which is the entire point.

One asymmetry shows up already, and it is a planning constraint, not a design difference: Cloud WAN allows exactly one CNE per Region per core network, a hard limit [f11]. Transit Gateway has no equivalent cap — you can run several TGWs in a Region. That single CNE isn't a bottleneck for the hub (a CNE fronts many attachments), but it is the shared regional object every attachment in that Region hangs off, so you plan it deliberately rather than spinning up one per team. Hold that thought; it returns in §7.

3The naive HA pair, and exactly why one flow stalls

The obvious way to build the pair is to drop two appliances in the hub VPC, attach them, and assume “two is redundant.” It usually isn't — not because two appliances is wrong, but because redundancy is a routing property you have to state out loud, and the naive build states it wrong in one of two ways.

Failure (a) — the accidental active/standby. Suppose appliance B advertises A's prefixes with a longer AS-PATH, or you simply never made them equal-cost. BGP does what you told it: it prefers A's shorter path and installs only A, leaving B healthy, advertising, and completely unused. Every flow funnels through A's single Connect peer, which tops out at 5 Gbps [f5]. You bought two appliances and got one, at a hard 5 Gbps ceiling, with a “spare” that only wakes up when A dies. Sometimes that's the design you want; when it happens by accident, it's a capacity incident waiting for a busy afternoon.

Failure (b) — the asymmetric return. Now fix the routing so both appliances share load, but leave the return path to chance. A request enters through one AZ and gets inspected at appliance A; the response comes back from the workload through a different AZ, and the fabric — with nothing telling it otherwise — hands that return packet to appliance B. B is a stateful firewall with no session for this conversation; it never saw the SYN, so it drops the reply [f29]. The forward half worked, the connection looks established, then it hangs. This is the exact symptom from §1, and it is silent: no route down, no metric spike — the firewall is simply doing its job on a packet that, from its point of view, arrived out of nowhere.

The two failures have one root cause between them: the naive pair leaves which appliance to chance on both the spread and the return. The pattern in §4 removes the chance from both — spread by routing, symmetry by flag.

4The pattern — ECMP by routing, symmetry by flag

The hub design is two decisions you make on purpose. First, make the pair equal-cost so the fabric spreads flows across it. Second, pin each flow's return to the same appliance it entered. Neither is a product feature you buy; both are properties you configure.

ECMP by routing

Active/active means each appliance forms its own Connect peer and advertises the same prefixes with the same AS-PATH and a matching ASN. That exact combination — same prefix, same AS-PATH length, matching ASN — is what the fabric recognizes as equal-cost multipath and installs as multiple paths [f15]. Miss any one and you're back in failure (a): a different AS-PATH breaks the tie, and a mismatched ASN prevents multipath entirely (the fabric won't ECMP across different ASNs). Active/standby, when you want it, is the same picture with the standby advertising the same prefix at a longer AS-PATH (or a worse MED — the default inbound MED on Connect is 100) [f26], so it only wins once the active withdraws.

Here is the sharp edge: ECMP spreads flows, not packets. The fabric hashes each 5-tuple to exactly one path and keeps that flow there for its life. Four Connect peers give ~20 Gbps of aggregate capacity across many flows [f6], but any single connection still pins to one peer at ≤5 Gbps [f5]. A 12 Gbps backup job over one TCP connection gets 5, not 12, no matter how many peers you add. You scale by flow count, not by widening a pipe — §6 puts numbers on this.

Symmetry by flag

Equal-cost spread is what creates the asymmetry risk — now that flows can land on either appliance, the return must be told which one. On Transit Gateway that instruction is a single per-VPC-attachment flag, appliance_mode_support = "enable": with it on, the gateway picks one appliance ENI by flow hash and reuses that same ENI for the return, giving bidirectional symmetry through one appliance [f28]; with it off, you get failure (b). On Cloud WAN the equivalent is service insertion through a Network Function Group, routed with the segment's send-via action — and it requires appliance mode too [f24]. Same job, different resource.

Illustrative — resource names only (full Terraform is Article 3)

# TGW realization — appliance mode on the security VPC attachment
resource "aws_ec2_transit_gateway_connect" hub { /* on a VPC/DX transport */ }
resource "aws_ec2_transit_gateway_connect_peer" a { /* one per appliance */ }
resource "aws_ec2_transit_gateway_vpc_attachment" sec {
  appliance_mode_support = "enable"   # the symmetry flag
}

# Cloud WAN realization — same shape, NFG for inspection
resource "aws_networkmanager_connect_attachment" hub { /* VPC transport */ }
resource "aws_networkmanager_connect_peer" a { /* one per appliance */ }
# service insertion via a Network Function Group in the core-network policy

The names differ; the design is one thing. The complete, working Terraform for both — inside CIDRs, ASNs, route-table wiring — is Article 3; here the point is only that the two realizations map hop-for-hop onto the same skeleton.

5The GWLB alternative — same goal, different defaults

Connect peers with ECMP are one way to build the pair. Gateway Load Balancer is the other, reaching the same goal — spread across the appliances, symmetric flows — through a completely different mechanism with defaults you must set on purpose. Instead of each appliance forming its own BGP peer, the appliances become targets behind a GWLB, and traffic reaches them through a routable GWLB endpoint you point routes at.

GWLB carries traffic to the appliances inside GENEVE encapsulation on port 6081 [f30], so the appliance sees the original packet intact for inspection. It distributes flows by 5-tuple stickiness — and if TGW appliance mode is on in front of it, 2-tuple and 3-tuple stickiness are no longer supported, so you must use 5-tuple [f30]. So far, much like ECMP. The differences are in the defaults, and each one is a decision:

The two that surprise people: GWLB fails open. If every target goes unhealthy it does not blackhole — it passes traffic uninspected [f27], a sane availability default for a load balancer but a compliance event for a firewall. And by default, when a single target dies, the flows already pinned to it stay on the dead target until you set target_failover.on_unhealthy = rebalance [f27]. Reach for GWLB when you want appliance health checks and lifecycle handled by ELB rather than BGP, or when your vendor's reference architecture is built around it; reach for Connect peers when you want the fabric's own routing to be the single source of truth for reachability.

6What the hub actually gives you — bandwidth, flow count, convergence

Now the numbers — because “how much bandwidth does the hub have” is the question people get wrong most often. The wrong answer adds up the aggregate; the right one asks how a single flow moves, because that's the ceiling a real backup job or database sync actually hits.

Read the table down the “single flow” column and the design rule falls out: size for flow count, not total. The aggregate figures are real, but they only materialize with many concurrent flows to spread. One fat connection is capped by whichever single path it hashes onto — ≤5 Gbps over a Connect peer [f5], 1.25 Gbps over a standard VPN tunnel [f1]. On the frozen VPN phrasing: it is always standard tunnel 1.25 Gbps / large-bandwidth tier 5 Gbps, and the 5 Gbps large-bandwidth tier is available on TGW and Cloud WAN attachments only, not on a VGW [f1][f2] — never a bare “5 Gbps per tunnel.”

Convergence is the other operational number, modest by design, because there is no BFD on Connect. When an appliance dies, its BGP session goes quiet and the route withdraws only after the hold timer expires — the default Connect timers are keepalive 10s / hold 30s [f49], so worst-case failure detection is about 30 seconds before the fabric rehashes surviving flows onto the remaining peers. Plan around that shape: a lost appliance is a ~30-second reconvergence, not an instant failover. The deep operate-and-debug treatment — flow logs, drop reasons, the full convergence walk — lives in Article 4.

7Where it bites — the hub choices that look right but aren't

Every item here is a design that passes review, deploys clean, and then fails in a way that doesn't look like a networking problem. Call it the negative knowledge — the things you'd much rather learn before you hit them.

• The default-ASN collision (TGW ↔ CNE). A Transit Gateway ships with default ASN 64512 [f13], and Cloud WAN CNEs use ASN ranges that include 64512. Peer a default-ASN TGW to a CNE in that range and the peering simply won't form — two devices can't share an ASN across an eBGP boundary. Set the ASNs deliberately; don't accept the default and expect them to peer.
• Assuming a Connect peer's two BGP sessions give ECMP. Each Connect peer's /29 inside CIDR yields one appliance BGP address and two fabric BGP addresses — two BGP sessions [f12], which are AWS-side control-plane redundancy. The fabric cannot ECMP across the two sessions of a single peer; ECMP happens across peers. Two sessions is not two pipes.
• Assuming per-flow ECMP gives single-flow throughput. The most common capacity miss: someone sizes a 20 Gbps aggregate and promises a 15 Gbps transfer. One flow pins to one path at ≤5 Gbps. ECMP scales flow count; it never widens a single connection.
• Appliance mode off → silent asymmetric drop. Forget appliance_mode_support (or the NFG's appliance-mode requirement) and the return half of a flow can land on the appliance that never saw the request; the stateful firewall drops it [f29]. Ping works, transfers hang. It is the single most common hub bug.
• GWLB fail-open and stuck flows. If every GWLB target goes unhealthy, GWLB fails open and passes traffic uninspected — a security surprise, not an outage [f27]. And a single dead target keeps its existing flows until you set target_failover.on_unhealthy = rebalance. Both are defaults you have to change on purpose.
• One CNE per Region as a planning constraint. Cloud WAN gives you exactly one CNE per Region per core network, and it's hard [f11] — not a throughput limit but a shared-object limit. Every attachment in that Region hangs off that one CNE, so treat it as the regional anchor you design around, not something you scale horizontally.

8References

All facts verified 2026-07-17 against current AWS documentation. Figures cite the series fact ledger (fact IDs in brackets throughout).

• AWS Transit Gateway Connect documentation — GRE + BGP unencrypted overlay on a trusted underlay; 5 Gbps per Connect peer, up to 4 peers (~20 Gbps aggregate); two BGP sessions per peer; /29 inside CIDR from 169.254.0.0/16; default timers keepalive 10s / hold 30s, no BFD. [f5][f6][f8][f12][f33][f49]
• AWS Transit Gateway quotas — Connect peer bandwidth and peers-per-attachment; VPC attachment 100 Gbps per AZ per direction. [f5][f6][f54]
• How Transit Gateways work — ECMP requires same prefix + same AS-PATH + matching ASN; active/standby via longer AS-PATH or worse MED (default inbound 100); appliance mode pins a flow to one ENI and reuses it for the return. [f15][f26][f28][f29]
• AWS Transit Gateway VPN attachments — TGW default ASN 64512 (customizable); collision risk against Cloud WAN CNE ASN ranges. [f13]
• AWS Cloud WAN quotas — one Core Network Edge per Region per core network (hard); Connect GRE 5 Gbps per peer / ~20 Gbps aggregate. [f9][f11]
• AWS Cloud WAN service insertion — inspection through a Network Function Group requires isolated mode and appliance mode; send-via for east-west. [f24]
• AWS Site-to-Site VPN limits — standard tunnel 1.25 Gbps; large-bandwidth tunnel tier 5 Gbps on TGW & Cloud WAN attachments only (not VGW). [f1][f2]
• AWS Transit Gateway Connect transport & BGP — Connect transport is a VPC or DX-gateway attachment (VPN is not valid); Cloud WAN Connect transport is VPC only; ASN/eBGP behavior. [f32][f34]
• AWS Gateway Load Balancer target-group attributes — 5-tuple stickiness (2/3-tuple unsupported with appliance mode); GENEVE port 6081; AZ affinity on / cross-zone off; fail-open; target_failover.on_unhealthy = rebalance. [f27][f30]
• AWS TGW ECMP support by attachment type — Connect automatic; cannot ECMP across different ASNs or attachment types. [f55]
• Example private ASN convention used in figures. [f14 · f16 (DX port speeds, context)]