AWS Cloud WAN series · Part 15 of 17

AWS Cloud WAN: Observability & Troubleshooting

Fifteenth in the AWS Cloud WAN series, and the SRE's chapter. Cloud WAN gives you a centralized, 15-month view of network health — but its telemetry stops at aggregate counters, with no per-flow log like Transit Gateway's. Here's what each signal actually tells you, where you have to reach for VPC flow logs to fill the gap, and a decision tree for the failures you'll really hit. Every claim is sourced to AWS docs.

What you get for free, and the one signal you don't

Cloud WAN ships a genuinely useful operational view out of the box, and it costs you nothing to turn on. The trap is assuming it covers the same ground as Transit Gateway. It mostly does — except for the one signal SREs reach for most under pressure. Four sources, three native and one you bring yourself (Figure 1):

CloudWatch metrics in the AWS/NetworkManager namespace are the health-and-capacity layer: bytes and packets in/out, and drop counts, published every 60 seconds at the attachment and Core Network Edge level, with 15 months of retention so you have real history for capacity planning. That retention is worth pausing on — it's long enough to see a quarterly trend, not just last week.

EventBridge events are the change-awareness layer: Cloud WAN streams near-real-time system events for changes to your network resources, so you wire a rule to attachment state changes and feed notifications, remediation, or a change log. Getting those events onto the Cloud WAN dashboards requires onboarding CloudWatch Logs first, which needs specific IAM permissions — the onboarding doc walks the setup.

The AWS Network Manager console is the reachability layer: a topology graph (a bubble per Region, nested per segment, then per VPC) and a view of all the routes Cloud WAN has generated, filterable by segment and edge location. This route view makes the abstraction concrete, and it's your first stop when traffic isn't reaching a destination.

What's missing is the fourth: per-flow detail. Cloud WAN has no native 5-tuple log. For "which IP talked to which IP on which port," you enable VPC flow logs yourself — the next two sections are about exactly where that line falls.

Why your metrics live in Oregon and never name a flow

Two facts about the metrics surprise people, and both change how you build dashboards and alarms.

The namespace is centralized in one Region. AWS/NetworkManager is available only in US West (Oregon), us-west-2 (GovCloud West for GovCloud), regardless of where your traffic actually flows. Don't go hunting for these metrics Region by Region — point every Cloud WAN dashboard and alarm at Oregon. This mirrors the home-Region rule from Article 1: Cloud WAN is a global service whose control and telemetry surface centrally.

There is no single "packets dropped" metric. Drops are split by reason, which is the most useful thing in the whole namespace once you know the names: BytesDropCountBlackhole and PacketDropCountBlackhole, BytesDropCountNoRoute and PacketDropCountNoRoute, and PacketDropTTLExpired — and only the Sum statistic is meaningful on them. The reason is the diagnosis: a rising ...NoRoute count means packets arrived with nowhere to go (a routing/segmentation gap), while ...Blackhole means they hit a blackhole route or an empty inspection group on purpose. You alarm on these separately because they point at different root causes.

The metrics carry four dimension sets — by core network (CoreNetwork, EdgeLocation), by attachment (Attachment, CoreNetwork), and crucially per-AZ for VPC attachments via (Attachment, AvailabilityZone, CoreNetwork). That per-AZ dimension is the one you'll lean on hardest: drops concentrated in a single zone are the classic fingerprint of an AZ impairment or an asymmetric-routing problem in an appliance-mode design (we get to that in the field guide). Separately, usage metrics map to your service quotas, so you can alarm as you approach the route or segment limits before they bite.

Where Cloud WAN goes blind — filling the gap with VPC flow logs

This is the difference that trips up engineers coming from Transit Gateway. Cloud WAN's standard metrics are attachment-level and edge-level — they carry no source/destination IP, port, or protocol. TGW flow logs give you per-flow 5-tuple records; Cloud WAN events and metrics tell you how much traffic moved and where it dropped, but never which specific flows were involved (Figure 2).

So when you need "which IP talked to which IP on which port" — the investigation TGW flow logs make trivial — you fill the gap with VPC flow logs on the VPCs connected to Cloud WAN. Enable them on the relevant VPCs, subnets, or ENIs, ship to CloudWatch Logs or S3, and query with CloudWatch Logs Insights (or Athena, or Grafana). Two techniques earn their keep: a metric filter on REJECT entries surfaces blocked traffic as a metric you can alarm on, and Contributor Insights ranks your top talkers or top rejected source IP/port pairs without writing a query each time. This is the standard way to recover TGW-flow-log-equivalent detail in a Cloud WAN world.

Budget for it in the design. The native telemetry is for health and capacity; VPC flow logs are for flow forensics — and they cost storage and query time, so enable them where you actually need per-flow visibility (DMZs, regulated VPCs, anything you'll have to forensically reconstruct) rather than everywhere by reflex.

Reading the routes the service writes for you

Cloud WAN generates a large number of routes automatically — even a modest proof-of-concept produces many per Region, segment, and attachment. The console route view lets you inspect them, and learning to read it is the core troubleshooting skill, because most reachability problems are visible right here before you touch a packet capture. Four things to look at:

• Propagated vs. static. Propagated routes are created automatically from your segment shares and the attachments that exist; static routes are the ones you defined explicitly in policy (a default route to an egress attachment, say). When a destination is unreachable, check whether the expected propagated route is present in the source Region's segment route table — its absence is the whole diagnosis.
• Filter by segment and edge location. The same segment has a route table per edge location, so multi-Region issues almost always reduce to "the route is in Region A's table but not Region B's." Filter to the Region that's actually failing.
• Next-hop redirection for inspection. With service insertion (Articles 8–10), routes show the local NFG attachment as next hop. If inspected traffic misbehaves, confirm the next hop is the inspection attachment you expect, in the Region you expect.
• The pre-routing-policy caveat. The list-core-network-routing-information API shows routing state before 2025.11 routing policies are applied (Article 2). If you run routing policies and the API output doesn't match observed behavior, that's why — the policy transforms it after this view.

When large transfers hang but pings fly — MTU and PMTUD

One failure mode deserves its own section because it leaves no metric behind and wastes hours. Cloud WAN supports an MTU of 8500 bytes for VPC-to-VPC traffic, including TGW peering and Tunnel-less Connect attachments — jumbo frames work between VPCs. It also does Path MTU Discovery for traffic ingressing on VPC attachments, generating the right ICMP messages (FRAG_NEEDED for IPv4, Packet Too Big for IPv6).

The gaps are where people get hurt. PMTUD is not supported on Connect, Site-to-Site VPN, Direct Connect, or peering attachments — only VPC attachments. And for the PMTUD that is supported to work at all, your security groups and NACLs have to allow the necessary ICMP. Silently-dropped ICMP is the classic cause of the "large transfers hang, small ones work" symptom: the big packet needs fragmentation, the ICMP that would say so gets filtered, and the sender just stalls. No Cloud WAN drop metric fires — it never reached the core to be counted — so this one you diagnose by checking SG/NACL ICMP rules, not the dashboard. If you run jumbo frames through an inspection VPC, verify the third-party appliances handle 8500-byte frames too.

A field guide to the failures you'll actually hit

Most Cloud WAN incidents are one of a handful of shapes, and each traces back to mechanics covered earlier in the series. Work them in the order of Figure 3 — attachment, route, symmetry, next hop — and only then open the flow logs.

No route — the implicit-deny drop. The same-segment-only, deny-by-default model (Article 3) means an unreachable destination is usually a missing route, not an explicit block — the packet is dropped because no route exists between the segments, and PacketDropCountNoRoute ticks up. Diagnose by checking the source Region's segment route table for the destination prefix. If it's absent, the cause is upstream: a missing share between segments, a non-transitive path you assumed was transitive (a shared-services segment bridging two others — it won't, by design), or an attachment that never reached Available. Fix the share or segment relationship, not the route.

Asymmetric routing breaking stateful inspection. The most common inspection failure, recurring through Articles 5–10. Return traffic hits a different appliance or AZ than the forward traffic, the appliance has no state for it, and it drops the flow — connections that establish but hang, or work only intermittently. Diagnose by confirming (a) appliance mode is enabled on the inspection VPC attachment, which pins a flow to one AZ for its lifetime, and (b) Region preference is consistent via edge-overrides or a routing policy. Per-AZ metrics showing drops concentrated in one zone are a strong signal.

Appliance-mode left off. A special case worth calling out because it's silent: appliance mode is independent of service insertion. Having service insertion configured does not turn appliance mode on — you need both, and it's set explicitly on each inspection VPC attachment. Forget it and stateful inspection fails asymmetrically exactly as above. Check it on every inspection attachment.

Empty NFG blackholing traffic. From Article 8: if a send-via or send-to targets a network function group with no attachment in a steered Region, that traffic is blackholed until you associate one — the policy deploys successfully, which is what makes it sneaky. Symptom: traffic to a particular Region's inspection path vanishes entirely, and ...Blackhole drops climb. Diagnose by confirming each steered Region has an inspection attachment in the NFG.

Wrong-Region egress / a random remote edge. From Article 7: when multiple edges advertise the same prefix with equal attributes, the forwarding edge can be chosen at random, and a Region with no local attachment may exit through an arbitrary remote Region. Symptom: unexpected latency or a cross-country detour. Diagnose with the route view — which edge is the next hop? — and fix with prefix specificity or a 2025.11 routing policy / edge-override to pin the preferred path.

Attachment stuck in Failed or Pending. From Article 4: an attachment has exactly eight lifecycle states, and Failed means an input error or a service-linked-role permission issue. If nothing flows because the attachment never came up, check the service-linked role and the attachment parameters first — there's no route to debug until the attachment is Available.

Where to look first

The monitoring baseline to stand up on day one

Pull it together into a default posture you put in place before you need it, not during the incident:

• Alarms on the drop metrics that diagnose themselves — ...DropCountNoRoute and ...DropCountBlackhole, overall and per-AZ — plus attachment throughput anomalies, and usage metrics approaching a quota. Remember they all live in Oregon.
• VPC flow logs on the connected VPCs where you need flow-level forensics, queried via Logs Insights, with a REJECT metric filter for blocked-traffic visibility.
• EventBridge rules on attachment state changes feeding notifications and a change log.
• One dashboard that puts Cloud WAN metrics and the relevant flow-log views side by side, so health and forensics aren't two different tabs at 2 a.m.
• The console route view as your reflex first stop for any reachability problem, filtered to the affected segment and Region.

That's the whole model: health and capacity from native telemetry, change awareness from events, reachability from the route view, and flow forensics from VPC flow logs you added on purpose. Know which question each one answers and you'll triage in minutes instead of guessing.

The last operational question is the one that decides whether Cloud WAN is worth running at all — cost. Article 16 breaks down the pricing dimensions (per-edge-hour, per-attachment, per-peering, data processing), works the examples AWS publishes, and gives you a framework for when the operator-time savings justify the spend.