When pods can't reach RDS across a Transit Gateway: the custom-networking asymmetric-routing trap

Custom networking did exactly what it promised: your pods moved onto a fat 100.64.0.0/16 secondary CIDR, the routable corporate IPv4 stopped bleeding, and the FailedCreatePodSandBox IP errors disappeared. Everything green. Pod-to-pod works. A pod curling the internet through the NAT gateway works. You ship it.

Then a Deployment that talks to an Amazon RDS instance — or an internal service, a Redis cluster, anything sitting in another VPC across a Transit Gateway — starts timing out. Not a clean refused connection: a silent hang that ends in dial tcp 10.1.20.14:5432: i/o timeout. You check the route tables. The pod subnet has a route to the TGW. The TGW has a route to the database VPC. Security groups allow 5432. Everything looks correct, and yet the packets vanish. tcpdump on the database side shows the SYN arriving and the SYN-ACK leaving — but the pod never sees the reply.

Welcome to the asymmetric-routing trap. The request and the response are taking different paths, and somewhere on the return leg a stateful device or the Linux kernel itself is silently dropping a packet it considers impossible. This is what that means, and how to fix it.

Why custom networking splits your node into two networks

Custom networking exists to stop pods from consuming routable corporate IPv4. It does that by parking pod IPs on a secondary VPC CIDR — the CG-NAT range 100.64.0.0/10 is the canonical choice because it rarely collides with on-prem RFC 1918 space. An ENIConfig per Availability Zone tells the CNI which secondary-CIDR subnet and security groups the pods' ENIs should use. That much you already have.

The consequence people underestimate is topological: a custom-networking node is now multi-homed. It has two categories of interface living on two different networks, and the kernel has to be told, per packet, which one to use.

• eth0, the primary ENI — sits on the primary, routable VPC CIDR (say 10.0.0.0/16). It carries the node's own identity: kubelet, the aws-node and kube-proxy host-network pods, and — critically — it is where off-VPC pod traffic gets source-NAT'd by default. With custom networking on, eth0 no longer hands its secondary IPs to pods.
• eth1, eth2, … the secondary ENIs — sit on the 100.64.0.0/16 secondary CIDR defined by your ENIConfig. Every application pod gets its IP from one of these.

Because pod traffic and node traffic live on different subnets, a single default route can't serve both. The CNI installs policy routing: a set of ip rule entries that pick a routing table based on the packet's source address, plus one dedicated route table per ENI. Run ip rule show on a custom-networking node and you'll see something close to this:

$ ip rule show
0:      from all lookup local
512:    from all to 100.64.12.34 lookup main
1025:   not from all to 10.0.0.0/16 lookup main
1536:   from 100.64.12.34 lookup 2
1536:   from 100.64.12.51 lookup 2
32766:  from all lookup main
32767:  from all lookup default

Read the three rules that matter, because the whole failure hides in them:

• Rule 512 — 512: from all to <pod-ip> lookup main: inbound traffic destined for a pod uses the main table, so replies find the pod.
• Rule 1025 — not from all to 10.0.0.0/16 lookup main: this is the "off-VPC" rule. Any pod traffic not headed to a VPC-local CIDR is pushed to the main table, whose default route points out eth0. This rule only exists when SNAT is enabled (the default).
• Rule 1536 — 1536: from <pod-ip> lookup 2: traffic originating from a pod uses that ENI's dedicated table (here table 2), whose default route points out the secondary ENI the pod is attached to.

Inspect an ENI table directly and you can see the interface it pins traffic to:

$ ip route show table 2
default via 100.64.0.1 dev eth1
100.64.0.0/24 dev eth1 proto kernel scope link src 100.64.12.34

So the node has two truths at once: node traffic and SNAT'd pod traffic leave via eth0 on 10.0; un-SNAT'd pod traffic leaves via a secondary ENI on 100.64. Which one a given packet takes depends entirely on whether SNAT is in play — and that single switch is the difference between a boring, symmetric path and a silent drop.

Where the packet actually dies: the asymmetric break

Here's the setup that triggers it. Somewhere along the way you set AWS_VPC_K8S_CNI_EXTERNALSNAT=true — maybe because the database's security group filters by source IP and needs to see the real pod, maybe because an on-prem system has to initiate connections into your pods, maybe because a tutorial told you to when your nodes moved to private subnets. Whatever the reason, that flag removes the SNAT iptables rule and the off-VPC rule (1025). Now a pod's traffic to another VPC keeps its 100.64.12.34 source address and leaves via the secondary ENI. Trace the packet:

1. Outbound. Pod 100.64.12.34 opens a connection to RDS at 10.1.20.14. Rule 1536 sends it to the ENI table; it egresses eth1 with its real source IP. The pod subnet's route table points 10.1.0.0/16 at the Transit Gateway. Good so far.
2. Across the TGW. The SYN reaches the TGW, which forwards it to the database VPC. tcpdump on the RDS host confirms the SYN arrives from 100.64.12.34 and a SYN-ACK is sent back toward it.
3. The return leg forks. For the reply to get home, the database VPC needs a route for 100.64.0.0/16 pointing at the TGW, and the TGW's route table needs 100.64.0.0/16 pointing back at your EKS VPC attachment. Miss either and the SYN-ACK is dropped at the cloud layer with no log you'll easily find. But say those routes exist — the reply still has to re-enter the node, and this is where two more traps wait.

Trap A — Transit Gateway AZ affinity

By default a Transit Gateway keeps Availability Zone affinity: it forwards traffic out of the same AZ it came in on. When source and destination are in different AZs across different VPCs, the request and the reply can traverse the TGW through different AZ ENIs. For plain routing that's harmless. But if there's a stateful device in the path — a firewall appliance, a NAT instance, an inspection VPC — it sees the SYN on one AZ path and the SYN-ACK on another, has no session state for the second, and drops it. AWS's own guidance is blunt: this "can cause traffic to be dropped," and the fix is to turn on appliance mode on the appliance VPC's attachment, which pins a flow to a single ENI for the life of the connection.

Trap B — Linux reverse-path filtering (rp_filter)

Even with clean cloud routing, the reply has to satisfy the node's kernel. Linux implements reverse-path filtering (RFC 3704). When a packet arrives, the kernel asks: if I had to send a packet back to this source address, would it go out the interface this packet just came in on? The behavior is controlled by net.ipv4.conf.<iface>.rp_filter:

In strict mode, a reply from RDS that arrives on the "wrong" interface — because AZ affinity or a return route steered it there, or because the pod's own source-based rules make the kernel expect it on a different ENI — fails the check and is discarded before it ever reaches the socket. It's silent by default; the only trace is a kernel log line if log_martians is on:

$ dmesg | grep -i martian
IPv4: martian source 100.64.12.34 from 10.1.20.14, on dev eth0

Read that line carefully: a packet from the database (10.1.20.14) carrying a reply for the pod (100.64.12.34) showed up on eth0 — but the kernel's route back to 100.64.12.34 is out a secondary ENI, not eth0. Strict rp_filter calls that a "martian" (an impossible source) and drops it. Your SYN-ACK died inside the kernel.

Confirming it on the node in five minutes

Before you touch a single route table, prove the diagnosis. Exec onto the node (or a privileged debug pod with hostNetwork: true) and work the layers from the inside out.

1. Is the pod even using SNAT? Check whether the off-VPC rule and the SNAT iptables entry exist. If rule 1025 is present and you see a SNAT POSTROUTING entry, pod traffic is being masked to eth0 and asymmetry is unlikely. If they're gone, EXTERNALSNAT=true is active and the pod IP is on the wire.

# the off-VPC "lookup main" rule only exists while SNAT is on
ip rule show | grep 1025

# the CNI's SNAT masquerade rule (empty output == external SNAT)
sudo iptables -t nat -S | grep 'AWS-SNAT'
# -A AWS-SNAT-CHAIN-0 ! -d 10.0.0.0/16 ... -j SNAT --to-source 10.0.4.20

2. Which interface does the kernel expect the reply on? Ask the routing table what path it would use back to the pod, then compare it to where replies actually land.

# the kernel's reverse path for the pod's source IP
ip route get 100.64.12.34
# 100.64.12.34 dev eth1 ...   <-- kernel expects replies on eth1

# what rp_filter is actually enforcing, per interface
sysctl net.ipv4.conf.all.rp_filter net.ipv4.conf.eth0.rp_filter \
       net.ipv4.conf.eth1.rp_filter
# net.ipv4.conf.all.rp_filter = 1     <-- strict floor, the smoking gun
# net.ipv4.conf.eth0.rp_filter = 2
# net.ipv4.conf.eth1.rp_filter = 2

3. Catch the drop in the act. Turn on martian logging and watch, then capture on the inbound interface to see the SYN-ACK arrive and go nowhere.

# log packets the kernel rejects as impossible sources
sudo sysctl -w net.ipv4.conf.all.log_martians=1
sudo dmesg -w | grep -i martian
# IPv4: martian source 100.64.12.34 from 10.1.20.14, on dev eth0

# confirm the reply reaches the node but never the pod
sudo tcpdump -ni eth0 'host 10.1.20.14 and port 5432'
# 10.1.20.14.5432 > 100.64.12.34.: Flags [S.]  <-- arrives, then dropped

A martian source line naming your pod IP and your database IP is the confirmation. Now you can fix it with intent instead of guessing.

The three levers that fix it — pick by requirement

There is no single "correct" fix; there's a decision. The question that picks the lever is: does the destination need to see the pod's real 100.64 address? If not, mask it and you're done. If yes, you must make the whole round trip symmetric — at the cloud layer and in the kernel.

Lever 1 · Mask the pod IP with default SNAT (usually the right call)

If the destination doesn't care about the pod's real address — most RDS, cache, and internal-service traffic doesn't — the cleanest fix is to not preserve it. With the default AWS_VPC_K8S_CNI_EXTERNALSNAT=false, the CNI source-NATs off-VPC pod traffic to the node's primary eth0 IP. The 100.64 CIDR never appears on the Transit Gateway, so there is no secondary CIDR to route back and no interface mismatch to trip rp_filter. The return path is symmetric by construction.

# revert to the default: pod egress is SNAT'd to the node's eth0 IP
kubectl set env daemonset -n kube-system aws-node \
  AWS_VPC_K8S_CNI_EXTERNALSNAT=false

# roll the nodes (or the aws-node pods) so the change takes effect
kubectl -n kube-system rollout restart daemonset aws-node

Lever 2 · Keep the pod IP and make the whole path symmetric

When the destination genuinely must see 100.64.12.34 — IP-based access control on the database, an appliance that logs pod identity, or inbound-initiated flows — set EXTERNALSNAT=true and then own the return path completely. Three things have to line up.

(a) Return routes for the secondary CIDR. The reply needs a way home at every hop. Add 100.64.0.0/16 pointing at the TGW in the destination VPC, and pointing at your EKS VPC attachment in the TGW route table.

# in the DATABASE VPC's route table: send pod-CIDR replies to the TGW
aws ec2 create-route \
  --route-table-id rtb-0db1example \
  --destination-cidr-block 100.64.0.0/16 \
  --transit-gateway-id tgw-0a1b2example

# in the TGW route table: send the pod CIDR to the EKS VPC attachment
aws ec2 create-transit-gateway-route \
  --transit-gateway-route-table-id tgw-rtb-0c3d4example \
  --destination-cidr-block 100.64.0.0/16 \
  --transit-gateway-attachment-id tgw-attach-0e5f6example

(b) Appliance mode, if a stateful device is in the path. If traffic passes through a firewall, NAT instance, or inspection VPC, TGW AZ affinity will split the flow across AZs and the appliance will drop the half it has no state for. Turn on appliance mode on that attachment so the TGW keeps the entire flow on one network interface in both directions.

# pin each flow to one path both ways (fixes stateful-appliance drops)
aws ec2 modify-transit-gateway-vpc-attachment \
  --transit-gateway-attachment-id tgw-attach-0e5f6example \
  --options ApplianceModeSupport=enable

(c) Loose rp_filter on the node. Even with symmetric cloud routing, if the reply can arrive on an interface that isn't the kernel's best reverse path, strict rp_filter drops it. Set loose mode — and remember to fix conf.all, the floor, not just one interface.

# loose mode (2): accept if the source is reachable via ANY interface.
# set both 'all' (the floor) and the interfaces.
sudo sysctl -w net.ipv4.conf.all.rp_filter=2
sudo sysctl -w net.ipv4.conf.default.rp_filter=2
sudo sysctl -w net.ipv4.conf.eth0.rp_filter=2

Pitfalls that send you chasing the wrong layer

• Trusting the console because "the route exists." Every route table can be correct and the flow still dies at rp_filter or a stateful appliance. Asymmetry is about the relationship between two paths, not the presence of either.
• Fixing eth0 but not all. The kernel uses max(conf.all, conf.<iface>). Loose on the interface with strict on all is still strict. This wastes hours.
• Assuming modern nodes are safe. The CNI sets the primary interface to loose, but a hardening baseline, a custom AMI, or the AL2023 systemd-networkd/CNI boot race can leave strict filtering in place. Check the live sysctl value, don't assume.
• Reaching for EXTERNALSNAT=true reflexively. It's the setting that causes the trap, not the one that fixes it. Only preserve the pod IP when a downstream requirement actually needs it; otherwise the default SNAT is simpler and safer.
• Forgetting appliance mode with centralized inspection. A pod-to-RDS flow through an inspection VPC is a textbook cross-AZ, cross-VPC stateful case. Without appliance mode it's a coin flip whether any given connection survives.
• Testing from the wrong pod. A pod pinned to the same AZ as the destination may route symmetrically and "work," masking the bug. Reproduce with a pod in a different AZ from the target before declaring victory.