What is Asymmetric Routing?
In Asymmetric routing, a packet traverses from a source to a destination in one path and takes a different path when it returns to the source.
Start from version 6.1 NSX Edge can work with ECMP – Equal Cost Multipath, ECMP traffic involved Asymmetric routing between Edges and DLR or between Edge and physical routers.
ECMP Consideration with Asymmetric Routing
ECMP with Asymmetric routing is not a problem by itself, but will cause problems when more than one NSX Edge in place and stateful services inserted in the path of the traffic.
Stateful services like firewall, Load Balanced Network Address Translation (NAT) can’t work with asymmetric routing.
Explain the problem:
User from outside try to access Web VM inside the Data Center. the traffic will pass through E1 Edge.
From E1 the traffic will go to DLR transverse NSX distributed firewall and get to Web VM.
When Web VM respond back the traffic will hit the DLR default gateway. DLR have two option to route the traffic E1 or E2.
If DLR choose E2 the traffic will get the E2 and will Dropped !!!
The reason for this is E2 does not aware the state of session started at E1, replay packet from Red VM arrived to E2 are not match any existing session at E2.
From E2 perspective this is new session need to validate, any new TCP session should start with SYN, since this is not the begin of the session E2 will drop it!!!
Note: NSX Distributed firewall is not part of this problem, NSX Distributed firewall implement at the vNic level, all traffic get in/out same vNic.
there is no Asymmetric route in the vNic level, btw this is the reason when we vMotion VM, the Firewall Rule, Connection state is move with the VM itself.
ECMP and Edge Firewall NSX
Starting from version 6.1 when we enable ECMP on NSX Edge get message:
The firewall service disabled by default:
Even if you try to enable it you will get warning message:
In version 6.1.2 when we enable ECMP we get same message:
But the BIG difference is Firewall Service is Not disable by default. (you need to turn it off)
Even if you have “Any, Any” rule with “Accept” action we still be subject for DROP packet subject of the Asymmetric routing problem!!!
Even in Syslog or LogInSight you will not see this DROP packet !!!
The end users expirese for will be some of the session’s are working just fine (this sessions are not asymmetric) other session will drop (asymmetric sessions)
The place i found we can learn packet are drops because state of the session is with the command: show tech-support:
show tech-support vShield Edge Firewall Packet Counters: ~~~~~~~~~~~~~~~ snip ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes) rid pkts bytes target prot opt in out source destination 0 20 2388 ACCEPT all -- * lo 0.0.0.0/0 0.0.0.0/0 0 12 720 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID 0 51 7108 block_out all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap0 --physdev-out vNic_+ 0 0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vNic_+ --physdev-out tap0 0 0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in na+ --physdev-out vNic_+ 0 0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vNic_+ --physdev-out na+ 0 0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 0 51 7108 usr_rules all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
From line 7 we can see DROP packet because of INVALID state.
When you enable ECMP and you have more then one NSX Edge in you topology, go to Firewall service and disable it by yourself otherwise you will spend lots of troubleshooting hours 🙁