NSX-V IP Discovery

Thanks to Dimitri Desmidt for feedbacks.

IP discovery Allow NSX to suppress the ARP message over the logical switch.

To understand why we need IP Discovery and how its work we need some background regard ARP.

IP Discovery

What is ARP (Address Resolution Protocol)?

When VM1 need to communicate with VM2 it need know VM2 MAC Address, the way to know MAC2 is to send a broadcast message (ARP request) to all VM in the same L2 segment (same VLAN or in the example above same VXLAN 5001).

ALL VM’s on this Logical Switch will see this message including VM3 since it’s a broadcast, but only VM’2 will respond. The response will come in Unicast Message from VM2 directly to VM1 mac@ with VM’2 MAC address (MAC2) in the response body.

 

topology

VM1 will cache the mac@ of VM2 IP@ in its ARP table. The entry it saved between few seconds to few minutes depending on the Operating System.

Windows 7 OS for example

http://support.microsoft.com/kb/949589

If VM1 and VM2 will not talk again in this Cache time window, VM1 will clear is ARP table entry for that MAC2, when next time VM1 will need to talk to VM2, VM1 OS will send again ARP message to relearn same MAC2 of VM2.

Note: In the unlikely event of the NSX Controller who dodoesn’tnow the mac@ for VM2-IP@, then the ARP request message is flooded, but only to the ESXi that have VMs in the logical switch 5001.

 

How IP Discovery works:

VMware NSX leverage NSX controller to achieve IP Discovery.

Inside ESXi host running with NSX software there is process called User World Agent (UWA), this process communicate with NSX  controller and update the controller directory MAC,IP,VTEP tables for VM’s reside inside this ESXi host.

NSX Arch

When VM connect to Logical switch there are few security services that pack a transverse, each service represent with different slot id.

Filter slots

SLOT 0 : implement vDS Access List

SLOT 1: Switch Security module (swsec) capture DHCP Ack and ARP message, this info then forward to NSX Controller.

SLOT2: NSX Distributed Firewall.

From the figure above we now understand that slot 1 is the service responsibly to implement the IP Discovery.

When VM1 power up even if the ip address is static the VM will send out ARP message do discover the MAC address of the default gateway, when swsec module see this ARP message he will forward to NSX Controller. That way NSX controller learn VM1 MAC1 address, same way Controller will learn VM2 MAC2 address.

Now when VM1 want to talk to VM2, MAC2 is not known to VM1, then ARP message will send out to VXLAN 5001.

The UWA will send out query to NSX controller and ask if he know MAC2, since controller already know this Controller will  send unicast message back to VM1 with MAC2, the ARP broadcast message will not send out to all VM’s  in VXLAN 5001.

Note: There is 3 min timer in NSX controller for ARP query, if host send same query in this time frame the controller ignore this request and broadcast message will be send out to all VM in the logical switch

 

IP Discovery Verification:

The easiest to know if IP discovery it actually works is to run Wireshark software in VM3, clear the ARP table in VM1 with the command: arp –d.

Now ping from VM1 to VM2, ARP broadcast message from VM1 should not see in VM3.

I would like to point out grate post explain in deep dive how IP discovery work by Dmitri Kalintsev

NSX-v under the hood: VXLAN ARP suppression

 

Posted in Install, Troubleshooting Tagged with:
2 comments on “NSX-V IP Discovery
  1. Parveen says:

    Thanks for providing details on the topic , but i need to know the packet Flow for VXLAN Control Plane Unicast and Hybrid Mode , is there any documentation which help me to understand the UTEP and MTEP , Proxy VTEP Concept and How the Packet flow for BUM Traffic . or can u write an Post on the same .

    Thanks .

Leave a Reply