NSX Minimum MTU


What is the Minimum MTU for VMware NSX  ?


The VXLAN RFC can be found at the link:

https://www.rfc-editor.org/rfc/rfc7348.txt

From each ESXi host we can run the command:
pktcap-uw –capture UplinkSnd –uplink vmnic1 -o /tmp/cap2.pcap

This command will capture all the traffic sent from the local VTEP toward the physical switch and save it in a file named cap2 with pcap format. While running this command, ping from one guest 192.168.1.1 to another guest 192.168.1.2 (hosted in a different ESXi host) to generate some traffic.

With WinSCP we can bring the pcap file from the ESXi host to my Windows PC and open it with WireShark.

Opening the file will show us something like this:

Wireshark 1

Wireshark 1

We can see udp traffic from VTEP host 192.168.64.130 to VTEP 192.168.64.131 dest to port 8472 (VXLAN) but where is the VXLAN header ?

View VXLAN Header

Wireshark can display VXLAN traffic, but for doing that we just need to change decode to VXLAN!!!
Right Click to the frame and chose “Decode As…”

wireshark decode as vxlan

wireshark decode as vxlan

 

Change the “Transport” to VXLAN

Transport k decode as vxlan

Transport k decode as vxlan

 

wireshark display VXLAN

wireshark display VXLAN

Now we can see the VXLAN header

Capture4

 

MTU Math Time

MTU Math

MTU Math

 

Outside MTU for IPv4 without Internal Guest OS dot1q Tagging = 20 + 8 + 8 + 14 + 1500  = 1550 bytes

Outside MTU for IPv4 with Internal Guest OS dot1q Tagging = 20 + 8 + 8 + 14 + 4 + 1500  = 1554 bytes

For IPv6 we will need to add more 20 bytes to Outer IPv4 so total max MTU  will be 1574 bytes

 

IPv4 with VXLAN

IPv4 with VXLAN

Conclusion

When we configure VXLAN in DSwitch keeping the default MTU 1600 will keep you in the safe side!!!

NSX MTU 1600

NSX MTU 1600