What is the Minimum MTU for VMware NSX ?
The VXLAN RFC can be found at the link:
From each ESXi host we can run the command:
pktcap-uw –capture UplinkSnd –uplink vmnic1 -o /tmp/cap2.pcap
This command will capture all the traffic sent from the local VTEP toward the physical switch and save it in a file named cap2 with pcap format. While running this command, ping from one guest 192.168.1.1 to another guest 192.168.1.2 (hosted in a different ESXi host) to generate some traffic.
With WinSCP we can bring the pcap file from the ESXi host to my Windows PC and open it with WireShark.
Opening the file will show us something like this:
We can see udp traffic from VTEP host 192.168.64.130 to VTEP 192.168.64.131 dest to port 8472 (VXLAN) but where is the VXLAN header ?
View VXLAN Header
Wireshark can display VXLAN traffic, but for doing that we just need to change decode to VXLAN!!!
Right Click to the frame and chose “Decode As…”
Change the “Transport” to VXLAN
Now we can see the VXLAN header
MTU Math Time
Outside MTU for IPv4 without Internal Guest OS dot1q Tagging = 20 + 8 + 8 + 14 + 1500 = 1550 bytes
Outside MTU for IPv4 with Internal Guest OS dot1q Tagging = 20 + 8 + 8 + 14 + 4 + 1500 = 1554 bytes
For IPv6 we will need to add more 20 bytes to Outer IPv4 so total max MTU will be 1574 bytes
When we configure VXLAN in DSwitch keeping the default MTU 1600 will keep you in the safe side!!!