- Utilize net-dvs to troubleshoot vNetwork Distributed Switch configurations
- Utilize vSphere CLI commands to troubleshoot ESXi network configurations
- Troubleshoot Private VLANs
- Troubleshoot vmkernel related network configuration issues
- Troubleshoot DNS and routing related issues
- Use esxtop/resxtop to identify network performance problems
- Analyze troubleshooting data to determine if the root cause for a given network problem originates in the physical infrastructure or vSphere environment
- Configure and administer Port Mirroring
- Utilize Direct Console User Interface (DCUI) and ESXi Shell to troubleshoot, configure, and monitor ESXi networking
Utilize net-dvs to troubleshoot vNetwork Distributed Switch configurations
There is not much official documentation on the net-dvs command. The reason for this is probably because the command is unsupported.
As you can see, most options are not documented. The most common options:
To show the config of all vSphere Distibuted Switches (vDS):
Or as Duncan Epping demonstrates:
# net-dvs –f /etc/vmware/dvsdata.db
To show the config of a specific vDS:
# net-dvs –l <vDS name>
With other options, it seems possible to control and edit a vDS on various levels, e.g. try this command and return to your vSphere Client to see what happened.
Note: dvSwitch02 must exist and be connected to the ESXi host.
# net-dvs -H “red:dvSwitch02 is Down” dvSwitch02
- I highly recommend reading this post and this post by Duncan Epping on vDS. The first post “Digging deeper into the VDS construct” shows usage of the net-dvs command.
- See also VMware KB 1020736 “Adding an ESX host into a Distributed Virtual Switch fails with the error: Unable to Create Proxy DVS”
Utilize vSphere CLI commands to troubleshoot ESXi network configurations
vSphere Command-Line Interface Concepts and Examples, Chapter 9 “Managing vSphere Networking”, page 109.
VMware does not make life easy when it comes to the CLI (some people won’t agree with me).
Identical actions can be performed using:
- vSphere Client;
- vSphere Web Client
- vSphere PowerCLI
- vSphere CLI
Before vSphere 5.x, most vSphere CLI commands were in the vicfg- or esx-cfg- format, e.g.: for an overview of vSphere switches:
# esxcfg-switch –l
But VMware is shifting to the ESXCLI command. For an overview, see my post. So to get an overview of connected distributed switches use this esxcli command:
# esxcli network vswitch dvs vmware list
For an overview of the Standard switches:
# esxcli network vswitch standard list
Troubleshooting starts with collecting information. Some useful commands:
For an overview of VMkernel ports:
# esxcli <conn_options> network ip interface list
For an overview of the configuration of all ipv4 VMkernel ports:
# esxcli <conn_options> network ip interface ipv4 get
For an overview of the configuration of a specific ipv6 VMkernel port:
# esxcli <conn_options> network ip interface ipv6 get -i vmk<X>
For information corresponding to the Linux netstat command, use the following ESXCLI command.
# esxcli <conn_options> network ip connection list
Note: <conn_options>, not needed while directly connected with ESXi console or SSH session. While using the vMA, you need to specify connection information.
Troubleshoot Private VLANs
vSphere Networking, Chapter 3 “Setting up Networking with vSphere Distributed Switches”, Section “Private VLANs”, page 27.
Private VLANs have been discussed in Objective 2.2.
- A good
Troubleshoot VMkernel related network configuration issues
The most important function of a VMkernel interface is for Management traffic of an ESXi host.
By default, on ESXi, Management Traffic is on VMkernel interface vmk0.
Because of the importance of Management Traffic, you are advised to create a secondary management interface to provide redundancy.
In an all down situation, the only way out is a Remote Access Interface (ILO, DRAC etc.) or a Console.
Management Traffic is highly important; to get an overview of all possible connections, have a look at this overview (Thank you Forbes Guthrie!).
Besides Management traffic, VMkernel interface are also used for:
- vMotion traffic;
- Fault Tolerant Logging;
- iSCSI traffic;
- not necessary but advised for NFS.
Each VMkernel interface has to be configured with a correct IP address and Subnet Mask.
Some tips for troubleshooting VMkernel interface issues:
- You cannot have more than one VMkernel Default Gateway
- If you use VLANs, VLAN IDs are correct and trunk port have been configured correctly?
- Another useful command for troubleshooting is
To get an idea, use
# esxcfg-route –help
- If you have lost connectivity to your Management network and convinced everything has been configured correctly, try restarting the Management Agents. You can commandline
or use the DCUI, option: Restart Management Network
- CDP can also be useful, see Objective 2.2.
Troubleshoot DNS and routing related issues
vSphere Command-Line Interface Concepts and Examples, Chapter 9 “Managing vSphere Networking”, section “Setting the DNS Configuration”, page 123.
DNS is important for many VMware vSphere features and therefore must be configured correctly. You can configure/edit DNS and routing with the vSphere Client or with CLI commands. Also important:
- DNS server(s) must be available and work correctly;
- ESXi host and the vCenter Server(s) must have entries in the DNS;
- You can check using various commands, like ping or nslookup;
- From the DCUI you can use the “Test Management Network”
This test will attempt to ping the default gateway, DNS servers and resolve the hostname.
- After applying changes flush the DNS cache on the vCenter Server using this command:
cmd> ipconfig /flushdns
- Other vSphere CLI tools are:
# esxcli network ip dns seach list
# esxcli network ip dns server list
A default gateway is only needed if multiple subnets / VLANs exist in your infrastructure. A default gateway is configured – just like any other PC or server – for the vCenter Server and ESXi hosts.
There is only one default gateway, on a ESXI host you (re)configure it with:
- vSphere Client
- # esxcfg-route
to specify a default gateway:
# esxcfg-route –a default <default gateway IP>
Use esxtop/resxtop to identify network performance problems
While using esxtop/restop to identify network performance problems. Read objective 3.4 how to use esxtop.
Watch out for Dropped packets Received [%DRPRX] at a virtual switch. This indicates that the VM network driver runs out of receive (Rx) buffers, so it’s a buffer overflow (Eric Sloof, thank you for this!).
You can resolve this by increasing the Rx buffers for the virtual network driver. This works for VMs configured with a VMXNET3 vmnic or E1000 with native driver installed in the guest OS.
Esxtop also presents information on questions like:
- How are my physical NICs doing, is load equally distributed over available NICs?
- Which VM are generating high network traffic?
Analyze troubleshooting data to determine if the root cause for a given network problem originates in the physical infrastructure or vSphere environment
General recommendations for troubleshooting virtual network troubleshooting:
- Start Bottom-up instead of Top Down;
- Start with physical Layer (L1) of the OSI Model and work your way up.
- Know the concepts of Standard switches and Distributed switches.
Understand the difference between VM portgroups and VMkernel Portgroups.
Know how to configure VMkernel Portgroups.
Understand physical uplinks, NIC teaming and Security settings.
Physical NICs are connected to physical switches.
Know how switch ports are configured, access port, trunk port, which VLANs are allowed.
- dvSwitches can standardize configurations across all hosts but also complicate troubleshooting.
- Avoid the urge to reboot and continue searching for the root cause (your evidence has usually gone after a reboot.
Based on the “vSphere Troubleshooting Training” by David Davis, Train Signal.
- VMware vSphere Troubleshooting Training by Trainsignal.
Configure and administer Port Mirroring
vSphere Networking, Chapter 6 “Advanced Networking”, Section “Working with Port Mirroring”, page 66.
Port Mirroring aalows you to mirror a port’s traffic to another switch port or physical switch port/
Port Mirroring is only available on Distributed Switches Version 5.0.0 and higher.
Configuring Port Mirroring is done on the vDS level, by creating a new Mirroring Session in four steps.
In the first step, at a minimum, you need to specify a Session name. Options are:
- Allow normal IO on destination ports.
If you do not select this option, mirrored traffic will be allowed out on destination ports, but no traffic will be allowed in.
- Encapsulation VLAN, allows you to create a new VLAN ID.
Note: If the original frames have a VLAN and Preserve original VLAN is not selected, the encapsulation VLAN replaces the original VLAN.
Choose the Traffic direction Egress, Ingress or both and the port IDs that should be mirrored. You can specify ranges and enter multiple values.
A destination can be:
- Physical uplink, to forward to a physical switch port;
- vDS Port ID.
Note: Port Mirroring is checked against the VLAN forwarding policy. If the VLAN of the original frames is not equal to or trunked by the destination port, the frames are not mirrored!
Verify the settings and do not forget to enable the configured port mirroring session!
- Video how to setup vSphere 5 Port Mirror by Eric Sloof.
Utilize Direct Console User Interface (DCUI) and ESXi Shell to troubleshoot, configure, and monitor ESXi networking
DCUI can be reached:
- Directly from the physical console or Remote Access Card (ILO, DRAC)
- From an existing SSH session to an ESXi host, type:
The DCUI offers you options for:
- Adjusting root password;
- Configure, Restart an Test the Management network;
- Restore Network Setting or even Restore a standard switch (very useful option, in case you have meshed up you vDS)
- Troubleshooting options, enabling SSH or the ESXi shell and restarting the Management Agents
- View the ESXi logging
- Finally, resetting the ESXi configuration to default settings!
The ESXi shell or a SSH gives you access to the “console” of an ESXi host.
Although much smaller than the “Classical” ESX console, the ESXi shell still has a lot to offer. To get an idea of available commands:
- To get an overview of available Unix-like utilities:
- The commands made available by Busybox are located in the /bin folder. Here you can also the symbolic links to the commands
- In the /sbin folder, you will find the more VMware specific commands, like the esxcfg- commands, esxcli, esxtop, net-dvs and vmkping
More information on thes commands can be found in vSphere Command-Line Interface Concepts and Examples document.