VCAP5-DCA Objective 6.3 -Troubleshoot Network Performance and Connectivity

Objectives

  • Utilize net-dvs to troubleshoot vNetwork Distributed Switch configurations
  • Utilize vSphere CLI commands to troubleshoot ESXi network configurations
  • Troubleshoot Private VLANs
  • Troubleshoot vmkernel related network configuration issues
  • Troubleshoot DNS and routing related issues
  • Use esxtop/resxtop to identify network performance problems
  • Analyze troubleshooting data to determine if the root cause for a given network problem originates in the physical infrastructure or vSphere environment
  • Configure and administer Port Mirroring
  • Utilize Direct Console User Interface (DCUI) and ESXi Shell to troubleshoot, configure, and monitor ESXi networking

Utilize net-dvs to troubleshoot vNetwork Distributed Switch configurations

Official Documentation:

Summary:
There is not much official documentation on the net-dvs command. The reason for this is probably because the command is unsupported.

Figure 1

As you can see, most options are not documented. The most common options:

To show the config of all vSphere Distibuted Switches (vDS):

# net-dvs

Or as Duncan Epping demonstrates:

# net-dvs –f /etc/vmware/dvsdata.db

To show the config of a specific vDS:

# net-dvs –l <vDS name>

With other options, it seems possible to control and edit a vDS on various levels, e.g. try this command and return to your vSphere Client to see what happened.
Note: dvSwitch02 must exist and be connected to the ESXi host.

# net-dvs -H “red:dvSwitch02 is Down” dvSwitch02

Other references:

 

Utilize vSphere CLI commands to troubleshoot ESXi network configurations

Official Documentation:
vSphere Command-Line Interface Concepts and Examples, Chapter 9 “Managing vSphere Networking”, page 109.

Summary:
VMware does not make life easy when it comes to the CLI (some people won’t agree with me).

Identical actions can be performed using:

  • vSphere Client;
  • vSphere Web Client
  • vSphere PowerCLI
  • vSphere CLI

Before vSphere 5.x, most vSphere CLI commands were in the vicfg- or esx-cfg- format, e.g.: for an overview of vSphere switches:

# esxcfg-switch –l

But VMware is shifting to the ESXCLI command. For an overview, see my post. So to get an overview of connected distributed switches use this esxcli command:

# esxcli network vswitch dvs vmware list

For an overview of the Standard switches:

# esxcli network vswitch standard list

Troubleshooting starts with collecting information. Some useful commands:

For an overview of VMkernel ports:

# esxcli <conn_options>  network ip interface list

For an overview of the configuration of all ipv4 VMkernel ports:

# esxcli <conn_options> network ip interface ipv4 get

For an overview of the configuration of a specific ipv6 VMkernel port:

# esxcli <conn_options> network ip interface ipv6 get -i vmk<X>

For information corresponding to the Linux netstat command, use the following ESXCLI command.

# esxcli <conn_options> network ip connection list

Note: <conn_options>, not needed while directly connected with ESXi console or SSH session. While using the vMA, you need to specify connection information.

Other references:

  • A

 

Troubleshoot Private VLANs

Official Documentation:
vSphere Networking, Chapter 3 “Setting up Networking with vSphere Distributed Switches”, Section “Private VLANs”, page 27.

Summary:
Private VLANs have been discussed in Objective 2.2.

Other references:

  • A good

 

Troubleshoot VMkernel related network configuration issues

Official Documentation:

Summary:
The most important function of a VMkernel interface is for Management traffic of an ESXi host.

By default, on ESXi, Management Traffic is on VMkernel interface vmk0.

Because of the importance of Management Traffic, you are advised to create a secondary management interface to provide redundancy.

In an all down situation, the only way out is a Remote Access Interface (ILO, DRAC etc.) or a Console.

Management Traffic is highly important; to get an overview of all possible connections, have a look at this overview (Thank you Forbes Guthrie!).

Besides Management traffic, VMkernel interface are also used for:

  • vMotion traffic;
  • Fault Tolerant Logging;
  • iSCSI traffic;
  • not necessary but advised for NFS.

Each VMkernel interface has to be configured with a correct IP address and Subnet Mask.

Some tips for troubleshooting VMkernel interface issues:

  • You cannot have more than one VMkernel Default Gateway
  • If you use VLANs, VLAN IDs are correct and trunk port have been configured correctly?
  • Another useful command for troubleshooting is
    # esxcfg-route
    To get an idea, use
    # esxcfg-route –help
  • If you have lost connectivity to your Management network and convinced everything has been configured correctly, try restarting the Management Agents. You can commandline
    # /sbin/services.sh
    or use the DCUI, option: Restart Management Network
  • CDP can also be useful, see Objective 2.2.

Other references:

  • A

 

Troubleshoot DNS and routing related issues

Official Documentation:
vSphere Command-Line Interface Concepts and Examples, Chapter 9 “Managing vSphere Networking”, section “Setting the DNS Configuration”, page 123.

Summary:
DNS is important for many VMware vSphere features and therefore must be configured correctly. You can configure/edit DNS and routing with the vSphere Client or with CLI commands. Also important:

  • DNS server(s) must be available and work correctly;
  • ESXi host and the vCenter Server(s) must have entries in the DNS;
  • You can check using various commands, like ping or nslookup;
  • From the DCUI you can use the “Test Management Network”

Figure 2

This test will attempt to ping the default gateway, DNS servers and resolve the hostname.

  • After applying changes flush the DNS cache on the vCenter Server using this command:
    cmd> ipconfig /flushdns
  • Other vSphere CLI tools are:
    # esxcli network ip dns seach list
    # esxcli network ip dns server list

Routing
A default gateway is only needed if multiple subnets / VLANs exist in your infrastructure. A default gateway is configured – just like any other PC or server – for the vCenter Server and ESXi hosts.

There is only one default gateway, on a ESXI host you (re)configure it with:

  • vSphere Client
  • # esxcfg-route
    to specify a default gateway:
    # esxcfg-route –a default <default gateway IP>

Other references:

  • A

 

Use esxtop/resxtop to identify network performance problems

Official Documentation:

Summary:
While using esxtop/restop to identify network performance problems. Read objective 3.4 how to use esxtop.

Watch out for Dropped packets Received [%DRPRX] at a virtual switch. This indicates that the VM network driver runs out of receive (Rx) buffers, so it’s a buffer overflow (Eric Sloof, thank you for this!).

Figure 3

You can resolve this by increasing the Rx buffers for the virtual network driver. This works for VMs configured with a VMXNET3 vmnic or E1000 with native driver installed in the guest OS.

Esxtop also presents information on questions like:

  • How are my physical NICs doing, is load equally distributed over available NICs?
  • Which VM are generating high network traffic?

Other references:

  • A

 

Analyze troubleshooting data to determine if the root cause for a given network problem originates in the physical infrastructure or vSphere environment

Official Documentation:

Summary:
General recommendations for troubleshooting virtual network troubleshooting:

  • Start Bottom-up instead of Top Down;
  • Start with physical Layer (L1) of the OSI Model and work your way up.
  • Know the concepts of Standard switches and Distributed switches.
    Understand the difference between VM portgroups and VMkernel Portgroups.
    Know how to configure VMkernel Portgroups.
    Understand physical uplinks, NIC teaming and Security settings.
    Physical NICs are connected to physical switches.
    Know how switch ports are configured, access port, trunk port, which VLANs are allowed.
  • dvSwitches can standardize configurations across all hosts but also complicate troubleshooting.
  • Avoid the urge to reboot and continue searching for the root cause (your evidence has usually gone after a reboot.

Based on the “vSphere Troubleshooting Training” by David Davis, Train Signal.

Other references:

 

Configure and administer Port Mirroring

Official Documentation:
vSphere Networking, Chapter 6 “Advanced Networking”, Section “Working with Port Mirroring”, page 66.

Summary:
Port Mirroring aalows you to mirror a port’s traffic to another switch port or physical switch port/

Port Mirroring is only available on Distributed Switches Version 5.0.0 and higher.

Configuring Port Mirroring is done on the vDS level, by creating a new Mirroring Session in four steps.

Figure 4

In the first step, at a minimum, you need to specify a Session name. Options are:

  • Description;
  • Allow normal IO on destination ports.
    If you do not select this option, mirrored traffic will be allowed out on destination ports, but no traffic will be allowed in.
  • Encapsulation VLAN, allows you to create a new VLAN ID.
    Note: If the original frames have a VLAN and Preserve original VLAN is not selected, the encapsulation VLAN replaces the original VLAN.

Figure 5

Choose the Traffic direction Egress, Ingress or both and the port IDs that should be mirrored. You can specify ranges and enter multiple values.

Figure 6

A destination can be:

  • Physical uplink, to forward to a physical switch port;
  • vDS Port ID.

Note: Port Mirroring is checked against the VLAN forwarding policy. If the VLAN of the original frames is not equal to or trunked by the destination port, the frames are not mirrored!

Figure 7

Verify the settings and do not forget to enable the configured port mirroring session!

Other references:

  • Video how to setup vSphere 5 Port Mirror by Eric Sloof.

 

Utilize Direct Console User Interface (DCUI) and ESXi Shell to troubleshoot, configure, and monitor ESXi networking

Official Documentation:

Summary:
DCUI can be reached:

  • Directly from the physical console or Remote Access Card (ILO, DRAC)
  • From an existing SSH session to an ESXi host, type:
    # DCUI

The DCUI offers you options for:

  • Adjusting root password;
  • Configure, Restart an Test the Management network;
  • Restore Network Setting or even Restore a standard switch (very useful option, in case you have meshed up you vDS)
  • Troubleshooting options, enabling SSH or the ESXi shell and restarting the Management Agents
  • View the ESXi logging
  • Finally, resetting the ESXi configuration to default settings!

The ESXi shell or a SSH gives you access to the “console” of an ESXi host.

Although much smaller than the “Classical” ESX console, the ESXi shell still has a lot to offer. To get an idea of available commands:

  • To get an overview of available Unix-like utilities:
    # busybox
  • The commands made available by Busybox are located in the /bin folder. Here you can also the symbolic links to the commands
  • In the /sbin folder, you will find the more VMware specific commands, like the esxcfg- commands, esxcli, esxtop, net-dvs and vmkping

More information on thes commands can be found in vSphere Command-Line Interface Concepts and Examples document.

Other references:

  • A
Advertisements

4 Responses to VCAP5-DCA Objective 6.3 -Troubleshoot Network Performance and Connectivity

  1. Adam says:

    I am pursuing either a VCAP-DCA or DCD but can’t decide right now. What drew you to datacenter admin?

    • paulgrevink says:

      Hello Adam,

      Thank you for your feedback. I have chosen to start with the DCA exam, because I also work as a sysadmin in VMware, Windows and Linux environments. I am especially interested in details like Troubleshooting, Performance Monitoring and Security related issues. My next move will be in the Design direction. It is a kind of bottom up strategy. I am also convinced that a good understanding of networking and storage will help you making a better design. I hope this will answer your question.

      Best regards,

      Paul

  2. […] Objective 6.3, section “Utilize Direct Console User Interface (DCUI) and ESXi Shell to troubleshoot, configure, and monitor ESXi networking”; […]

  3. […] Objective 6.3, section “Utilize Direct Console User Interface (DCUI) and ESXi Shell to troubleshoot, configure, and monitor ESXi networking”; […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: