VCAP5-DCA Objective 3.4 – Utilize advanced vSphere Performance Monitoring tools

paulg

Objectives

  • Configure esxtop/resxtop custom profiles
  • Determine use cases for and apply esxtop/resxtop Interactive, Batch and Replay modes
  • Use vscsiStats to gather storage performance data
  • Use esxtop/resxtop to collect performance data
  • Given esxtop/resxtop output, identify relative performance data for capacity planning purposes

Configure esxtop/resxtop custom profiles

Official Documentation:
vSphere Monitoring and Performance Guide, Chapter 7 “Performance Monitoring Utilities: resxtop and esxtop”, page 45.
And not the Resource Management Guide as the Blueprint states…

Summary:
Chapter 7 presents a nice overview.

The resxtop and esxtop command-line utilities provide a detailed look at how ESXi uses resources in real time. For thos familiar with Unix/Linux esxtop is the vSphee equivalent of the well known top command.

esxtop can be run from the Shell of an ESXi server and can only be used locally on a ESXi host. You need root privileges to run esxtop.
resxtop stands for remote esxtop and is found in the vMA or in the vSphere CLI. For remote connections, you can connect to a host either directly or through vCenter Server.
Whenusing the vMA, resxtop is vifp aware.

Both utilities operate in 3  modes:

  • interactive (default),
  • batch
  • replay.

A lot has been written about the available options. A few tips to get started:

If you do forgot about options,  for both commands type:

# esxtop –help or # esxtop -h

The output is slightly different. Esxtop has the Replay mode and a few experimental features. Resxtop, of course has the options for remote connections.

Figure 1 –  esxtop

And

Figure 2 – rexstop

You start esxtop in interactive mode, this way:

# esxtop

Figure 3 – CPU resource

By default esxtop presents the CPU resource utilization panel. This panel (as most of the others) consists of:

  • The upper part displays server-wide statistics
  • The lower part presents statistics for individual worlds (everything below the grey column header)

By pressing the following keys, you can switch to the other panels:

c          Switch to the CPU resource utilization panel.

p          Switch to the CPU Power utilization panel.

m         Switch to the memory resource utilization panel.

d          Switch to the storage (disk) adapter resource utilization panel.

u          Switch to storage (disk) device resource utilization screen.

v          Switch to storage (disk) virtual machine resource utilization screen.

n          Switch to the network resource utilization panel.

i           Switch to the interrupt panel

Each panel shows you a lot of information. vSphere Monitoring and Performance Guide, Chapter 7 provides  information about individual Columns.

When you have started esxtop, the easiest way to get support, is to press the ‘h’ key to get help.

Figure 4 – h

Most commands are executed by pressing a single letter.
Note: esxtop is Case-Sensitive, so lower case L and capital L are different options.

You can define the order of fields displayed in interactive mode by pressing the f,F,o or O key. The f and F key let you select columns. The o and O key let you change the order.

Now we come to the subject of this objective. You have played around, picked your favourite view, columns and sort order and want to keep this settings, do nthe following:

  • Press ‘W’;
  • Enter a name for you configuration file and press enter

Figure 5 – Save settings

Now you can start esxtop with your saved configuration file, using this command:

# esxtop –c esxtop-mem

Eventually, the complete path to your config file.

It is good idea to get familiar with esxtop and play around. esxtop is extremely useful while troubleshooting.

Please, read the “Other references” section for more information.

You start esxtop in batch mode, this way:

# esxtop –c <config file> -b –d 2 –n 5 > <output.csv>

Where:

-c <config file>, first prepare a view, so you get only the data you really need

-b, the command to start batch mode

-d <delay in seconds>

-i <number of iterations>

In replay mode, esxtop replays resource utilization statistics collected using vm-support.

First you run:
# vm-support –p –d <duration> -i <interval in seconds>

You start esxtop in replay mode, this way:

# esxtop –R <path to output vm-support>

Default location for vm-support output is /var/tmp. First you have to unzip the data, using the command:

# tar –xzf <path to output vm-support>

Figure 6 – vm-support options and action

Other references:

Determine use cases for and apply esxtop/resxtop Interactive, Batch and Replay modes

Official Documentation:
vSphere Monitoring and Performance Guide, Chapter 7 “Performance Monitoring Utilities: resxtop and esxtop”, page 45.

Summary:
See also the first section of this objective.

Imho, in most cases you will use esxtop/resxtop in Interactive mode. In cases you want to collect data for reference, demonstration or support usage, Batch mode is the preferred way.

Replaymode is only useful combined with the vm-support tool. vm-support is intended to collect data for VMware support incidents.

Other references:

  • A

Use vscsiStats to gather storage performance data

Official Documentation:
“Using vscsiStats for Storage Performance Analysis”, from the VMware Communities seems to be the official documentation  on this subject. Unofficial are a lot of excellent Blog posts. I will mention a few in the “Other references” section.

Summary:
From the Communities:

“esxtop is a great tool for performance analysis of all types.  However,  with only latency and throughput statistics, esxtop will not provide the  full picture of the storage profile.  Furthermore, esxtop only provides  latency numbers for Fibre Channel and iSCSI storage.  Latency analysis  of NFS traffic is not possible with esxtop.

Since ESX 3.5, VMware has provided a tool specifically for profiling  storage: vscsiStats.  vscsiStats collects and reports counters on  storage activity.  Its data is collected at the virtual SCSI device  level in the kernel.  This means that results are reported per VMDK (or  RDM) irrespective of the underlying storage protocol.  The following  data are reported in histogram form:

  • IO size
  • Seek distance
  •  Outstanding IOs
  • Latency (in microseconds)
  • More!”

The following is a quick step guide to vscsiStats.

  • Login on a ESXi host as user with root privileges.
  • Want to monitor 1 VM? Determine VM worldgroupid with:
    # vscsiStats –l

Figure 7

  • Start collecting for one VM:
    # vscsiStats –s –w <vmwgid>

Figure 8 – Colllecting data for VC5

Figure 9

  • Create new macro, Download the macro from here and copy en paste everything between:
    Sub Process_data()
    and
    End Function
    From the menu: Run Macro.
  • Interpreting the data?
    Go to: “Using vscsiStats for Storage Performance Analysis”,
    section “Using vscsiStats Results”.
  • Example: Good Write performance on my storage. Most write commands complete under 5 ms.

Figure 10

  • Ready.

Other references:

Use esxtop/resxtop to collect performance data

Official Documentation:
vSphere Monitoring and Performance Guide, Chapter 7 “Performance Monitoring Utilities: resxtop and esxtop”, page 45.

Summary:
See also the first section of this objective.

Other references:

  • Read this  excellent post on esxtop by Duncan Epping. It contains great information how to run esxtop in batch mode and present the data using tools like excel, perfmon and esxplot.

 

Given esxtop/resxtop output, identify relative performance data for capacity planning purposes

Official Documentation:
None.

Summary:
While interpreting esxtop statistics, two question arise:

  • Which performance counters are important and which are less important?
  • What are the thresholds on the important performance counters?

An excellent answer on both questions can be found on Duncan’s post on esxtop (I am very sorry for naming this post time after time)

In this post you see that performance comes to these four essential resources:

  • CPU
  • Memory
  • Disk
  • Networking

And the following metrics:

Display            Metric

CPU                %RDY

CPU                %CSTP

CPU                %SYS

CPU                %MLMTD

CPU                %SWPWT

MEM               MCTLSZ

MEM               SWCUR

MEM               SWR/s

MEM               SWW/s

MEM               CACHEUSD

MEM               ZIP/s

MEM               UNZIP/s

MEM               N%L

NETWORK    %DRPTX

NETWORK    %DRPRX

DISK              GAVG

DISK              DAVG

DISK              KAVG

DISK              QUED

DISK              ABRTS/s

DISK              RESETS/s

DISK              CONS/s

Other references:

3 thoughts on “VCAP5-DCA Objective 3.4 – Utilize advanced vSphere Performance Monitoring tools”

  1. Pingback: VCAP5-DCA Objective 6.2 – Troubleshoot CPU and memory performance « Adventures in a Virtual World
  2. Pingback: VCAP5-DCA Objective 6.3 -Troubleshoot Network Performance and Connectivity « Adventures in a Virtual World
  3. Pingback: VCAP5-DCA Objective 6.4 – Troubleshoot storage performance and connectivity « Adventures in a Virtual World

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.