VCAP5-DCA Objective 6.4 – Troubleshoot storage performance and connectivity

Objectives

  • Use esxcli to troubleshoot multipathing and PSA-related issues
  • Use esxcli to troubleshoot VMkernel storage module configurations
  • Use esxcli to troubleshoot iSCSI related issues
  • Troubleshoot NFS mounting and permission issues
  • Use esxtop/resxtop and vscsiStats to identify storage performance issues
  • Configure and troubleshoot VMFS datastores using vmkfstools
  • Troubleshoot snapshot and resignaturing issues
  • Analyze log files to identify storage and multipathing problems

Use esxcli to troubleshoot multipathing and PSA-related issues

Official Documentation:
vSphere Command-Line Interface Concepts and Examples, Chapter 4 “Managing Storage”, section “Managing Paths”, page 42.

Summary:
Multipathing, PSA and the related commands have been discussed in Objective 1.3 “Configure and manage complex multipathing and PSA plugins”.

See also this post for an graphical overview of the ESXCLI command.

Other references:

  • A

Use esxcli to troubleshoot VMkernel storage module configurations

Official Documentation:
vSphere Storage Guide, Chapter 16 ”VMKernel and Storage”, page 149.

Summary:
I am not sure what to expect from this one. Have a look at this rather theoretical chapter. VMware presents a nice graphic that goes from a VM to the actual storage device drivers.

Figure 1 – Graphic by VMware

This graphic also indicates that every esxcli (namespace) storage command is part of this overview.

Get familiar with the esxcli command, practice and use the vSphere Command-Line Interface Concepts and Examples, Chapter 4 “Managing Storage” as a reference.

The esxcli system module namespace allows you to view load and enable VMKernel modules. To get an overview use this command:

# esxcli system module list
Name                 Is Loaded  Is Enabled
-------------------  ---------  ----------
vmkernel                  true        true
procfs                    true        true
vmkplexer                 true        true
vmklinux_9                true        true
vmklinux_9_2_0_0          true        true
random                    true        true

Other references:

  • A

Use esxcli to troubleshoot iSCSI related issues

Official Documentation:
vSphere Command-Line Interface Concepts and Examples, Chapter 5 “Managing iSCSI Storage”, page 53.

Summary:
Chapter 5 presents a nice overview of the esxcli commands to Setup iSCSI Storage and for listing and setting iSCSI options and Parameters.

Remember while troubleshooting iSCSI issues, iSCSI highly depends on IP technology, so also take in consideration issues like:

  • IP configuration of NICs
  • MTU settings on NIICs and switches
  • Configuration of vSwitches

So besides the esxci issci commands, you will also need the esxcli network command to troubleshoot network related issues.

Note: iSCSI Parameters options can be found on four different levels (Red in Figure 2).

Note: CHAP authentication options can be found on three levels (Blue in Figure 2):

  • Adapter level (General)
  • Discovery level
  • Target level

Figure 2

Other references:

Troubleshoot NFS mounting and permission issues

Official Documentation:
vSphere Command-Line Interface Concepts and Examples, Chapter 4 “Managing Storage”, section “Managing NFS/NAS Datastores”, page 48.

Summary:
The previous objectives point to the esxcli command, this one seems more general.

The esxcli has a name space on nfs: esxcli storage nfs

You can list, add and remove nfs storage.

Recommended reading on this objective is VMware KB “Troubleshooting connectivity issues to an NFS datastore on ESX/ESXi hosts”.

Other references:

Use esxtop/resxtop and vscsiStats to identify storage performance issues

Official Documentation:

Summary:
ESXTOP is very useful for troubleshooting storage performance issues.

Figure 3

Important metrics are:

  • DAVG; The latency seen at the device driver, usually caused by the disk array. Max. value is 25;
  • KAVG; Disk latency caused by the VMKernel. Max. value is 2.
  • GAVG; is the sum of DAVG + KAVG. Max. value is 25.

Also beware of iSCSI Reservation conflicts, counter CONS/s (SCSI Reservation Conflicts per second), Max. allowed is 20.

vscsiStats has been discussed in Objective 3.4.

Other references:

Configure and troubleshoot VMFS datastores using vmkfstools

Official Documentation:
vSphere Storage Guide, Chapter 22,”Using vmkfstools”, page 205.

Summary:
The official documentation is good reading on how to use the vmkfstools command:

~ # vmkfstools
No valid command specified

OPTIONS FOR FILE SYSTEMS:
vmkfstools -C --createfs [vmfs3|vmfs5]
               -b --blocksize #[mMkK]
 -              S --setfsname fsName
           -Z --spanfs span-partition
           -G --growfs grown-partition
   deviceName
           -P --queryfs -h --humanreadable
           -T --upgradevmfs
   vmfsPath

OPTIONS FOR VIRTUAL DISKS:

vmkfstools -c --createvirtualdisk #[gGmMkK]
               -d --diskformat [zeroedthick|
                                thin|
                                eagerzeroedthick]
               -a --adaptertype [buslogic|lsilogic|ide]
           -w --writezeros
           -j --inflatedisk
           -k --eagerzero
           -K --punchzero
           -U --deletevirtualdisk
           -E --renamevirtualdisk srcDisk
           -i --clonevirtualdisk srcDisk
               -d --diskformat [zeroedthick|
                                thin|
                                eagerzeroedthick|
                                rdm:<device>|rdmp:<device>|
                                2gbsparse]
               -N --avoidnativeclone
           -X --extendvirtualdisk #[gGmMkK]
               [-d --diskformat eagerzeroedthick]
           -M --migratevirtualdisk
           -r --createrdm /vmfs/devices/disks/...
           -q --queryrdm
           -z --createrdmpassthru /vmfs/devices/disks/...
           -v --verbose #
           -g --geometry
           -I --snapshotdisk srcDisk
           -x --fix [check|repair]
           -e --chainConsistent
    vmfsPath

OPTIONS FOR DEVICES:

           -L --lock [reserve|release|lunreset|targetreset|busreset|readkeys|readresv]
    /vmfs/devices/disks/...
           -B --breaklock /vmfs/devices/disks/...

The vmkfstools command without options presents a comprehensive overview. From here we can see the tree main options:

  • For File systems;
  • For Virtual disks;
  • For Devices;

The File Systems option allows you to:

  • List attributes of a VMFS file system;
~ # vmkfstools   -P /vmfs/volumes/IX2-iSCSI-01 -h
VMFS-3.54 file system spanning 1 partitions.
File system label (if any): IX2-iSCSI-01
Mode: public
Capacity 299.8 GB, 216.8 GB available, file block size 8 MB
UUID: 4f9eca2e-3a28f563-c184-001b2181d256
Partitions spanned (on "lvm"):
        naa.5000144f77827768:1
Is Native Snapshot Capable: NO

~ #

  • Create a VMFS file system;
  • Extend an existing VMFS file system;
  • Upgrading a VMFS datastore.

The Virtual Disks options are huge, you can:

  • Create virtual disks, use option: -c
  • Delete virtual disks, use option: -U
  • Initialize a virtual disk, use option: -w
  • Inflate a Thin disk, use option: -j
  • Remove Zeroed Blocks, use option: -K
  • Convert a Zeroedthick to an Eagerzeroedthick virtual disk, option: -k
  • Rename a virtual disk, use option: -E
  • Clone a virtual disk or RDM, use option: -i
  • And many more

Two important Device options are available:

  • Option –L –lock [reserve|release|lunreset|targetreset|busreset], lets you reserve a SCSI LUN for exclusive use by the ESXi host, release a reservation so that other hosts can access the LUN, and reset a reservation, forcing all reservations from the target to be released.
  • Option –B –breaklock, allows you to forcibly break the device lock on a particular partition

Other references:

Troubleshoot snapshot and resignaturing issues

Official Documentation:

Summary:
Resignaturing has been discussed in Objective 1.1.

There is also a CLI utility: esxcfg-volume to support resignaturing operations.

esxcfg-volume
-l|--list               List all volumes which have been
                        detected as snapshots/replicas.

-m|--mount              Mount a snapshot/replica volume, if
                        its original copy is not online.

-u|--umount             Umount a snapshot/replica volume.
-r|--resignature        Resignature a snapshot/replica volume.

-M|--persistent-mount   Mount a snapshot/replica volume
                        persistently, if its original copy is
                        not online.

-U|--upgrade            Upgrade a VMFS3 volume to VMFS5.

-h|--help               Show this message.

/vmfs/volumes #

The esxcli storage vmfs snaphot command has the same functionality.

Other references:

  • a

Analyze log files to identify storage and multipathing problems

Official Documentation:

Summary:
See also Objective 6.1 on analyzing Log files.

Other references:

  • A
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: