Veni, vidi, VAAI

Veni, vidi, VAAI

Recently, I was asked to investigate a performance issue on a VMware environment. Configuration of the ESXI hosts and storage was just finished and engineers were busy deploying Virtual Machines from a Template. The main issue was; “Actual time to deploy a VM is much more than expected”.

The Environment consists of the following components:

• Dell Equallogic PS4100X, with 24 x 300 GB 10K SAS drives in RAID-50 config;
• Storage switches, two Dell PowerConnect 6224 stacked;
• ESXi servers, three Dell R815, 32 GB and 6 NICs;
• VMware vSphere 5 Enterprise Edition.

Veni

First step was to check the configuration of the components. Configuration of the servers and storage was according to Dell’s “Configuring VMware vSphere Software iSCSI with Dell EqualLogic PS Series Storage”.

At first glance, everything seems to be OK, so now it is time to do some testing and see with my own eyes what is going on. I started the deployment of a template of a VM, configured with a 60 GB virtual disk on the same ESXI server, at the same time logged in on the console and started esxtop and switched to the “disk adapter” view. During the deployment, the iSCSI software adapter (vmhba40) showed a steady 60 MB/s read (MBREAS/s) and 60 MB/s write (MBWRTN/s). The Total Latency (GAVG/cmd) is 4.10 ms., which is fine (as Mr. Eric Sloof explained in an Advanced Troubleshooting presentation).

The total time for this deployment was at least 15 minutes.

Vidi

These performance figures did not seem too bad, and now?
Luckily, I had the opportunity to repeat this Deployment test at another location. The shared storage was also a Dell Equallogic 4100 series. It was surprising to see that the deployment took only a few minutes and esxtop showed this: (the iSCSI software adapter is vmhba37).

Remarkable, no data Reads and no Writes, just an extremely high latency, GAVG/cmd= 102…

VAAI

Time for a cup of coffee. While taking a break, I remembered something about VAAI. A good introduction to VAAI is this post on the VMware communities. It is not 100% up-to-date, but contains useful information and some good links to other information.

Other recommended reading is the “vSphere Storage” guide, Chapter 18 “Storage Hardware Acceleration”. To be more precise, since vSphere version 5, VAAI has been renamed to “Storage APIs – Array Integration” and includes “Hardware Acceleration APIs” and “Array Thin Provisioning APIs”. All of these are part of a much larger family of Storage APIs.

In short, VAAI was the acronym for vStorage APIs for Array Integration and was introduced in vSphere 4.1. VAAI can offload specific storage operations to compliant storage hardware, which results in less CPU, memory and storage fabric bandwidth consumption. VAAI can enhance standard vSphere operations like:

• Storage vMotions;
• Provisioning a VM from a template;
• Using the Thin provisioning functionality

As said before, in vSphere 5, VAAI has been renamed and functionality has been extended to support NAS. There are of course a few conditions, VAAI requires vSphere Enterprise or Enterprise Plus licenses and compatible storage.

But does it work?

On a ESXI host, Hardware Acceleration for block storages devices is enabled by default. You can check with your vSphere Client. Go to Configuration, Storage (Hardware section), under Datastores, the last column shows the Hardware Acceleration Status. Three options: Supported, Not Supported and Unknow.

It is possible to disable the hardware acceleration functions. In the vSphere Client, go to the Configuration section, and enter the “Advanced Settings”. Check if any of these values were set to 0 (means disabled) :

• VMFS3.HardwareAcceleratedLocking
• DataMover.HardwareAcceleratedMove
• DataMover.HardwareAcceleratedInit

You should also check if the required single VAAI filter and vendor specific plug-ins are loaded on a ESXi host. There are several ways to complete this action, with help of the vSphere CLI, the vMA or by opening a SSH session to an ESXi host (I do not recommend to leave the SSH service running while in production, but it is very useful during the implementation…):

# esxcli storage core plugin list --plugin-class=VAAI
Plugin name    Plugin class
-------------  ------------
VMW_VAAIP_EQL  VAAI

# esxcli storage core plugin list --plugin-class=Filter
Plugin name  Plugin class
-----------  ------------
VAAI_FILTER  Filter

To verify the hardware acceleration support status of all devices on a host:

# esxcli storage core device list
naa.64ed2a25171b27d689f6c4010000604d
   Display Name: EQLOGIC iSCSI Disk (naa.64ed2a25171b27d689f6c4010000604d)
   Has Settable Display Name: true
   Size: 30720
   Device Type: Direct-Access
   Multipath Plugin: NMP
   Devfs Path: /vmfs/devices/disks/naa.64ed2a25171b27d689f6c4010000604d
   Vendor: EQLOGIC
   Model: 100E-00
   Revision: 5.1
   SCSI Level: 5
   Is Pseudo: false
   Status: on
   Is RDM Capable: true
   Is Local: false
   Is Removable: false
   Is SSD: false
   Is Offline: false
   Is Perennially Reserved: false
   Thin Provisioning Status: yes
   Attached Filters: VAAI_FILTER
   VAAI Status: supported
   Other UIDs: vml.020000000064ed2a25171b27d689f6c4010000604d313030452d30

The important line here is: VAAI Status: supported
With this command, you will get the Hardware Acceleration Support Details:

# esxcli storage core device vaai status get
naa.64ed2a25171b27d689f6c4010000604d
VAAI Plugin Name: VMW_VAAIP_EQL
ATS Status: supported
Clone Status: supported
Zero Status: supported
Delete Status: supported
mpx.vmhba32:C0:T0:L0
VAAI Plugin Name:
ATS Status: unsupported
Clone Status: unsupported
Zero Status: unsupported
Delete Status: unsupported

Each block storage device managed by a VAAI plug-in needs 2 claim rules, one that specifies the hardware acceleration filter and another that specifies the hardware acceleration plug-in for the device:

# esxcli storage core claimrule list --claimrule-class=Filter
Rule Class   Rule  Class    Type    Plugin       Matches
----------  -----  -------  ------  -----------  ----------------------------
Filter      65429  runtime  vendor  VAAI_FILTER  vendor=MSFT model=Virtual HD
Filter      65429  file     vendor  VAAI_FILTER  vendor=MSFT model=Virtual HD
Filter      65430  runtime  vendor  VAAI_FILTER  vendor=EMC model=SYMMETRIX
Filter      65430  file     vendor  VAAI_FILTER  vendor=EMC model=SYMMETRIX
Filter      65431  runtime  vendor  VAAI_FILTER  vendor=DGC model=*
Filter      65431  file     vendor  VAAI_FILTER  vendor=DGC model=*
Filter      65432  runtime  vendor  VAAI_FILTER  vendor=EQLOGIC model=*
Filter      65432  file     vendor  VAAI_FILTER  vendor=EQLOGIC model=*
And some more vendors…

# esxcli storage core claimrule list --claimrule-class=VAAI
Rule Class   Rule  Class    Type    Plugin            Matches
----------  -----  -------  ------  ----------------  ----------------------------
VAAI        65429  runtime  vendor  VMW_VAAIP_MASK    vendor=MSFT model=Virtual HD
VAAI        65429  file     vendor  VMW_VAAIP_MASK    vendor=MSFT model=Virtual HD
VAAI        65430  runtime  vendor  VMW_VAAIP_SYMM    vendor=EMC model=SYMMETRIX
VAAI        65430  file     vendor  VMW_VAAIP_SYMM    vendor=EMC model=SYMMETRIX
VAAI        65431  runtime  vendor  VMW_VAAIP_CX      vendor=DGC model=*
VAAI        65431  file     vendor  VMW_VAAIP_CX      vendor=DGC model=*
VAAI        65432  runtime  vendor  VMW_VAAIP_EQL     vendor=EQLOGIC model=*
VAAI        65432  file     vendor  VMW_VAAIP_EQL     vendor=EQLOGIC model=*

All this seems to be OK.

The last section of Chapter 18 of the “vSphere Storage Guide” is called “Hardware Acceleration Considerations”. Perhaps “Hardware Acceleration Limitations” is a better choice. 🙂

The VMFS data mover does not leverage hardware offloads and instead uses software data movement when one of the following occurs:

• The source and destination VMFS datastores have different block sizes.
• The source file type is RDM and the destination file type is non-RDM (regular file).
• The source VMDK type is Eager Zeroed Thick and the destination VMDK type is Thin.
• The source or destination VMDK is in sparse or hosted format.
• The source virtual machine has a snapshot.
• The logical address and transfer length in the requested operation are not aligned to the minimum alignment required by the storage device. All datastores created with the vSphere Client are aligned automatically.
• The VMFS has multiple LUNs or extents, and they are on different arrays.
• Hardware cloning between arrays, even within the same VMFS datastore, does not work.

This section is a nice checklist!
It is time to do some investigation on the Template VM. After converting the Template to a VM, the performance issue becomes clear. The VM has snapshots, so the VMFS data mover is not able to offload the deployment to the storage array. Also notice the disk provisioning type, in some cases this type can also cause problems.

After removing the snapshots, deployments are now finished within 1 minute instead of 20.

To conclude, this issue has learned me a few things about the Hardware Acceleration functionality. As always, I hope you enjoyed reading this troubleshooting session and I welcome your comments.

BTW, the title of this post was derived from the sentence “Veni, vidi, vici”, made famous by Julius Caesar, see this link for more information.