Veni, vidi, VAAI
Recently, I was asked to investigate a performance issue on a VMware environment. Configuration of the ESXI hosts and storage was just finished and engineers were busy deploying Virtual Machines from a Template. The main issue was; “Actual time to deploy a VM is much more than expected”.
The Environment consists of the following components:
- Dell Equallogic PS4100X, with 24 x 300 GB 10K SAS drives in RAID-50 config;
- Storage switches, two Dell PowerConnect 6224 stacked;
- ESXi servers, three Dell R815, 32 GB and 6 NICs;
- VMware vSphere 5 Enterprise Edition.
First step was to check the configuration of the components. Configuration of the servers and storage was according to Dell’s “Configuring VMware vSphere Software iSCSI with Dell EqualLogic PS Series Storage”.
At first glance, everything seems to be OK, so now it is time to do some testing and see with my own eyes what is going on. I started the deployment of a template of a VM, configured with a 60 GB virtual disk on the same ESXI server, at the same time logged in on the console and started esxtop and switched to the “disk adapter” view. During the deployment, the iSCSI software adapter (vmhba40) showed a steady 60 MB/s read (MBREAS/s) and 60 MB/s write (MBWRTN/s). The Total Latency (GAVG/cmd) is 4.10 ms., which is fine (as Mr. Eric Sloof explained in an Advanced Troubleshooting presentation).
These performance figures did not seem too bad, and now?
Luckily, I had the opportunity to repeat this Deployment test at another location. The shared storage was also a Dell Equallogic 4100 series. It was surprising to see that the deployment took only a few minutes and esxtop showed this: (the iSCSI software adapter is vmhba37).
Remarkable, no data Reads and no Writes, just an extremely high latency, GAVG/cmd= 102…
Time for a cup of coffee. While taking a break, I remembered something about VAAI. A good introduction to VAAI is this post on the VMware communities. It is not 100% up-to-date, but contains useful information and some good links to other information.
Other recommended reading is the “vSphere Storage” guide, Chapter 18 “Storage Hardware Acceleration”. To be more precise, since vSphere version 5, VAAI has been renamed to “Storage APIs – Array Integration” and includes “Hardware Acceleration APIs” and “Array Thin Provisioning APIs”. All of these are part of a much larger family of Storage APIs.
In short, VAAI was the acronym for vStorage APIs for Array Integration and was introduced in vSphere 4.1. VAAI can offload specific storage operations to compliant storage hardware, which results in less CPU, memory and storage fabric bandwidth consumption. VAAI can enhance standard vSphere operations like:
- Storage vMotions;
- Provisioning a VM from a template;
- Using the Thin provisioning functionality
As said before, in vSphere 5, VAAI has been renamed and functionality has been extended to support NAS. There are of course a few conditions, VAAI requires vSphere Enterprise or Enterprise Plus licenses and compatible storage.
But does it work?
On a ESXI host, Hardware Acceleration for block storages devices is enabled by default. You can check with your vSphere Client. Go to Configuration, Storage (Hardware section), under Datastores, the last column shows the Hardware Acceleration Status. Three options: Supported, Not Supported and Unknow.
It is possible to disable the hardware acceleration functions. In the vSphere Client, go to the Configuration section, and enter the “Advanced Settings”. Check if any of these values were set to 0 (means disabled) :
You should also check if the required single VAAI filter and vendor specific plug-ins are loaded on a ESXi host. There are several ways to complete this action, with help of the vSphere CLI, the vMA or by opening a SSH session to an ESXi host (I do not recommend to leave the SSH service running while in production, but it is very useful during the implementation…):
# esxcli storage core plugin list --plugin-class=VAAI Plugin name Plugin class ------------- ------------ VMW_VAAIP_EQL VAAI # esxcli storage core plugin list --plugin-class=Filter Plugin name Plugin class ----------- ------------ VAAI_FILTER Filter
To verify the hardware acceleration support status of all devices on a host:
# esxcli storage core device list naa.64ed2a25171b27d689f6c4010000604d Display Name: EQLOGIC iSCSI Disk (naa.64ed2a25171b27d689f6c4010000604d) Has Settable Display Name: true Size: 30720 Device Type: Direct-Access Multipath Plugin: NMP Devfs Path: /vmfs/devices/disks/naa.64ed2a25171b27d689f6c4010000604d Vendor: EQLOGIC Model: 100E-00 Revision: 5.1 SCSI Level: 5 Is Pseudo: false Status: on Is RDM Capable: true Is Local: false Is Removable: false Is SSD: false Is Offline: false Is Perennially Reserved: false Thin Provisioning Status: yes Attached Filters: VAAI_FILTER VAAI Status: supported Other UIDs: vml.020000000064ed2a25171b27d689f6c4010000604d313030452d30
The important line here is: VAAI Status: supported
With this command, you will get the Hardware Acceleration Support Details:
# esxcli storage core device vaai status get naa.64ed2a25171b27d689f6c4010000604d VAAI Plugin Name: VMW_VAAIP_EQL ATS Status: supported Clone Status: supported Zero Status: supported Delete Status: supported mpx.vmhba32:C0:T0:L0 VAAI Plugin Name: ATS Status: unsupported Clone Status: unsupported Zero Status: unsupported Delete Status: unsupported
Each block storage device managed by a VAAI plug-in needs 2 claim rules, one that specifies the hardware acceleration filter and another that specifies the hardware acceleration plug-in for the device:
# esxcli storage core claimrule list --claimrule-class=Filter Rule Class Rule Class Type Plugin Matches ---------- ----- ------- ------ ----------- ---------------------------- Filter 65429 runtime vendor VAAI_FILTER vendor=MSFT model=Virtual HD Filter 65429 file vendor VAAI_FILTER vendor=MSFT model=Virtual HD Filter 65430 runtime vendor VAAI_FILTER vendor=EMC model=SYMMETRIX Filter 65430 file vendor VAAI_FILTER vendor=EMC model=SYMMETRIX Filter 65431 runtime vendor VAAI_FILTER vendor=DGC model=* Filter 65431 file vendor VAAI_FILTER vendor=DGC model=* Filter 65432 runtime vendor VAAI_FILTER vendor=EQLOGIC model=* Filter 65432 file vendor VAAI_FILTER vendor=EQLOGIC model=* And some more vendors… # esxcli storage core claimrule list --claimrule-class=VAAI Rule Class Rule Class Type Plugin Matches ---------- ----- ------- ------ ---------------- ---------------------------- VAAI 65429 runtime vendor VMW_VAAIP_MASK vendor=MSFT model=Virtual HD VAAI 65429 file vendor VMW_VAAIP_MASK vendor=MSFT model=Virtual HD VAAI 65430 runtime vendor VMW_VAAIP_SYMM vendor=EMC model=SYMMETRIX VAAI 65430 file vendor VMW_VAAIP_SYMM vendor=EMC model=SYMMETRIX VAAI 65431 runtime vendor VMW_VAAIP_CX vendor=DGC model=* VAAI 65431 file vendor VMW_VAAIP_CX vendor=DGC model=* VAAI 65432 runtime vendor VMW_VAAIP_EQL vendor=EQLOGIC model=* VAAI 65432 file vendor VMW_VAAIP_EQL vendor=EQLOGIC model=*
All this seems to be OK.
The last section of Chapter 18 of the “vSphere Storage Guide” is called “Hardware Acceleration Considerations”. Perhaps “Hardware Acceleration Limitations” is a better choice.
The VMFS data mover does not leverage hardware offloads and instead uses software data movement when one of the following occurs:
- The source and destination VMFS datastores have different block sizes.
- The source file type is RDM and the destination file type is non-RDM (regular file).
- The source VMDK type is Eager Zeroed Thick and the destination VMDK type is Thin.
- The source or destination VMDK is in sparse or hosted format.
- The source virtual machine has a snapshot.
- The logical address and transfer length in the requested operation are not aligned to the minimum alignment required by the storage device. All datastores created with the vSphere Client are aligned automatically.
- The VMFS has multiple LUNs or extents, and they are on different arrays.
- Hardware cloning between arrays, even within the same VMFS datastore, does not work.
This section is a nice checklist!
It is time to do some investigation on the Template VM. After converting the Template to a VM, the performance issue becomes clear. The VM has snapshots, so the VMFS data mover is not able to offload the deployment to the storage array. Also notice the disk provisioning type, in some cases this type can also cause problems.
After removing the snapshots, deployments are now finished within 1 minute instead of 20.
To conclude, this issue has learned me a few things about the Hardware Acceleration functionality. As always, I hope you enjoyed reading this troubleshooting session and I welcome your comments.
BTW, the title of this post was derived from the sentence “Veni, vidi, vici”, made famous by Julius Caesar, see this link for more information.