Wots … Uh the Driver?

A modern server is besides our favorite ESXi hypervisor loaded with all kinds of additional software, like BIOS, and firmware and drivers for items like; Baseboard management , Remote support interfaces, Storage controllers, NICs, Power Supplies, to name a few.
If trouble arises and you must contact your hardware vendor or VMware support, in many situations you are kindly requested to install the latest updates, firmware versions and drivers.

Some vendors provide ISO images or repositories containing the actual updates, you may run the update process and voilà, ready and done.
If you want to stay in control and want some more insight in this subject, please read on.

It comes down to these four questions:

  • What hardware is in the server?
  • How to determine the current firmware and or driver?
  • Which driver and or firmware do I need?
  • How do I upgrade drivers and firmware?

What hardware is in the server?

Two examples, first a storage controller, second a NIC.

Storage Controller.

The command to list all available storage controllers is:

/tmp # esxcfg-scsidevs -a
vmhba38 iscsi_vmk         online   iscsi.vmhba38                           iSCSI Software Adapter
vmhba0 ahci             link-n/a sata.vmhba0                             (0:0:31.2) Intel Corporation Ibex Peak 6 port SATA AHCI Controller
vmhba32 usb-storage       link-n/a usb.vmhba32                             () USB
/tmp #

We focus on vmhba0, it’s an onboard SATA AHCI controller. The second column provides information which driver this SATA controller uses, in this example an ‘ahci’ type driver.

As many IO devices like storage controllers and NICs are PCI devices, each PCI device can be identified by an unique code consisting of four 16-bit ID’s; the Device ID (DID), the Vendor ID (VID), the Subsystem ID (SSID) and the Subsystem Vendor ID (SVID). To determine these ID’s use the following command: vmkchdev –l or vmkchdev –l | grep vmhba0 to search for a specific device

~ # vmkchdev -l | grep vmhba0
0000:00:1f.2 <strong>8086:3b22 103c:3118</strong> vmkernel vmhba0
~ #

In this example vmhba0 can be identified as:
VID        : 8086
DID        : 3b22
SVID       : 103c
SSID       : 3118

Physical NICs

The command to list all available NICs is: esxcli network nic list

~ # esxcli network nic list
Name   PCI Device     Driver Link Speed Duplex MAC Address         MTU Description
------ ------------- ------ ---- ----- ------ ----------------- ---- -------------------------------------------------------
vmnic0 0000:010:00.0 e1000e Up     1000 Full   00:1b:21:82:9f:d4 1500 Intel Corporation 82574L Gigabit Network Connection
vmnic1 0000:01e:00.0 tg3    Up     1000 Full   1c:c1:de:0f:9a:eb 1500 Broadcom Corporation NetXtreme BCM5723 Gigabit Ethernet
vmnic2 0000:020:00.0 e1000e Up     1000 Full   68:05:ca:19:a0:d5 1500 Intel Corporation 82574L Gigabit Network Connection
~ #

To determine driver and firmware of a particular NIC, e.g. vmnic0:

~ # <strong>esxcli network nic get -n vmnic0</strong>
Advertised Auto Negotiation: true
Advertised Link Modes: 10baseT/Half, 10baseT/Full, 100baseT/Half, 100baseT/Full, 1000baseT/Full
Auto Negotiation: true
Cable Type: Twisted Pair
Current Message Level: 1
Driver Info:
      Bus Info: 0000:10:00.0
      Driver: e1000e
      Firmware Version: 1.8-0
      Version: 1.1.2-NAPI
Link Detected: true
Link Status: Up
Name: vmnic0
PHYAddress: 1
Pause Autonegotiate: true
Pause RX: false
Pause TX: false
Supported Ports: TP
Supports Auto Negotiation: true
Supports Pause: true
Supports Wakeon: true
Transceiver: internal
Wakeon: MagicPacket(tm)
~ #

Now we can see, the driver is an “e1000e”, version 1.1.2 and the firmware is a version 1.8-0.

Using the command vmkchdev, we can determine the unique PCI device ID’s.

~ # vmkchdev -l | grep vmnic0
0000:10:00.0 8086:10d3 8086:a01f vmkernel vmnic0

For this NIC vmnic0:
VID        : 8086
DID        : 10d3
SVID       : 8086
SSID       : a01f

How to determine the current firmware and or driver

In the previous section, we already found some information regarding the SATA controller and the driver (ahci)

To find the actual driver, use the command: esxcli software vib list | grep ahci

~ # esxcli software vib list | grep ahci
sata-ahci                      3.0-18vmw.550.1.15.1623387           VMware VMwareCertified   2014-04-29
~ #

To find the actual driver for vmnic0, (driver ‘e1000e’) use this command:

~ # esxcli software vib list | grep e1000e
net-e1000e                     1.1.2-4vmw.550.1.15.1623387           VMware VMwareCertified   2014-04-29
~ #

We can see the driver is indeed a version 1.1.2.

Which driver and/or firmware do I need?

To find out about supported drivers, switch to the VMware Compatibility Guide and select IO Devices: http://www.vmware.com/resources/compatibility/search.php?deviceCategory=io

20151221-01Figure 1 – Compatibility Guide

In right hand side, enter the VID, DID, SVID and SSID values, for instance for vmnic0. After providing the information, switch to the lower side of the screen to watch the results. In this case:

20151221-02Figure 2

While hovering the mouse cursor over the Model, the PCI device ID’s show up. This device is supported in many releases. Select the link to get detailed information.

20151221-03Figure 3

Now you need to verify your current release, while on the CLI, use the command: vmware –v

~ # vmware -v
VMware ESXi 5.5.0 build-1746018

And use KB 1014505 “Correlating VMware product build numbers to update levels” to determine the correct release. Build 1746018 matches “ESXi 5.5 Update 1a””= ESXi 5.5 U1.

The correct device driver for this release should be version

By the way, drivers of type “VMware Inbox” have been written by VMware. The other category is type “Partner Async”. These drivers have been provided by third parties. In the driver name you will often find the letters “OEM”

Sometimes it gets a bit harder. Recently we had some issues with a NIC occasionally causing a PSOD. The advice was “first, update driver and firmware…”

After following the procedure described before, this was the outcome:

20151221-04Figure 4

Various device drivers are listed, but which one to choose? In the opinion of VMware Support, third party drivers are responsibility of the Vendors and you are kindly requested to go and see your vendor.

VMware KB 20130818 “Supported drivers’ firmware versions for I/O devices” will guide you. For instance, in the example above, the 533FLR-T NIC is a HP part. In KB 2030818, open the link to HPE, you will be directed to HPE’s Software Delivery Repository. Here you will find bulletins.

20151221-05Figure 5

Open the latest bulletin HPE-VMware-Recipe.pdf and search for the device:

20151221-06Figure 6

The first column shows the product, the second is driver type, the third is the ESXi version, the fourth column shows the current recommended driver and the final column shows the current recommended Firmware.
The HPE recommend driver 2.712.50.v60.6 matches the first listed driver in the VMware Compatibility Guide.
BTW, the HPE bulletin also provides direct links to the recommended drivers.

How do I upgrade drivers and firmware

The method of upgrading drivers and firmware depends on the Vendor and your support contract. For instance with HPE, if the hardware is eligible for support, you can download the latest HP Service Pack for ProLiant, this is an easy way to update firmware and drivers to the latest recommended versions. If you do not have support, you can search and download drivers and perform a manual installation.

Also VMware KB Installing async drivers on VMware ESXi 5.x and ESXi 6.0.x (2005205) provides a lot of useful information regarding the installation of async drivers in various scenario’s.

Some useful VMware KBs:

Determining Network/Storage firmware and driver version in ESXi/ESX 4.x, ESXi 5.x, and ESXi 6.x (1027206)

Determining which storage or network driver is actively being used on ESXi host (1034674)

Identifying correct driver for ESXi/ESX host PCI devices (HBA) using VMware Hardware Compatibility Guide (HCL) (1031534)

Supported drivers’ firmware versions for I/O devices (2030818)

Installing async drivers on VMware ESXi 5.x and ESXi 6.0.x (2005205)

As always, I thank you for reading.

The title of this post is inspired by this song.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: