ESXi and USB failure?

Updated 2011-03-06

Last week, my former colleague Gabrie van Zanten wrote a very interesting article about using USB sticks to boot an ESXi host.

After reading this excellent post, I remembered that in the early ESX 2.5 days, it was (and still is…) good practice to install ESX on a RAID-1 volume to keep your host in business in case of a disk failure. One day we received a first generation IBM Blade Center. Each blade has just one 2,5 inch disk and a small amount of memory. It was decided to install ESX (ESXi did not even exist…) and to get some experience  with blades. The blades suffered several disk crashes, which immediately led to host failure and disk replacement…

In recent years, servers with embedded USB storage have become common practice. Today, all major hardware vendors deliver servers with embedded ESXi. Even in my home lab, servers are equipped with an onboard USB connector, USB stick and ESXi. Recently, on one host, the USB stick was moved to an external connector. So after watching an episode of Myth Busters, I was wondering, what would happen with an ESXi host with USB stick failure. Or even worse, pulling the USB stick.

So, after booting up my 2 node cluster, I made a fresh backup of a few important VMs and checked the vCenter Service Status. Now, it is time to remove the USB stick from one host. And this is what happened:

  • VMs on the affected host are still running.
  • Task & Events of the affected host shows this message “Lost connectivity to storage device mpx.vmhba32:C0:T0:L0. Path vmhba32:C0:T0:L0 is down. Affected datastores: “Hypervisor1”, “Hypervisor2”, “Hypervisor3”.”.
  • Followed by 3 Alarms “Cannot connect to storage”.
  • Another message in Tasks & Events is “Boot partition /bootbank cannot be found (0:02:33:03.304 cpu1:30722)”.
  • Time for some testing, all these actions do work: Power On a VM, Migrate a VM, host in Maintenance Mode, Exit Maintenance Mode (HA Agent is configured correctly).
  • Also the ESXi console is doing fine, System Customization is in place, and so are the System Logs.
  • From time to time above messages are repeated and in some occasions while migrating VMs “The Operation is not allowed in the current state” messages are received.
  • After 24 hours, the host is still running, and performing. So finally, I decided to enter the host in Maintenance Mode and shut it down. The power down took about 10 minutes ( less then 2 minutes is normal).
  • After insertion of the USB stick, the host was powered on and was automatically reconnected to the cluster.

At this time, my tentative conclusion is that failure, or even an missing USB stick does not have much impact on a ESXi host. Thanks for reading and I’m very interested in your experience and opinions concerning this subject.

P.S. A few days after posting, I stumbled onto this post, written by Alan Renouf. In the first part it is explained why ESXi keeps running without USB boot device.

About these ads

11 Responses to ESXi and USB failure?

  1. Gabrie says:

    Hi

    Nice tests you performed and thanks for referring to my post. I was also wondering what would happen if you change a configuration setting. Like add a vSwitch or PortGroup.

    Gabrie

    • paulgrevink says:

      Hello Gabrie,

      Thanks for the feedback and a good question. I will perform some additional testing this week.

      Regards, Paul

    • paulgrevink says:

      Changing the configuration is possible (example adding a vSwitch). As expected changes are lost after reboot. I also added a new link that refers to an interesting post by Alan Renouf about this subject.

  2. Gabrie says:

    Good to know :-)

  3. Michael says:

    Hello,
    I have to deploy one ESXi host on DL380G7.
    Read a lot about installation on USB key and installed it today just for fun. It run perfectly…

    Couple of questions about usb key.

    1. Cannot find VMWare recommended USB key brand and exact model. Please recommend. I read about Sundisk and did my test installation on 4GB Sundisk key (first available). But would like to have the exact model.

    2. Your tests encourage to use USB key. What will be the procedure for creating a backup USB key in case of failure of Main one.

    Thanks,
    Michael.

    • paulgrevink says:

      Hello Michael,

      Thanks for the feedback.
      VMware published KB 1010574
      VMware supports installation of ESXi 4.x on USB when thes conditions are met:
      * The server on which you want to install ESXi 4.x is on the ESXi 4.x Hardware Compatibility Guide.

      AND

      * You have purchased a server with ESXi 4.x Embedded on the server from a certified vendor.

      OR

      * You have used a USB or SD flash device that is approved by the server vendor for the particular server model on which you want to install ESXi 4.x on a USB or SD flash storage device.

      In my own experience, a few years ago Dell sold PE2950 servers with Embedded ESXi 3.5 (in those days)
      The embedded ESXi was a Kingston USB stick on the mobo.
      In your case, which brand / model USB device is supported by HP.Assuming that the DL380G7 is on the HCL

      It is possible to create a backup of the configuartion of an ESXi host and restore it to the same host.
      The command that does the trick is: vicfg-cfgbackup.pl
      This command is part of the vSphere CLI, see http://www.vmware.com/support/developer/vcli/

      This excellent post on vmwaretips.com exactly answers your question.

      Regards,

      Paul

  4. KazO says:

    I don’t know if this was attributable to 4.x vs 3.5, but I’ve experienced several USB failures under 3.5 (write failures, which haven’t repeated in 4.x, presumably due to better optimizations; i.e. less writes to flash). In all of those cases, the VMs continued to run, but virtually nothing else worked; no management connection to vcenter, unable/difficulty getting into TechSupport mode, etc.

    Another issue is that USB flash boots and patches ESXi far slower than disk.

  5. iwansetiawan says:

    i got my usb stolen, any idea to install ESXi + old configuration. it’s still running with ups (hope the power not suddenly down too long). sorry my poor english

  6. Truelis says:

    Today our UPS failed and our 4 server west down… One of those was an ESXi on USB…
    At the reboot the ESXi installation appeared corrupted! :-(
    I did an ESXi upgrade mounting the ISO via iLO (HP server far away from me) to solve the problem.
    My official restore procedure involve the use of vmProtect 7, but it was not able to identify the USB key as a valid boot device… I never did a test of the restore before… X-D
    Lesson learnd! :-)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 355 other followers

%d bloggers like this: