ESXi boot fatal error 33 inconsistent data


Another quick write-up. Recently, while installing patches and rebooting ESXi hosts, I encountered the following error message during the boot process of an ESXi host: “esxi boot fatal error 33 inconsistent data”, accompanied by the filename causing the inconsistency.

A quick search on the Internet returned various useful tips, ranging from re-installation of ESXi, re-running the installer or replacing the damaged file. However, not all workarounds are applicable in all situations.

Perhaps it is too obvious, but this solution is also to be considered, especially while updating hosts;
Revert the ESXi host to it’s previous version. Now the ESXi host can boot and the installation of the patches can be continued.

This VMware KB outlines how to revert an ESXi host to it’s previous state.

If during the update process of ESXi this error shows up: “The host returns esxupdate error code:15. The package manager transaction is not successful. Check the Update Manager log files and esxupdate log files for more details
Also the esxupdate.log shows “esxupdate: esxupdate: ERROR: InstallationError: (”, ‘There was an error checking file system on altbootbank, please see log for detail.’)”, it is time to have a look at this KB.

I hope this will help. Thank you for reading.

vCSA and trusted AD sources


Just a quick write up for my own convenience. Large organizations tend to have a lot of everything, from buildings and employees to Domain Controllers.
In times were Domain Controllers undergo maintenance, like an upgrade or relocation, dependent services may be impacted.
The way identity sources are configured differs per product, fortunately less often hard-coded by specifying a single domain controller, usually more flexible by specifying the AD domain.

For a vCenter Server Appliance (vCSA), additional identity sources can be configured, one commonly used is the Active Directory (Integrated Windows Authentication).


BTW, As a prerequisite, the vCSA should be joined to the Windows domain.

Read the rest of this entry »

vCSA, root partition is (almost) full


hwA short post on a topic that I recently experienced on vCenter Server Appliance, version 6.0.
After receiving an alert that the root “/” partition was quickly filling up, it is time to act quickly. When the root partition reaches 100% of it’s capacity, service disruption can occur.
First step is to check the capacity of the vCSA partitions. Log in to the vCSA through SSH, if you are running the appliance shell, enable and access the Bash shell:

Command> shell.set --enabled true
Command> shell

In the Bash shell run this command to check the capacity of the partitions:

# df -h

The second line of the output (starting with /dev/sda3) shows the status of the root partition. If the value under Use% reaches 100%, you are in trouble. Also notice that the root partition is only 11 GB.
Second step is to determine the root cause of the full partition. A good strategy is to look for large consumers. The next command searches for files larger then 100 MB, only on the root partition:

# find / -xdev -type f -size +100M

In my case some interesting results:


The most eye-catching files are: the wrapper.log and the dnsmasq.log files.

Read the rest of this entry »



I recently encountered an interesting question, maybe not the one you will see every day. A vCenter Center server runs a large number of Clusters; the VMs on those clusters are controlled by a considerable number of DRS rules. The question that raised; “How do we know if the DRS rules we once designed are still in place?” In the course of time, rules can be disabled, VM or Host groups does not match any more. Trying to answer this question by going through the vCenter Server configuration is not the way to go.

Thankfully, the VMware PowerCLI contains a useful Cmdlet Get-DrsRule that enables you to create a dump of the configured rules for each cluster. This makes checking your configuration a lot easier.

But there is another thing, now we know about the configuration, but what do we know about the actual situation? For instance, VM to Host affinity has “should” and “must” rules, but to what extent is a “should” rule fulfilled?

So time to create a PowerShell script which performs the following tasks; for each Cluster within a vCenter Server, a dump of the configured DRS rule is made. The second part of the script determines on which host a VM is running and compares it to the configured rules. The script will also report if a DRS rule is disabled and displays the power state of each VM. You will probably worry less about a powered down VM.

The script can be found here on GitHub.

I am aware that the script and my programming skills are far from perfect, so expect updated versions in the future.

vCSA default shell is BASH


A quick post about a little caveat while working in the vCenter Server Appliance (vCSA) shells. Yes correctly, shells in plural. The vCSA is bundled with at least two different shells:

  • Appliance Shell (default)
  • BASH shell

The “Appliance shell” is the default shell. After you log in to the vCSA, it will present the following well known screen.
Fig 1.

The appliance shell can be used for updating the vCSA, using the software-packages command and has some other use cases. From here you can enable the BASH shell as shown in the Fig 1. for the duration of your session with the following commands:

# shell.set --en -s /bin/bash root
# chsh -s /bin/bash root

You can also set the BASH shell as the default shell by performing the following command. Make sure, you first enable the BASH shell as shown above:

# chsh -s /bin/bash root

For the change to take effect, log out and log in again. Now you will directly enter the BASH shell.

But while working in the BASH shell, you need to temporarily switch to the Appliance shell?
In that case, provide the following command:

# appliancesh

That’s it. A shell is nothing more or less than an executable; the “Appliance shell” is no exception and can be found as /bin/appliancesh.

For more information, see: VMware KB “Toggling the vCenter Server Appliance 6.x default shell (2100508)

Best way to create vCSA support bundles


In general, during contact with a Customer Support team, whether being a Hardware vendor (Servers, Storage) or a Software vendor, the likelihood that you will be asked to upload some log files for further investigation is significantly.

In case of VMware vSphere, you will have multiple options for collecting a support bundle; to name a few:

  • Windows vSphere Client
  • vSphere Web Client
  • PowerShell scripts

See also VMware KB “Collecting diagnostic information for VMware vCenter Server 4.x, 5.x and 6.x (1011641)” for a complete overview.

Using some of these methods is very convenient; however there is a little caveat, when the vCenter Server is a vCenter Server Appliance (vCSA) 6.x.

Under certain conditions, a vCSA might contain core dump files. When requested to create a support bundle these core dump files will be added to the support bundle, together with the log files. The issue that may arise is that the location where the support bundle will be created (partition /storage/log) has a fixed size and possibly is too small.

Is that is the case, the creation of the support bundle will halt with error “Cannot create a diagnostic bundle” and the desired support bundle will not be created.

VMware recommendation is to create and download a support bundle using the web browser. To do so enter the following URL:
https ://<VCSA Hostname or IP address>/appliance/support-bundle

Fig 1

After providing the credentials, the support-bundle (filename is: vm-support.tgz) will start downloading. The progress of the process will be shown in the browser.

Using this method, the files will be directly downloaded to your local computer, instead of being prepared on the vCSA.

As always, I thank you for reading.

All Hands on Deck / VMware


201701-01Although VMware is best known for its core virtualization product vSphere, it may not come as a surprise that since their first product releases, a lot has changed. Over the years, on top of this solid foundation, VMware built a “Software Defined Everything” stack (SDDC), a Cloud stack with all the Automation and Orchestration tools, Monitoring tools, an End User Compute environment etc. This has led to a very large number of VMware products.
As you are as curious as I am, product
data sheets and documentation maybe helpful, but the best way in my humble opinion is to download, install and try the bytes yourself.
In addition to options already there, VMware launched a new one; Product Walkthroughs. So time for an overview how to get familiar with these fine products.

Read the rest of this entry »