Troubleshooting CIM on ESXi


Recently, a number of ESXi hosts were updated from version 6.0 to the latest 6.7 update. Soon after, we detected the following error message “An application (/bin/sfcbd) running on ESXi host has crashed (1 time(s) so far). A core file might have been created at /var/core/sfcb-vmware_bas-zdump.000.”. The core file was indeed created, luckily this was not a PSOD, the host was still up and running, workloads were not impacted. We also noticed that all upgraded hosts were impacted, it also became clear that after (re)booting a host, after about 24 hours the same event re-occurred, creating a new dump file.

After some digging around in the log files, searching for events at the time the dump file was created we found in the syslog.log:
“sfcb-vmware_base[2100157]: tool_mm_realloc_or_die: memory re-allocation failed(orig=400000 new=800000 msg=Cannot allocate memory, aborting”,
followed by: “sfcb-ProviderManager[2100151]: handleSigChld:166681408 provider terminated, pid=2100157, exit=0 signal=6”. This looks like some memory related issue.

As this is not an ideal situation, it was time to engage VMware support. Before we continue, some background; sfcbd stands for “Small Footprint CIM Broker (SFCB) daemon”. For performance and health monitoring ESXi enables an agent less approach using industry standards like CIM (Common Information Model) and WBEM (Web-Based Enterprise Management). At the ESXi side, there is the CIM agent, represented by the sfcbd. CIM providers are the counter part, often supplied by 3rd parties like hardware vendors. CIM providers come as .VIB files. After detecting 3rd party CIM provider, the sfcbd (with that the WBEM services) is automatically started by ESXi.

Read the rest of this entry »

What is the sharedPolicyRefCount?


Just a quick write-up for my own convenience.
Recently while working on a configuration management baseline for a vSphere environment, I stumbled on a particular advanced setting, present in ESXi. The setting is named config.globalsettings.guest.commands.sharedpolicyrefcount,
with description “Reference count to enable guest operations” and can have an integer value between 0 and 2147483647.

From its name I know it has something to do with the guest OS. A quick Google search did not reveal very useful information, in particular which value needs to be set (as I found “0” and “100” mentioned as preferred values).

From VMware Support, thank you Pranita Kumari, I learned that vRealize Infrastructure Navigator uses VMware tools to access the machines and configure the hosts and virtual machine for the discovery process. vRealize Infrastructure Navigator needs to set the ‘sharedpolicyrefcount’ parameter in order to do agent-less discovery.
If you don’t use vRealize Infrastructure Navigator ( as this product is end of distribution and GS), the best practice would be to set this option to default value 0.

That’s all, I thank you for reading.

Vester and DSC, a comparison


Over the past couple of months, I have published several posts about Configuration drift and tools like Vester and DSC Resources for VMware. Because Vester and DSC Resources for VMware serve the same goal, let us review what these tools have in common and see some of the differences.
Some topics; general information about the tool, configuration of the tool, the tool in daily operations, performance and a summary.


Both tools are built with PowerShell. Vester has been on the market for the longest time and dates from 2017. Vester comes as a PowerShell module and depends on two other modules; Pester and PowerCLI. Vester consists of three parts;

  • Commands that do the actual work, like creating configuration files, verifying the actual configuration and do remediation in case the actual configuration does not match the desired confguration.
  • Set of Test files. Each test file contains code that checks and applies a configuration item.
  • Config files, are key-value pairs with the desired values of the configuration items. Some examples: NTP settings, DNS servers, etc.

Desired State Configuration (DSC) was introduced in PowerShell 4 and brings a declarative model for the configuration of Windows Servers. DSC can copy files, edit the registry, install Windows features and components. After initial configuration, DSC can also test the desired configuration and if necessary perform remediation.
DSC Resources are what can be configured on a Windows server, but today not only on Windows Servers! DSC Resources for VMware was first released in December 2018. Instead of Windows servers, these resources can configure ESXi hosts and vCenter Servers, although the first edition had only a few resources. The second edition, released in June 2019 offered considerably more resources.
Both tools are available in the PowerShell Gallery and can be found in Github.

Read the rest of this entry »

Another Vester Test file generator and more vCenter checks


Some time after finishing the “Vester Test file generator”, I was wondering how to get more configuration settings out of a vCenter Server. Then I realized that vCenter Servers also contain a large number of advanced settings.




To get an overview of ALL Advanced Settings in vSphere, connect to a vCenter Server and run the following line:

PS> Get-AdvancedSetting -Entity *

In the output you will discover three large groups:
VIServer, vCenter Server settings
VMHost, ESXi host settings (see other Vester Generator)
VM, Virtual machine settings

And finally, two small groups “Compute Cluster DRS” (9 settings) and “Datastore Cluster” (3 settings).

Read the rest of this entry »

About VMSA, CVE, CVSS and more


A while back I was alerted to something remarkable regarding the VMware security bulletins. But let me first provide some background on these bulletins.

Regularly VMware publishes VMware Security Announcements, also known as VMSA’s.

Fig. 1 – Example VMSA

As VMware states; “VMware Security Advisories document remediation for security vulnerabilities that are reported in VMware products”. The bulletins are created by the VMware Security Response Center (VSRC). The VSRC works closely with customers and security researchers on the analysis and remediation of security issues within VMware products. After validating a report, the VSRC works with VMware R&D on providing solutions. Upon the remediation of an issue a VMSA will be released.
If you don’t receive notifications on the release of new VMSA’s, on this page (or here) you can sign up for the Security Advisories and fond more information.

The latest security advisories can be found on this page. A comprehensive overview over the past years can be found here.

Read the rest of this entry »

Securing DSC resources for VMware


Recently DSC Resources for VMware 2.0 was released. This new version comes with a lot of new resources and other features, like availability in the PowerShell Gallery. If DSC Resources for VMware is completely new,
I recommended reading the “Getting started” blog post, but do not follow the installation instructions. Instead install directly from the PowerShell Gallery, use something like this:

PS> Find-Module *VMware.vSphereDSC* | Install-Module

So after exploring “Vester”, the other DSC solution, it is now time to have a look at the DSC Resources for VMware 2.0.

Disclaimer: Windows PowerShell Desired State Configuration (from now on “DSC”) is often used for configuration management of Windows systems and as such is new to me. This post focuses on the use of DSC in a vSphere environment.

My setup;  I used an old Windows Server 2012R2 as a LCM. The vSphere environment is a VCSA version 6.5 and two ESXi hosts.
This post contains links to some script. All files mentioned in this post can be downloaded from this location. Then on the LCM, create a new folder named C:\VMwareDSC and place all the files in this folder.

One of my first goals was to understand how to create a good configuration. Luckily, the VMware DSC module contains an example folder, and I selected the VMHost_Config.ps1 configuration, an sample script for configuring an ESXi host.

Read the rest of this entry »

Vester Test file generator


In previous posts (see below), I presented some tips for creating new Vester Test files. As you may know, ESXi hosts have a large number of so called “Advanced System Settings” Some of these settings are already present as Vester test files. These Advanced System Settings can be handled with the Get-AdvancedSetting and Set-AdvancedSetting cmdlets. With this knowledge and some PowerShell code, it is not to difficult to create a complete set (>1.100) of Vester Test files.

The New-VesterHostAdvanced.ps1 script can be found here.

A brief description how it works. After connecting to a vCenter Server, one of the available ESXi hosts needs to be selected. The selected host will be used to create an overview of all available Advanced System Settings.

Key in creating the scripts is the concept of Here documents, in PowerShell known as Here-String. See for a brief overview. Key in Here-Strings is the usage of single or double quotes with variables. A Here-String with double quotes allows the usage of variables. Run the following code to see the difference.

$var = 'MyValue'
$formatText1 = @"
Here-String with double quotes
The variable $var
Variable replacement


$formatText2 = @'
Here-String with single quotes
The variable $var
Test as-is

Read the rest of this entry »