Home lab refresh


Since 2010, my VMware home lab was running on two servers; a HP ProLiant ML 110 G5 and a ML 110 G6 . First the G5 was taken out of active duty because of its 8 GB memory limit. Fortunately, it was possible to upgrade the memory of the G6 from the supported 16 GB to 32 GB, so the G6 remained usable quite some time, for labs with a vCenter Server and 3 virtual ESXi hosts. Recently, it became too tedious to run the latest vSphere editions.

A home lab is a valuable resource for various reasons. When preparing for a VMware exam, like the Datacenter VCP, you can practice installation and configuration of ESXi, vCenter Server, but also other tools like NSX, vROPS or LogInsight. A home lab is also useful for investigations which cannot be done at work in a production environment, to practice changes or upgrades and last but not least break and fix (one of my favorite use cases and highly educational).

If you want to practice with vSphere and other products, there are several options, which mainly depend on available budget, but also on other factors. The possibilities vary from a lab-in-the-cloud such as VMware Hands On Labs to VMware Workstation or a 19-inch rack filled with servers and switches. In my situation, decisive factors were limited space (I live in an apartment) low noise production and low energy consumption and the requirement to run a nested ESXi cluster with tools like LogInsight and vROPS. For a full vSphere 7 plus Kubernetes lab, however, a reasonable amount of hardware is required!

The old and the new, small but powerful

After some searching on the Internet you will soon come across the Intel NUCs, although not mentioned on the official VMware HCL, beloved by the community, see here and here.

Intel NUCs currently support 64GB of memory. The tenth generation is besides an i3, available in an i5 (4 cores) and an i7 (6 cores). My choice fell on the i5 (budget). Intel NUCs come with a processor, but without memory and disk(s), the final composition can be found on my Gear page.

The set-up of the Intel NUCs is not difficult, on the previously mentioned blogs of Virtuallyghetto.com and Virten.net you can find enough information for a successful installation.

The NUCs are installed with the latest ESXi 7.0 and are managed by a vCSA. To support the deployment of vSphere 6.7 and 7.0 labs, I use two Windows domain controllers (DNS and DHCP), a Windows scripting host and a pfSense firewall. For the deployment of the labs I gratefully use the nested ESXi appliances and the deployment scripts as provided by William Lam. With this a complete environment will be available in no time.

PowerShell Tips 1


As you probably know, PowerShell is built on .NET, to be more precise Windows PowerShell is built on the .NET Framework, where PowerShell Core is built on .NET Core.

When you work with PowerShell in many cases you won’t be very concerned about this fact, but in some cases you can’t ignore it.

The other day while working on a PowerCLI script to get and set the logforwarding for a vCenter Server Appliance (vCSA), see also this older post.
The “get” part worked well. To retrieve the hostname, the port and protocol of the forwarding log servers run the following line of code:

(Get-CisService -name 'com.vmware.appliance.logging.forwarding').get()

For the set part, I created:

$spec = New-Object PSObject -Property @{

(Get-CisService -name 'com.vmware.appliance.logging.forwarding').set($spec)

However this failed, creating the following error message:


Parameter 'cfg_list' expects values of type  'System.Collections.Generic.List`1[[System.Management.Automation.PSObject, 
System.Management.Automation, Version=, Culture=neutral, PublicKeyToken=31bf3856ad364e35]]' 
but received value of type 'System.Management.Automation.PSObject'.
At line:1 char:1
+ (Get-CisService -name 'com.vmware.appliance.logging.forwarding').set( ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (:) [], CisException
    + FullyQualifiedErrorId : VMware.VimAutomation.Cis.Core.Types.V1.CisException

From the documentation, it was already known to me that a vCSA supports a total of 3 log forwarding hosts – hence the ‘cfg_list’, but how to interpret this error message? The parameter ‘cfg_list’ must be of a certain type, but how to solve this. Luckily my colleague Bouke (you can see what is on his mind on https://www.jume.nl ), quickly showed me the solution by specifying the variable in the correct type.

The following piece of code does the ‘set’ job. The solution is in the first line; setting the correct type (variable $speclist) for the ‘cfg_list’ parameter.

$speclist = [System.Collections.Generic.List[PSobject]]::new()

$spec = New-Object PSObject -Property @{
$spec = New-Object PSObject -Property @{

(Get-CisService -name 'com.vmware.appliance.logging.forwarding').set($speclist)

As always, I thank you for reading.

The importance of good data / How to set-up a baseline document?


Lately I’ve been working on machine learning and more specifically the Python Scikit library.
What I especially learned from this is the need to have a good
data-set before you want to do any kind of analysis or prediction.

But what does that have to do with subjects I usually write about? In the past period I have blogged regularly about configuration drift and tools like Vester and DSC resources for VMware.
We are also working on this within the company where I work.
Recently the assignment came to set up a baseline for the vCenter Server Appliances – you can’t solve configuration drift without thinking about the desired values, so time for a baseline. Apparently this seems simple, a baseline is a finite list of key-value pairs with the setting on one side and the value on the other side. In practice this seems a bit more complicated. I have to add that this baseline is not meant for a single vCenter Server, but for quite a few.

To get started, after connecting to a vCenter Server, the following command produces an overview of all settings for that vCenter:

PS> Get-AdvancedSettig -Entity <vCSA FQDN or IP>

Next to the fields Name and Value, you will also get the Type (of the Value) and sometimes a brief Description. Since vSphere 6.5 and up, you can also collect many appliance related settings using the API.
Now you can think, of all vCenters, collect the settings, set the desired values and done! In practice, however, there soon seemed to be some obstacles, such as:

  1. Not all vCenters are on the same version. Settings come and go. Some settings from vSphere 6.5 have disappeared in version 6.7, new settings have been introduced in version 6.7 and 7.0.
  2. Sometimes a setting exists, but returns an empty string. This is not equal to a setting that does not exist.
    Why worry about a setting with an empty string? What if, for whatever reason, a value does appear at any time?
  3. Not all settings are actually settings, but contain (status)information. We want to filter these out from our Configuration management tooling.

The baseline was created using PowerShell and the PowerCLI. The first step is to collect the settings of all vCenters as described above. The result is a .csv file for each vCenter. Incorporate the name of the vCenter in the filename like “vc01.csv”.

Read the rest of this entry »

Troubleshooting CIM on ESXi


UPDATE Oktober 2020:
1. Hidden in the comments is a link to VMware KB SFCB crashing and generating dumps: sfcb-vmware_bas-zdump (78046).
2. In the recently released VMware ESXi 6.7, Patch Release ESXi670-202008001, in the Resolved issues sections, you will find:
PR 2560686: The small footprint CIM broker (SFCB) fails while fetching the class information of third-party provider classes
SFCB fails due to a segmentation fault while querying with the getClass command third-party provider classes such as OSLS_InstCreation, OSLS_InstDeletion and OSLS_InstModification under the root/emc/host namespace“.
We are still in the process of evaluating this patch in test, so cannot tell if it really resolves the issue.

Recently, a number of ESXi hosts were updated from version 6.0 to the latest 6.7 update. Soon after, we detected the following error message “An application (/bin/sfcbd) running on ESXi host has crashed (1 time(s) so far). A core file might have been created at /var/core/sfcb-vmware_bas-zdump.000.”. The core file was indeed created, luckily this was not a PSOD, the host was still up and running, workloads were not impacted. We also noticed that all upgraded hosts were impacted, it also became clear that after (re)booting a host, after about 24 hours the same event re-occurred, creating a new dump file.

After some digging around in the log files, searching for events at the time the dump file was created we found in the syslog.log:
“sfcb-vmware_base[2100157]: tool_mm_realloc_or_die: memory re-allocation failed(orig=400000 new=800000 msg=Cannot allocate memory, aborting”,
followed by: “sfcb-ProviderManager[2100151]: handleSigChld:166681408 provider terminated, pid=2100157, exit=0 signal=6”. This looks like some memory related issue.

As this is not an ideal situation, it was time to engage VMware support. Before we continue, some background; sfcbd stands for “Small Footprint CIM Broker (SFCB) daemon”. For performance and health monitoring ESXi enables an agent less approach using industry standards like CIM (Common Information Model) and WBEM (Web-Based Enterprise Management). At the ESXi side, there is the CIM agent, represented by the sfcbd. CIM providers are the counter part, often supplied by 3rd parties like hardware vendors. CIM providers come as .VIB files. After detecting 3rd party CIM provider, the sfcbd (with that the WBEM services) is automatically started by ESXi.

Read the rest of this entry »

What is the sharedPolicyRefCount?


Just a quick write-up for my own convenience.
Recently while working on a configuration management baseline for a vSphere environment, I stumbled on a particular advanced setting, present in ESXi. The setting is named config.globalsettings.guest.commands.sharedpolicyrefcount,
with description “Reference count to enable guest operations” and can have an integer value between 0 and 2147483647.

From its name I know it has something to do with the guest OS. A quick Google search did not reveal very useful information, in particular which value needs to be set (as I found “0” and “100” mentioned as preferred values).

From VMware Support, thank you Pranita Kumari, I learned that vRealize Infrastructure Navigator uses VMware tools to access the machines and configure the hosts and virtual machine for the discovery process. vRealize Infrastructure Navigator needs to set the ‘sharedpolicyrefcount’ parameter in order to do agent-less discovery.
If you don’t use vRealize Infrastructure Navigator ( as this product is end of distribution and GS), the best practice would be to set this option to default value 0.

That’s all, I thank you for reading.

Vester and DSC, a comparison


Over the past couple of months, I have published several posts about Configuration drift and tools like Vester and DSC Resources for VMware. Because Vester and DSC Resources for VMware serve the same goal, let us review what these tools have in common and see some of the differences.
Some topics; general information about the tool, configuration of the tool, the tool in daily operations, performance and a summary.


Both tools are built with PowerShell. Vester has been on the market for the longest time and dates from 2017. Vester comes as a PowerShell module and depends on two other modules; Pester and PowerCLI. Vester consists of three parts;

  • Commands that do the actual work, like creating configuration files, verifying the actual configuration and do remediation in case the actual configuration does not match the desired confguration.
  • Set of Test files. Each test file contains code that checks and applies a configuration item.
  • Config files, are key-value pairs with the desired values of the configuration items. Some examples: NTP settings, DNS servers, etc.

Desired State Configuration (DSC) was introduced in PowerShell 4 and brings a declarative model for the configuration of Windows Servers. DSC can copy files, edit the registry, install Windows features and components. After initial configuration, DSC can also test the desired configuration and if necessary perform remediation.
DSC Resources are what can be configured on a Windows server, but today not only on Windows Servers! DSC Resources for VMware was first released in December 2018. Instead of Windows servers, these resources can configure ESXi hosts and vCenter Servers, although the first edition had only a few resources. The second edition, released in June 2019 offered considerably more resources.
Both tools are available in the PowerShell Gallery and can be found in Github.

Read the rest of this entry »

Another Vester Test file generator and more vCenter checks


Some time after finishing the “Vester Test file generator”, I was wondering how to get more configuration settings out of a vCenter Server. Then I realized that vCenter Servers also contain a large number of advanced settings.

To get an overview of ALL Advanced Settings in vSphere, connect to a vCenter Server and run the following line:

PS> Get-AdvancedSetting -Entity *

In the output you will discover three large groups:
VIServer, vCenter Server settings
VMHost, ESXi host settings (see other Vester Generator)
VM, Virtual machine settings

And finally, two small groups “Compute Cluster DRS” (9 settings) and “Datastore Cluster” (3 settings).

Read the rest of this entry »

About VMSA, CVE, CVSS and more


A while back I was alerted to something remarkable regarding the VMware security bulletins. But let me first provide some background on these bulletins.

Regularly VMware publishes VMware Security Announcements, also known as VMSA’s.

Fig. 1 – Example VMSA

As VMware states; “VMware Security Advisories document remediation for security vulnerabilities that are reported in VMware products”. The bulletins are created by the VMware Security Response Center (VSRC). The VSRC works closely with customers and security researchers on the analysis and remediation of security issues within VMware products. After validating a report, the VSRC works with VMware R&D on providing solutions. Upon the remediation of an issue a VMSA will be released.
If you don’t receive notifications on the release of new VMSA’s, on this page (or here) you can sign up for the Security Advisories and fond more information.

The latest security advisories can be found on this page. A comprehensive overview over the past years can be found here.

Read the rest of this entry »

Securing DSC resources for VMware


Recently DSC Resources for VMware 2.0 was released. This new version comes with a lot of new resources and other features, like availability in the PowerShell Gallery. If DSC Resources for VMware is completely new,
I recommended reading the “Getting started” blog post, but do not follow the installation instructions. Instead install directly from the PowerShell Gallery, use something like this:

PS> Find-Module *VMware.vSphereDSC* | Install-Module

So after exploring “Vester”, the other DSC solution, it is now time to have a look at the DSC Resources for VMware 2.0.

Disclaimer: Windows PowerShell Desired State Configuration (from now on “DSC”) is often used for configuration management of Windows systems and as such is new to me. This post focuses on the use of DSC in a vSphere environment.

My setup;  I used an old Windows Server 2012R2 as a LCM. The vSphere environment is a VCSA version 6.5 and two ESXi hosts.
This post contains links to some script. All files mentioned in this post can be downloaded from this location. Then on the LCM, create a new folder named C:\VMwareDSC and place all the files in this folder.

One of my first goals was to understand how to create a good configuration. Luckily, the VMware DSC module contains an example folder, and I selected the VMHost_Config.ps1 configuration, an sample script for configuring an ESXi host.

Read the rest of this entry »

Vester Test file generator


In previous posts (see below), I presented some tips for creating new Vester Test files. As you may know, ESXi hosts have a large number of so called “Advanced System Settings” Some of these settings are already present as Vester test files. These Advanced System Settings can be handled with the Get-AdvancedSetting and Set-AdvancedSetting cmdlets. With this knowledge and some PowerShell code, it is not to difficult to create a complete set (>1.100) of Vester Test files.

The New-VesterHostAdvanced.ps1 script can be found here.

A brief description how it works. After connecting to a vCenter Server, one of the available ESXi hosts needs to be selected. The selected host will be used to create an overview of all available Advanced System Settings.

Key in creating the scripts is the concept of Here documents, in PowerShell known as Here-String. See for a brief overview. Key in Here-Strings is the usage of single or double quotes with variables. A Here-String with double quotes allows the usage of variables. Run the following code to see the difference.

$var = 'MyValue'
$formatText1 = @"
Here-String with double quotes
The variable $var
Variable replacement


$formatText2 = @'
Here-String with single quotes
The variable $var
Test as-is

Read the rest of this entry »