Troubleshooting CIM on ESXi


Recently, a number of ESXi hosts were updated from version 6.0 to the latest 6.7 update. Soon after, we detected the following error message “An application (/bin/sfcbd) running on ESXi host has crashed (1 time(s) so far). A core file might have been created at /var/core/sfcb-vmware_bas-zdump.000.”. The core file was indeed created, luckily this was not a PSOD, the host was still up and running, workloads were not impacted. We also noticed that all upgraded hosts were impacted, it also became clear that after (re)booting a host, after about 24 hours the same event re-occurred, creating a new dump file.

After some digging around in the log files, searching for events at the time the dump file was created we found in the syslog.log:
“sfcb-vmware_base[2100157]: tool_mm_realloc_or_die: memory re-allocation failed(orig=400000 new=800000 msg=Cannot allocate memory, aborting”,
followed by: “sfcb-ProviderManager[2100151]: handleSigChld:166681408 provider terminated, pid=2100157, exit=0 signal=6”. This looks like some memory related issue.

As this is not an ideal situation, it was time to engage VMware support. Before we continue, some background; sfcbd stands for “Small Footprint CIM Broker (SFCB) daemon”. For performance and health monitoring ESXi enables an agent less approach using industry standards like CIM (Common Information Model) and WBEM (Web-Based Enterprise Management). At the ESXi side, there is the CIM agent, represented by the sfcbd. CIM providers are the counter part, often supplied by 3rd parties like hardware vendors. CIM providers come as .VIB files. After detecting 3rd party CIM provider, the sfcbd (with that the WBEM services) is automatically started by ESXi.

Read the rest of this entry »

Hardware, firmware and drivers (2nd edition)


Because the subject is still actual, I created an updated version of this post from 2015.

A modern server is besides our favorite ESXi hypervisor loaded with all kinds of additional software, like the BIOS, firmware and drivers for items like; Baseboard management , Remote support interfaces, Storage controllers, NICs, Power Supplies, to name a few.

Some vendors provide ISO images or repositories containing the actual updates, you may run the update process and voila, ready and done.
If you want to stay in control and want some more insight in this subject, please read on.

It comes down to these four questions:

  • What hardware is in the server?
  • How to determine the current firmware and or driver?
  • Which driver and or firmware do I need?
  • How do I upgrade drivers and firmware?

Read the rest of this entry »

About Long Fat Networks and TCP tuning


Recently I came about a data communications subject that was pretty unknown to me, known as the Bandwidth-delay product. Knowledge about this can help you to recognize certain network issues and ways to resolve them. It is all about two Linux hosts, a source and a destination host, communicating with each other over a high capacity network link. The question is how can you, given this scenario, reach maximum throughput over the network?


The first step is to determine the Bandwidth-delay product for this network. Bandwidth-delay product (BDP) is defined as the product of a data link’s capacity (in bits per second) and its round-trip delay time (in seconds). The result, the amount of data (in bits or bytes), is the maximum amount of data on the network at any given time, that is data that has been transmitted but not yet acknowledged.
Why is this important? The TCP protocol is designed for reliable transmission of data, acknowledgements are an essential part of the protocol. A high BDP value has impact on the efficiency of TCP, because the protocol can only achieve optimum throughput if a sender sends a sufficiently large quantity of data before being required to stop and wait until a confirming message (acknowledgement) is received from the receiver, acknowledging successful receipt of that data.

Read the rest of this entry »

vCSA how to disable IPv6?


For me it was already a common practice to disable IPv6 on ESXi hosts, but until recently I did not realize that vCenter Server can also benefit from it. For vCenter Server on Windows, you reconfigure the Windows network configuration. But how do you disable IPv6 on the vCSA?

I recently found that a vCSA 6.0 has at least three options to reconfigure the network settings. But only one option enables you to disable IPv6.

Using a web browser you can log in to the vCSA Web console by entering URL: https: //<vCSA hostname or IP address>:5480

From there go to Networking, under Networking Interfaces, choose Edit to open the “Edit IP Configuration” window. Here you can configure IPv4 and IPv6 and disable IPv4, but no option to disable IPv6.


Read the rest of this entry »

Do you need to know Network Virtualization ?


Why NV?

20170304-01I recently took the VMware Certified Professional 6 Network Virtualization Exam. Preparation for a technical exam like one of the available VCP exams takes a lot of your free time, so why choose this one?
In recent years, I increasingly encountered the product NSX Manager, usually in VDI deployments with endpoint protection products like McAfee Move, Trend Micro Deep Security, to name a few. And while working on the upgrade of a VMware View environment, also comes the question, how to handle the endpoint protection part; How do we upgrade these components?
In the concept of the SDDC, besides the well known Compute and Memory providers, I consider Storage virtualization (like vSAN) and Network virtualization (NSX) as fundamental building blocks that should be part of your “basic” VMware knowledge.
I also noticed that VMware is doing a lot of promotion for the subject of micro-segmentation, and for a good reason.
So, I decided the time has come, to extend my knowledge. So where do you start? If you are on a VCP-DCV level, but cannot tell the difference between layer 2 and layer 3, I recommend start reading a book like “Networking for VMware Administrators” by Chris Wahl and Steve Pantol.
At that time, I was in between jobs with no budget, to attend regular VMware training courses like the “VMware NSX – Install, Configure and Manage” course.

Read the rest of this entry »

VCAP5-DCA Objective 2.4 – Administer vNetwork Distributed Switch settings



  • Understand the use of command line tools to configure appropriate vDS settings on an ESXi host
  • Determine use cases for and apply Port Binding settings
  • Configure Live Port Moving
  • Given a set of network requirements, identify the appropriate distributed switch technology to use
  • Configure and administer vSphere Network I/O Control
  • Use command line tools to troubleshoot and identify configuration items from an existing vDS

Understand the use of command line tools to configure appropriate vDS settings on an ESXi host

Official Documentation:
Good reading on the use of CLI tools on vSphere Networking is the vSphere Command-Line Interface Concepts and Examples document. Chapter 9 “Managing vSphere Networking”,  section “Setting Up vSphere Networking with vSphere Distributed Switch”, page 122.

The CLI commands available to configure a vDS are limited. The following actions should be performed using the vSphere Client:

  • create distributed switches
  • can add hosts
  • create distributed port groups
  • edit distributed switch properties and policies

However you can add and remove uplinks with use of the command: vicfg-vswitch or esxcfg-vswitch.

To Add an uplink port.

vicfg-vswitch  --add-dvp-uplink <vmnic>  --dvp <DVPort ID> <vDS>


vicfg-vswitch  -P <vmnic> -V <DVPort ID> <vDS>

Read the rest of this entry »

VCAP5-DCA Objective 2.3 – Deploy and maintain scalable virtual networking



  • Understand the NIC Teaming failover types and related physical network settings
  • Determine and apply Failover settings
  • Configure explicit failover to conform with VMware best practices
  • Configure port groups to properly isolate network traffic

Understand the NIC Teaming failover types and related physical network settings

Official Documentation:
vSphere Networking, Chapter 5 “Networking Policies”, Section “Load balancing and Failover policies”, page 43

Load Balancing and Failover policies determines how network traffic is distributed between adapters and how to reroute traffic in the event of an adapter failure.

The Load Balancing policy is one of the available Networking Policies, such as: VLAN, Security, Traffic Shaping Policy and so on.

The Failover and Load Balancing policies include three parameters:

  • Load Balancing policy: The Load Balancing policy determines how outgoing traffic is distributed among the network adapters assigned to a standard switch. Incoming traffic is controlled by the Load Balancing policy on the physical switch.
  • Failover Detection: Link Status/Beacon Probing
  • Network Adapter Order (Active/Standby)

Editing these policies for the vSS and vDS are done in two different locations within the Vsphere Client.

vSS, Host and Clusters, Configuration, Hardware, Networking. Select the desired vSS. “NIC teaming ” tab on the vSwitch level. Override on the Portgroup level.

Figure 1 vSS

Read the rest of this entry »

VCAP5-DCA Objective 2.2 – Configure and maintain VLANs, PVLANs and VLAN settings



  • Determine use cases for and configure VLAN Trunking
  • Determine use cases for and configure PVLANs
  • Use command line tools to troubleshoot and identify VLAN configurations

Determine use cases for and configure VLAN Trunking

Updated: 14-09-2012

Official Documentation:
vSphere Networking, Chapter 7 “Advanced Networking”, Section, “VLAN Configuration”, page 68.

On a vSS you can only configure one VLAN ID per Portgroup.

A vDS allows you to configure a range of VLAN IDs per portgroup. In fact there are four options for VLAN type on a vDS:

  1. None
    VLAN tagging will not be performed by this dvPort group
  2. VLAN
    Enter in a valid VLAN ID (1-4094).  The dvPort group will perform VLAN tagging using this VLAN ID
  3. VLAN Trunking
    Enter a range of VLANs you want to be trunked
  4. Private VLAN
    Select a private VLAN you want to use – the Private VLAN must be configured first under the dvSwitch settings prior to this option being configurable

Now you can join physical VLANs to virtual networks.

Remember these VLAN IDs:
VLAN 0 = None;
VLAN 1-4094 = Valid IDs;
VLAN 4095 = All IDs.

Ingress= vDS incoming traffic
Egress = vDS outgoing traffic

Configure VLAN trunking

By default a dvUplink Group is configured for all VLAN IDs.

Figure 1

Read the rest of this entry »