Terraform and vSphere – Part 2: DSC

Desired State monitoring with Terraform?

Some time ago, I wrote a post about Terraform and vSphere. In this post, I showed how appliances (.ova files) can be deployed with Terraform. Another use case for Terraform is deploying virtual machines from templates.
But with this, we would almost forget the most important use of Terraform, deploying vSphere Infrastructure, in Terraform terminology resources like Clusters, vSwitches, Datastores and more. In doing so, I discovered an interesting feature of Terraform.
I’ve long been interested in configuration management for vSphere, see older posts on Vester and DSC Resources for VMware.

In a nutshell, Configuration Management is a systematic process for setting and maintaining the configuration of a resource over its lifetime.
In my experience, maintaining it during its lifetime is the trickiest part.
And that is where Terraform differs from other tools I have seen in recent years. As an example, we compare the configuration of a Compute Cluster in a vCenter Server using “DSC Resources for VMware” (see the example in this post) on the one hand and Terraform on the other.
A simple DSC configuration for creating a Cluster may look like this:

Configuration DCKoedood {
    Import-DscResource -ModuleName VMware.vSphereDSC -ModuleVersion 2.2.0.84

    vSphereNode $AllNodes.NodeName {

        Datacenter "DCKoedood" {
            Name = 'DCKoedood'
            Location = [string]::Empty
            Ensure = 'Present'
        }

        Cluster "Cluster-02" {
            Name = 'Cluster-02'
            Location = [string]::Empty
            DatacenterName = 'DCKoedood'
            DatacenterLocation = [string]::Empty
            Ensure = 'Present'
            HAEnabled = $true
            DrsEnabled = $true
            DependsOn = "[Datacenter]DCKoedood"
        }
   }
}

Continue reading

Log Insight 8.8.2 – API update

I wrote it in my previous postIn addition, Log Insight has an even larger number of APIs with “Tech Preview” status.”.
“Tech Preview” status also means in VMware’s words “Their design or implementation may change without warning and should not be depended on. A tech preview API may become supported, have forwards-incompatible changes or be removed during an upgrade.“.
And that is exactly what has happened with the release of Log Insight version 8.8.2 and the API endpoint for groups and roles.

By the way, have you also noticed that the Log Insight API has been a version 2 for some time now? To be exact, since the release of Log Insight version 8.6.0.

In version 8.8.2 endpoint /api/v2/user-groups replaces the /api/v2/authgroups. On the positive side, the way to assign roles to a group has been improved, as we will see later.

Using snippets of PowerShell code, basic operations according to the CRUD (Create, Read, Update and Delete) schema are shown for Directory groups and Roles.
First, the code for setting up a session with the Log Insight API, the code is written for PowerShell Core 7.x. The variable $vLISessionHeader is part of the Invoke-Restmethod cmdlet used in the other examples.

Continue reading

Log Insight – automating Groups and Roles

Important: The examples described in this blog are applicable for Log Insight version 8.8.0 and below, but no longer function in version 8.8.2 due to changes in the API. More on this in a forthcoming post.

Adding Directory Groups

A little while ago I wrote about a poc how to use Ansible and a pipeline to upgrade Log Insight. Shortly thereafter, I looked at the capabilities of vRealize Suite Lifecycle Manager (from now on: vRSLCM) and did deployments of Log Insight and vRealize Operations Manager. vRLSCM can take care of some of the configuration of Log Insight, such as NTP, DNS, authentication and Cluster VIPs, in addition to the deployment. Configuration of Directory Groups and Roles, also important, is (currently) not included. If you want to automate the whole process of deployment and configuration of Log Insight, additional action is needed.
In this post, I’ll show you how to configure groups and roles in Log Insight 8.8.0 with PowerShell 5.x but also with Ansible 2.12.6 (a better understanding of Ansible, is one of my goals for this year and nothing is better than practice).
The starting point is the documentation of the Log Insight REST API, available at link: https: //<fqdn Log Insight or IP address>/rest-api.
The APIs presented here do have a “Supported” status. In addition, Log Insight has an even larger number of APIs with “Tech Preview” status. The Tech Preview APIs are in most cases incompletely documented.
Available documentation can be found by including the word “internal” in the link: https: //<fqdn Log Insight or IP address>/internal/rest-api.
See also my post Log Insight REST API.

Continue reading

DSCR for VMware 2.2

Over the past few years I have devoted several posts to configuration management of vCenter Server and ESXi. At that time I also reviewed one of the first versions of DSC Resources for VMware. At the time, I was not undividedly enthusiastic, especially with regard to security aspects.

In February 2021 the latest version 2.2 was released and a lot has changed. Besides support for PowerShell 5.1 and 7.0, there is now also support for PowerShell Core on Linux.

The best improvement in my opinion is that the developers have made good use of the Invoke-DSCResource cmdlet introduced by Microsoft that allows DSC resources to be executed without having to use the PowerShell LCM engine. This eliminates the need for the Windows proxy server (also one of my objections). Cmdlet Invoke-DSCResource is part of the new module PSDesiredStateConfiguration.

Based on these new capabilities, VMware has made available the module Vmware.PSDesiredStateConfiguration. Looking at the contents of this module we see the following features:
Get-VmwDscConfiguration, New-VmwDscConfiguration, Start-VmwDscConfiguration and Test-VmwDscConfiguration. In these we recognize the three basic DSC functions: Test, Set (Start) and Get.

Another interesting enhancement, available only for PowerShell 7, is vSphereNode. vSphereNode is a keyword that represents a connection to a vCenter Server. A configuration can contain one or more vSphereNodes. The advantage, with a normal DSC Resource Server and Credential properties must be declared for each DSC, vSphereNode uses a connection set up with the familiar Connect-VIServer cmdlet to a vCenter Server. This, in my opinion, makes the configuration much more manageable. Here are examples of configuration with and without vSphere Nodes.

Continue reading

Log Insight REST API

I am currently working on some PowerShell scripts to verify the user and group permissions on VMware products. For vCenter Server, the PowerCLI provides cmdlets to do the job. However for vRealize Operations Manager (vROPS) and Log Insight this is not the case (Yes, vROPS has some cmdlets but not for getting permissions). Luckily, both products do include a REST API, so time to investigate.

PowerShell offers two commands to interact with REST API’s; Invoke-WebRequest and Invoke-RestMethod. After reading this post from Adam Bertram I decided to give Invoke-RestMethod a try, the next step was how to start?

Reading “Introduction to PowerShell Rest API Authentication” from Joshua Stenhouse, was very helpful. So after some practicing with his vROPS example, which can be found here, it was time to figure out how to setup authentication for Log Insight.

A good starting point is the API documentation. Besides documentation, the Log Insight GUI also provides access; In the upper-right corner, open the drop-down menu and select Help. On this page, you will find a link to the REST API Documentation.

The REST-API can also directly be accessed by this URL:
https://fqdnLogInsight/rest-api.

Continue reading

PowerShell Tips 1

As you probably know, PowerShell is built on .NET, to be more precise Windows PowerShell is built on the .NET Framework, where PowerShell Core is built on .NET Core.

When you work with PowerShell in many cases you won’t be very concerned about this fact, but in some cases you can’t ignore it.

The other day while working on a PowerCLI script to get and set the logforwarding for a vCenter Server Appliance (vCSA), see also this older post.
The “get” part worked well. To retrieve the hostname, the port and protocol of the forwarding log servers run the following line of code:

 
(Get-CisService -name 'com.vmware.appliance.logging.forwarding').get()

For the set part, I created:


$spec = New-Object PSObject -Property @{
	hostname="logger1.net"
	port=514
	protocol="UDP"
}

(Get-CisService -name 'com.vmware.appliance.logging.forwarding').set($spec)

However this failed, creating the following error message:

 

Parameter 'cfg_list' expects values of type  'System.Collections.Generic.List`1[[System.Management.Automation.PSObject, 
System.Management.Automation, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]]' 
but received value of type 'System.Management.Automation.PSObject'.
At line:1 char:1
+ (Get-CisService -name 'com.vmware.appliance.logging.forwarding').set( ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (:) [], CisException
    + FullyQualifiedErrorId : VMware.VimAutomation.Cis.Core.Types.V1.CisException

From the documentation, it was already known to me that a vCSA supports a total of 3 log forwarding hosts – hence the ‘cfg_list’, but how to interpret this error message? The parameter ‘cfg_list’ must be of a certain type, but how to solve this. Luckily my colleague Bouke (you can see what is on his mind on https://www.jume.nl ), quickly showed me the solution by specifying the variable in the correct type.

The following piece of code does the ‘set’ job. The solution is in the first line; setting the correct type (variable $speclist) for the ‘cfg_list’ parameter.


$speclist = [System.Collections.Generic.List[PSobject]]::new()

$spec = New-Object PSObject -Property @{
	hostname="logger1.net"
	port=514
	protocol="UDP"
}
$speclist.add($spec)
$spec = New-Object PSObject -Property @{
	hostname="logger2.net"
	port=514
	protocol="UDP"
}
$speclist.add($spec)

(Get-CisService -name 'com.vmware.appliance.logging.forwarding').set($speclist)


As always, I thank you for reading.

The importance of good data / How to set-up a baseline document?

Lately I’ve been working on machine learning and more specifically the Python Scikit library.
What I especially learned from this is the need to have a good
data-set before you want to do any kind of analysis or prediction.

But what does that have to do with subjects I usually write about? In the past period I have blogged regularly about configuration drift and tools like Vester and DSC resources for VMware.
We are also working on this within the company where I work.
Recently the assignment came to set up a baseline for the vCenter Server Appliances – you can’t solve configuration drift without thinking about the desired values, so time for a baseline. Apparently this seems simple, a baseline is a finite list of key-value pairs with the setting on one side and the value on the other side. In practice this seems a bit more complicated. I have to add that this baseline is not meant for a single vCenter Server, but for quite a few.

To get started, after connecting to a vCenter Server, the following command produces an overview of all settings for that vCenter:

PS> Get-AdvancedSettig -Entity <vCSA FQDN or IP>

Next to the fields Name and Value, you will also get the Type (of the Value) and sometimes a brief Description. Since vSphere 6.5 and up, you can also collect many appliance related settings using the API.
Now you can think, of all vCenters, collect the settings, set the desired values and done! In practice, however, there soon seemed to be some obstacles, such as:

  1. Not all vCenters are on the same version. Settings come and go. Some settings from vSphere 6.5 have disappeared in version 6.7, new settings have been introduced in version 6.7 and 7.0.
  2. Sometimes a setting exists, but returns an empty string. This is not equal to a setting that does not exist.
    Why worry about a setting with an empty string? What if, for whatever reason, a value does appear at any time?
  3. Not all settings are actually settings, but contain (status)information. We want to filter these out from our Configuration management tooling.

The baseline was created using PowerShell and the PowerCLI. The first step is to collect the settings of all vCenters as described above. The result is a .csv file for each vCenter. Incorporate the name of the vCenter in the filename like “vc01.csv”.

Continue reading

Another Vester Test file generator and more vCenter checks

Some time after finishing the “Vester Test file generator”, I was wondering how to get more configuration settings out of a vCenter Server. Then I realized that vCenter Servers also contain a large number of advanced settings.

To get an overview of ALL Advanced Settings in vSphere, connect to a vCenter Server and run the following line:

PS> Get-AdvancedSetting -Entity *

In the output you will discover three large groups:
VIServer, vCenter Server settings
VMHost, ESXi host settings (see other Vester Generator)
VM, Virtual machine settings

And finally, two small groups “Compute Cluster DRS” (9 settings) and “Datastore Cluster” (3 settings).

Continue reading

Securing DSC resources for VMware

Recently DSC Resources for VMware 2.0 was released. This new version comes with a lot of new resources and other features, like availability in the PowerShell Gallery. If DSC Resources for VMware is completely new,
I recommended reading the “Getting started” blog post, but do not follow the installation instructions. Instead install directly from the PowerShell Gallery, use something like this:

PS> Find-Module *VMware.vSphereDSC* | Install-Module

So after exploring “Vester”, the other DSC solution, it is now time to have a look at the DSC Resources for VMware 2.0.

Disclaimer: Windows PowerShell Desired State Configuration (from now on “DSC”) is often used for configuration management of Windows systems and as such is new to me. This post focuses on the use of DSC in a vSphere environment.

My setup;  I used an old Windows Server 2012R2 as a LCM. The vSphere environment is a VCSA version 6.5 and two ESXi hosts.
This post contains links to some script. All files mentioned in this post can be downloaded from this location. Then on the LCM, create a new folder named C:\VMwareDSC and place all the files in this folder.

One of my first goals was to understand how to create a good configuration. Luckily, the VMware DSC module contains an example folder, and I selected the VMHost_Config.ps1 configuration, an sample script for configuring an ESXi host.

Continue reading

Vester Test file generator

In previous posts (see below), I presented some tips for creating new Vester Test files. As you may know, ESXi hosts have a large number of so called “Advanced System Settings” Some of these settings are already present as Vester test files. These Advanced System Settings can be handled with the Get-AdvancedSetting and Set-AdvancedSetting cmdlets. With this knowledge and some PowerShell code, it is not to difficult to create a complete set (>1.100) of Vester Test files.

The New-VesterHostAdvanced.ps1 script can be found here.

A brief description how it works. After connecting to a vCenter Server, one of the available ESXi hosts needs to be selected. The selected host will be used to create an overview of all available Advanced System Settings.

Key in creating the scripts is the concept of Here documents, in PowerShell known as Here-String. See for a brief overview. Key in Here-Strings is the usage of single or double quotes with variables. A Here-String with double quotes allows the usage of variables. Run the following code to see the difference.

$var = 'MyValue'
$formatText1 = @"
Here-String with double quotes
The variable $var
Variable replacement

"@
$formatText1

$formatText2 = @'
Here-String with single quotes
The variable $var
Test as-is
'@
$formatText2

Continue reading