It is that time of year, I did not use the vRealize Suite Lifecycle Manager (from now on: vRLCM. New name is “VMware Aria Suite Lifecycle”) for quite some time. So to be able to upgrade Log Insight with vRLCM to the latest version, the first step is to upgrade the vRLCM by performing a System Upgrade. Second step is installing the latest Product Support Pack for vRLCM. If that is all done, you are good to start upgrading the products.
After the upgrade, vRLCM reported that for the VMware Identity Manager (vIDM) that comes with the vRLCM (See Environment => Global Environment), a new version 3.3.7 is also available. So I started upgrading the vIDM.
The first step is running an Inventory Sync.
Terraform and vSphere – Part 4: More Compute Cluster Resources
Introduction
In my previous posts we talked about Terraform, Desired State (and used a Compute Resource as an example) and Importing resources in a follow-up post. However VM/Host Groups, VM/Host Rules and VM Overrides are also part of the Cluster configuration.
Unlike the resource vsphere_compute_cluster where the state of a large number of options is automatically monitored by Terraform, this is not the case for the options discussed in this post.
Here it’s what you see is what you get. If you use Terraform, then it is more than a good practice to let Terraform take care of everything.
By this I mean the following; if you have created an affinity rule with Terraform for 2 VMs called VM1 and VM2, then the state of this affinity rule is monitored by Terraform. But if you manually create a second affinity rule for 2 VMs called VM3 and VM4, then this second affinity rule is unknown to Terraform and its state is not monitored. Should this occur, it can of course be solved by importing the second rule into the Terraform configuration afterwards.
Having said this, time for an overview of these cluster resources.
In vSphere Compute Clusters, 4 types of VM/Host rules can be created:
Keep virtual Machines together: Terraform resource:
vsphere_compute_cluster_vm_affinity_rule
Separate Virtual machines: Terraform resource:
vsphere_compute_cluster_vm_anti_affinity_rule
Virtual Machines to Hosts: Terraform resources:
vsphere_compute_cluster_host_group
vsphere_compute_cluster_vm_group
vsphere_compute_cluster_vm_host_rule
Virtual machines to Virtual machines: Terraform resource:
vsphere_compute_cluster_vm_dependency_rule
vSphere Compute clusters also allow you to create Overrides for individual objects. Terraform has 3 separate resources; to add a DPM override to a cluster for an ESXi hosts and to add DRS and HA overrides for virtual machines.
vsphere_dpm_host_override
vsphere_drs_vm_override
vsphere_ha_vm_override
Before showing examples, a few words about the flow of work. If you are working in a greenfield (that is adding resources), the flow is:
– add new resource(s) to the Terraform configuration file,
– run terraform plan and
– run terraform apply.
If you need to import existing cluster rules, the flow is a bit more complicated:
- Add new resource(s) to the Terraform configuration file. In some cases you can start with a basic configuration and add additional code after importing the resource
- Import the new resource
- Run terraform plan to detect missing parts
- Update the configuration file in case of errors or missing parts
- Run terraform plan again to check the configuration. If everything went well, terraform plan end with “No changes”
BTW, if you want to import existing cluster rules, you can run a PowerShell script like this to gather info.
Terraform and vSphere – Part 3: Import Resources
In the previous post “Terraform and vSphere – part 2: DSC”, we ended with a cluster created by Terraform called “Cluster-02” and showed that despite minimal configuration, all settings can be monitored by Terraform.
However, there is also a cluster “Cluster-01” in this Datacenter that was manually configured at the time. We would also like to manage this cluster with Terraform, together with our new Cluster.
Fortunately, Terraform offers an option for this too, how that works we will show in this post.
In the previous post, we ended with code for creating a cluster called Cluster-02. Before proceeding with adding Cluster-01, we check the current configuration by running the following command:
$ terraform plan
It should report: No changes
Now add the following lines to the code, these three lines is the minimum for starting the import of cluster Cluster-01.
# Existing Cluster to be imported
resource "vsphere_compute_cluster" "compute_cluster1" {
name = "Cluster-01"
datacenter_id = data.vsphere_datacenter.dc.id
}
Terraform and vSphere – Part 2: DSC
Desired State monitoring with Terraform?
Some time ago, I wrote a post about Terraform and vSphere. In this post, I showed how appliances (.ova files) can be deployed with Terraform. Another use case for Terraform is deploying virtual machines from templates.
But with this, we would almost forget the most important use of Terraform, deploying vSphere Infrastructure, in Terraform terminology resources like Clusters, vSwitches, Datastores and more. In doing so, I discovered an interesting feature of Terraform.
I’ve long been interested in configuration management for vSphere, see older posts on Vester and DSC Resources for VMware.
In a nutshell, Configuration Management is a systematic process for setting and maintaining the configuration of a resource over its lifetime.
In my experience, maintaining it during its lifetime is the trickiest part.
And that is where Terraform differs from other tools I have seen in recent years. As an example, we compare the configuration of a Compute Cluster in a vCenter Server using “DSC Resources for VMware” (see the example in this post) on the one hand and Terraform on the other.
A simple DSC configuration for creating a Cluster may look like this:
Configuration DCKoedood {
Import-DscResource -ModuleName VMware.vSphereDSC -ModuleVersion 2.2.0.84
vSphereNode $AllNodes.NodeName {
Datacenter "DCKoedood" {
Name = 'DCKoedood'
Location = [string]::Empty
Ensure = 'Present'
}
Cluster "Cluster-02" {
Name = 'Cluster-02'
Location = [string]::Empty
DatacenterName = 'DCKoedood'
DatacenterLocation = [string]::Empty
Ensure = 'Present'
HAEnabled = $true
DrsEnabled = $true
DependsOn = "[Datacenter]DCKoedood"
}
}
}
Log Insight 8.8.2 – API update
I wrote it in my previous post “In addition, Log Insight has an even larger number of APIs with “Tech Preview” status.”.
“Tech Preview” status also means in VMware’s words “Their design or implementation may change without warning and should not be depended on. A tech preview API may become supported, have forwards-incompatible changes or be removed during an upgrade.“.
And that is exactly what has happened with the release of Log Insight version 8.8.2 and the API endpoint for groups and roles.
By the way, have you also noticed that the Log Insight API has been a version 2 for some time now? To be exact, since the release of Log Insight version 8.6.0.
In version 8.8.2 endpoint /api/v2/user-groups replaces the /api/v2/authgroups. On the positive side, the way to assign roles to a group has been improved, as we will see later.
Using snippets of PowerShell code, basic operations according to the CRUD (Create, Read, Update and Delete) schema are shown for Directory groups and Roles.
First, the code for setting up a session with the Log Insight API, the code is written for PowerShell Core 7.x. The variable $vLISessionHeader is part of the Invoke-Restmethod cmdlet used in the other examples.
Log Insight – automating Groups and Roles
Important: The examples described in this blog are applicable for Log Insight version 8.8.0 and below, but no longer function in version 8.8.2 due to changes in the API. More on this in a forthcoming post.
Adding Directory Groups
A little while ago I wrote about a poc how to use Ansible and a pipeline to upgrade Log Insight. Shortly thereafter, I looked at the capabilities of vRealize Suite Lifecycle Manager (from now on: vRSLCM) and did deployments of Log Insight and vRealize Operations Manager. vRLSCM can take care of some of the configuration of Log Insight, such as NTP, DNS, authentication and Cluster VIPs, in addition to the deployment. Configuration of Directory Groups and Roles, also important, is (currently) not included. If you want to automate the whole process of deployment and configuration of Log Insight, additional action is needed.
In this post, I’ll show you how to configure groups and roles in Log Insight 8.8.0 with PowerShell 5.x but also with Ansible 2.12.6 (a better understanding of Ansible, is one of my goals for this year and nothing is better than practice).
The starting point is the documentation of the Log Insight REST API, available at link: https: //<fqdn Log Insight or IP address>/rest-api.
The APIs presented here do have a “Supported” status. In addition, Log Insight has an even larger number of APIs with “Tech Preview” status. The Tech Preview APIs are in most cases incompletely documented.
Available documentation can be found by including the word “internal” in the link: https: //<fqdn Log Insight or IP address>/internal/rest-api.
See also my post Log Insight REST API.
vRSLCM – Exception while validating for Scale-Out
Recently, I have been exploring the capabilities of VMware’s vRealize Suite Lifecycle Manager (from now on vRSLCM). vRSLCM is a product for deployment, configuration, upgrading & patching, scale-up and scale-out of VMware products like; vRealize Automation, Orchestrator, Operations Manager, Network Insight, Log Insight and Business for Cloud. See this link for more information.
I usually do this by first installing the product and then running various scenarios, such as this one for Log Insight:
1. Deploy a 3 node Log Insight cluster, version 8.6.0
2. Upgrade to version 8.6.2
3. Scale-Out, by adding an extra worker node to the cluster
My first attempt adding an extra worker node to the existing Log Insight cluster ended, after choosing “ADD COMPONENTS”, with the following message:
Fig. 1
The existing Log Insight nodes were up and running, so what happened?
A log file named /var/log/vrlcm/vmware_vrlcm.log was very useful and explains what is happening during this action, see the following lines:
2022-05-26 12:57:06.750 INFO [pool-3-thread-20] c.v.v.l.p.v.VrliScaleoutOvaValidationTask - -- vRLI instance :: {
"vrliHostName" : "vRLI-1.virtual.local",
"port" : "9543",
"username" : "admin",
"password" : "JXJXJXJX",
"provider" : "Local",
"agentId" : null
}
2022-05-26 12:57:09.817 ERROR [pool-3-thread-20] c.v.v.l.d.v.r.c.VRLIRestClient - -- Failed to get the vRLI authentication token. No route to host (Host unreachable)
2022-05-26 12:57:12.889 ERROR [pool-3-thread-20] c.v.v.l.p.v.VrliScaleoutOvaValidationTask - -- Exception while validating the vRLI VA OVA for Scaleout :
Port 9543 is used while communicating with the Log Insight API, the “Failed to get the vRLI authentication token” makes it clear that communication with the primary node, named vRLI-1.virtual.local, is not possible, hence the “No route to the host”. A ping command from the vRSLCM to the primary node by hostname, yields no results and is a confirmation that the DNS registration has gone haywire.
After the DNS registration is restored, the primary node is resolvable again and the scale-out can be continued.
Bottom line, when you see this message, check DNS and/or network connectivity to the targets.
Upgrade Log Insight using Ansible and Azure DevOps
Intro
Some time ago, I demonstrated how Terraform can be applied in a vSphere environment to deploy .OVA files, like one or more Log Insight nodes. Now it is time to upgrade our Log Insight node(s), and this time we will use Ansible to do the job.
In my daily life, Ansible combined with Azure DevOps is the tool of choice to perform all types of work. This post can be seen as a demonstration how to perform an upgrade of Log Insight with the mentioned tools.
Disclaimer
- This is a proof of concept, the code shown is a minimal viable product, no effort has been made to protect accounts and passwords. The code can be used as a basis for a production environment, but will need to be modified to meet the requirements of the organization.
- I am also aware that you can still manually upgrade Log Insight or use tools like the vRealize Suite Lifecycle Manager.
- Code for creating snapshots is not included, as well as other non-essentials.
Preparation
The first step is to install three Log Insight nodes version 8.4.1. After deployment of the first node, the option “Start new Deployment” was chosen and a basic installation was done, including a Virtual IP address, this is now the primary node. After this a power down was done and a snapshot was taken. The other two nodes have been powered down immediately after deployment and a snapshot has been made of these nodes as well. These nodes are later used as additional nodes to create a 3-node cluster.
Somewhat anticipating the topics to come, I found out pretty quickly that a web server is required to have the upgrade packages (.PAK files) available during an upgrade. In my case, I solved that by installing a simple web server on an existing Windows host, such as Miniweb. Installation is unzipping the downloaded file, create a new folder under “htdocs” and copy the Log Insight upgrade packages (.PAK) files to this new folder.
Now we can start with the first question, how to upgrade Log Insight with Ansible?
Log Insight Duplicate Webhooks
After upgrading Log Insight to version 8.4.x, something strange happened, let me explain.
Log Insight has the ability to forward Alerts for further processing via a Webhook. For some information about usage of Webhooks, see this post.
Webhooks are configured separately and can be used hereafter in the configuration of an Alert as a notification option, whether or not in combination with email address(es).
Fig. 1 – Alert with email configured, webhook not yet selected
Log Insight Filtering and Masking
In this post, I want to focus on two features of Log Insight that may not be known to everyone, namely Log Filtering and Log Masking. Since I regularly work with Log Insight’s log forwarding functionality, I also want to know the impact of filtering and masking on this functionality.
Both Log Masking and Log Filtering can be found under Administration \ Log Management. Here also the Log Forwarding and Index Partitions can be found.
Log Filtering
Log filtering should not be confused with event filtering, a Log Filter drops ingested events that match defined filter criteria. Dropped events are not stored in Log Insight. The advantages of a Log Filter, saving storage space and only desired events are shown. Disadvantage, an incorrectly configured Log Filter can cause desired events to “disappear”.
To create a new Filter, click “New Configuration”. Each filter needs a Name (no spaces allowed) and a Drop Filter, consisting of a field, an operator and a value (wildcards * and ? are supported).
A Field can be Hostname, text, appname, source, facility etc.
Four operators are available; matches, does not match, starts with and does not start with.
Values are exact matches or can be combined with wildcard * (zero or more characters) and ? (zero or any single character) pre- and/or postfix.
The drop filter can be expanded with additional filter rules, together they form an AND construct.
After saving the configuration, the filter will become active. New filters are enabled by default. Disabling a filter will stop dropping events.
Note: NOT selecting a filter, will drop ALL logs!
As a first example, I created a Log Filter, to drop all VSAN related events from a host named nuc10-01. Other events originating from host nuc10-01 must pass the filter, the same goes for VSAN events from other hosts like nuc10-02 and nuc10-03.
Fig.1
Adventures in a Virtual World 