Aria Suite Lifecycle – Error LCMVRLICONF40004

Recently, I wanted to start a belated upgrade of an Aria Operations for Log cluster with the help of tool VMware Aria Suite Lifecycle. The first step is running an inventory. Within seconds, error LCMVRLICONF40004 was presented.
Fig. 1
The “invalid hostname” message was not very helpful, the “show more” section provided the following information:

com.vmware.vrealize.lcm.vrli.
Exception: Cannot execute ssh commands.
Exception encountered : Session.connect: java.security.spec.
at com.vmware.vrealize.lcm.
at com.vmware.vrealize.lcm.
at java.base/java.util.
at java.base/java.util.
at java.base/java.lang.Thread.

The first step, based on the “Cannot execute ssh commands”, was checking if the nodes of the Aria Operations for Log cluster were reachable and the correctness of the root password used by Lifecycle. Result everything OK.

Next step, after some ‘googling´, I have found VMware KB “InvalidKeySpecException Error Code : ‘LCMVRNICONFIG90115’ when performing inventory sync in Aria Suite Lifecycle Manager Inventory Sync for Aria Operations for Networks (96553)” which contains a reference to error LCMVRLICONF40004.

The KB reveals the cause of the issue “Recent Aria Suite Lifecycle PSPACKs specifically version 8.14 Pspack 4 and above have hardened the SSH settings on the Aria Suite Lifecycle appliance. This can cause communication issues for products which do not support any of the newer macs or ciphers.”, so the cause is clear.

For the final fix, open KB “Steps for removing weak SHA1 algorithms and ciphers from VMware Aria Products (95835)”, mentioned here.

From the second KB, follow the instructions in section “VMware Aria Operations for Logs”, pointing to the KB “Remove SHA1 from SSH service in VMware Aria Operations for Logs 8.12.x and 8.14.x (95974)”. The third KB finally contains the steps that will solve the issue. For VMware Aria Operations for Logs version 8.12 follow the steps. It comes down to saving the current the version of the /etc/sshd/sshd_config file and make some modifications as described. Do not forget to restart the sshd daemon and repeat the steps for all nodes of the VMware Aria Operations for Logs cluster.

Now you should be able to successfully update the VMware Aria Operations for Logs cluster.

Aria Operations for Logs 8.14.1

VMware recently published VMware Aria Operations for Logs version 8.14 (8.14 Build 22564181, released on 19-10-2023).
Like most updates, this one comes with functional improvements and security enhancements.
In my opinion, the main reason to install this version is because it fixes a serious vulnerability (CVSS score 8.1), present in the previous versions since 8.6. See this bulletin for more information.

For a complete overview of the enhancements, see the 8.14 release notes. The most prominent are:

• The product now supports the external NSX Advanced Load Balancer, providing more load-balancing configurations than the Aria Operations internal Load Balancer.
• New Content packs are added for OpenShift and Tanzu Kubernetes Grid

Aria Operations for Logs, found a bug in Access Control

It looks like VMware Aria Operations for Logs (previously Log Insight) contains a bug in the Access Control part. I will explain; Access is granted by assigning a role to users or groups. A role is a group of permissions that ultimately determines what a user is allowed to do.
For more information about the Roles and permissions in Aria Operations for logs, read my post.Fig. 1 – Four predefined Roles

Aria Operations for Logs API and Workspace ONE issue

A brief description, recently I ran into a issue when working with an Aria Operations for Logs cluster configured to use Workspace One Access enabled for Active Directory support as an authentication source. Logging in via the GUI was not the problem, I was updating scripts as described in this post to work with Workspace One Access.

While working with the Aria Operations for Logs API, the first step is to authenticate. Authentication requires 3 parameters: username, password and provider.

The username and password do not require further explanation, although, one should also pay attention to the username in some cases.
The provider refers to the three supported authentication sources: Local (in case the local admin account is used), Active Directory (in case Active Directory support has been configured) and finally Workspace One Access.

The first part of the solution; the name of the provider should be written with the capital letters in the right places, like “Local”, “ActiveDirectory” and finally “vIDM” for Workspace One Access.

The second part, note the username. As an example when Workspace One Access has been enabled for an Active Directory named “acme.com”, the username should be something like “user@acme.com”. This notation will not work: “acme\user”.

I hope you can benefit from this, thank you for reading.

Aria Operations for Logs 8.12

Recently VMware released vRealize Log Insight, correction “VMware Aria Operations for Logs” version  8.12. The new name for this product had been going around for some time, but this is the first version to be released as “VMware Aria Operations for Logs” as you can see after booting the product.Fig. 1

Although things are still not always going well with the new naming as you can see below.Fig. 2

Apart from the branding, what else comes with the 8.12 version? The full overview can be found in the release-notes can be found here.

Log Insight – Critical issue!

UPDATE per May 2023: KB 91441, mentioned in the post below has been replaced by KB 92080, title “Replace expired internal certificate in vRealize Log Insight”.
The reason is that after the internal certificate expired as of 30 April, the instructions needed to be updated. By the way, you are automatically redirected to the new KB.

I also discovered that if you want to build a cluster based on a pre 8.12 version after 30 April 2023, you will run into problems. The worker nodes do not communicate with the primary node, retrospectively applying KB 92080 is not the solution. What did work in my case was: 1. deploy primary node, 2. upgrade certificate on primary node, 3. deploy additional worker nodes, 4. upgrade certificate first, 5. add nodes to the primary node.

A short but important post if you are running VMware Aria Operations for Logs (formerly VMware vRealize Log Insight)!

Log Insight comes with self-signed certificate (GUI), but also with an internal “Cassandra” certificate. Both certificates are set to expire on April 30th 2023 and it is critical that you take action before the April 30th deadline to avoid any outages.

Even if you have replaced the self-signed certificate for the GUI in the past, you will still need to take action to replace the internal certificate.

Please read VMware KB “Updating the vRealize Log Insight Internal Certificate (91441)”.
There are basically two ways to fix this issue;
a) wait for the new release (Version 8.12?) and run the update, or
b) replace the impacted certificates by following the detailed instruction in the KB.

Today, we did a quick test, following “Workaround Option 1”, which was successfull.

Log Insight – Cassandra 101

Intro

Recently, I was involved in some Log Insight incidents. To successfully resolve them, an action in Log Insight’s Cassandra database was required. Besides application logic, Cassandra, the distributed NoSQL database, is an important component of Log Insight. So, time for a closer look at Cassandra in Log Insight.

Warning: Be careful, after a login into the database; never execute commands without knowing the impact. Certain commands can render Log insight unusable!

Access

For this article, I am using Log Insight version 8.10. VMware has published KB 57901 “How to Access the Cassandra Database in vRealize Log Insight”, which contains instructions on how to access the Cassandra database. First requirement is to start an SSH session to a Log Insight node and log in as user root. In older versions of Log Insight, the process was a bit more cumbersome and required retrieving the password, since version 8.8 just one command suffices:

root@li-vip3 [ ~ ]# cqlsh-no-pass
Connected to loginsight at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.11.11 | CQL spec 3.4.4 | Native protocol v4]
Use HELP for help.
lisuper@cqlsh> 

Now cqlsh, the CLI for interacting with Cassandra using the Cassandra query language (CQL) has been started, as user “lisuper”.

vRLCM – Recover from Error LcmVidmImport0018

It is that time of year, I did not use the vRealize Suite Lifecycle Manager (from now on: vRLCM. New name is “VMware Aria Suite Lifecycle”) for quite some time. So to be able to upgrade Log Insight with vRLCM to the latest version, the first step is to upgrade the vRLCM by performing a System Upgrade. Second step is installing the latest Product Support Pack for vRLCM. If that is all done, you are good to start upgrading the products.
After the upgrade, vRLCM reported that for the VMware Identity Manager (vIDM) that comes with the vRLCM (See Environment => Global Environment), a new version 3.3.7 is also available. So I started upgrading the vIDM.
The first step is running an Inventory Sync.

Terraform and vSphere – Part 4: More Compute Cluster Resources

Introduction

In my previous posts we talked about Terraform, Desired State (and used a Compute Resource as an example) and Importing resources in a follow-up post. However VM/Host Groups, VM/Host Rules and VM Overrides are also part of the Cluster configuration.

Unlike the resource vsphere_compute_cluster where the state of a large number of options is automatically monitored by Terraform, this is not the case for the options discussed in this post.
Here it’s what you see is what you get. If you use Terraform, then it is more than a good practice to let Terraform take care of everything.
By this I mean the following; if you have created an affinity rule with Terraform for 2 VMs called VM1 and VM2, then the state of this affinity rule is monitored by Terraform. But if you manually create a second affinity rule for 2 VMs called VM3 and VM4, then this second affinity rule is unknown to Terraform and its state is not monitored. Should this occur, it can of course be solved by importing the second rule into the Terraform configuration afterwards.

Having said this, time for an overview of these cluster resources.
In vSphere Compute Clusters, 4 types of VM/Host rules can be created:

Keep virtual Machines together: Terraform resource:
vsphere_compute_cluster_vm_affinity_rule

Separate Virtual machines: Terraform resource:
vsphere_compute_cluster_vm_anti_affinity_rule

Virtual Machines to Hosts: Terraform resources:
vsphere_compute_cluster_host_group
vsphere_compute_cluster_vm_group
vsphere_compute_cluster_vm_host_rule

Virtual machines to Virtual machines: Terraform resource:
vsphere_compute_cluster_vm_dependency_rule

vSphere Compute clusters also allow you to create Overrides for individual objects. Terraform has 3 separate resources; to add a DPM override to a cluster for an ESXi hosts and to add DRS and HA overrides for virtual machines.
vsphere_dpm_host_override
vsphere_drs_vm_override
vsphere_ha_vm_override

Before showing examples, a few words about the flow of work. If you are working in a greenfield (that is adding resources), the flow is:
– add new resource(s) to the Terraform configuration file,
– run terraform plan and
– run terraform apply.

If you need to import existing cluster rules, the flow is a bit more complicated:

• Add new resource(s) to the Terraform configuration file. In some cases you can start with a basic configuration and add additional code after importing the resource
• Import the new resource
• Run terraform plan to detect missing parts
• Update the configuration file in case of errors or missing parts
• Run terraform plan again to check the configuration. If everything went well, terraform plan end with “No changes”

BTW, if you want to import existing cluster rules, you can run a PowerShell script like this to gather info.

Terraform and vSphere – Part 3: Import Resources

In the previous post “Terraform and vSphere – part 2: DSC”, we ended with a cluster created by Terraform called “Cluster-02” and showed that despite minimal configuration, all settings can be monitored by Terraform.

However, there is also a cluster “Cluster-01” in this Datacenter that was manually configured at the time. We would also like to manage this cluster with Terraform, together with our new Cluster.
Fortunately, Terraform offers an option for this too, how that works we will show in this post.

In the previous post, we ended with code for creating a cluster called Cluster-02. Before proceeding with adding Cluster-01, we check the current configuration by running the following command:

$ terraform plan

It should report: No changes
Now add the following lines to the code, these three lines is the minimum for starting the import of  cluster Cluster-01.

 
# Existing Cluster to be imported
resource "vsphere_compute_cluster" "compute_cluster1" {
  name            = "Cluster-01"
  datacenter_id   = data.vsphere_datacenter.dc.id
}