vCSA, root partition is (almost) full

18/02/2018

hwA short post on a topic that I recently experienced on vCenter Server Appliance, version 6.0.
After receiving an alert that the root “/” partition was quickly filling up, it is time to act quickly. When the root partition reaches 100% of it’s capacity, service disruption can occur.
First step is to check the capacity of the vCSA partitions. Log in to the vCSA through SSH, if you are running the appliance shell, enable and access the Bash shell:

Command> shell.set --enabled true
Command> shell

In the Bash shell run this command to check the capacity of the partitions:

# df -h

The second line of the output (starting with /dev/sda3) shows the status of the root partition. If the value under Use% reaches 100%, you are in trouble. Also notice that the root partition is only 11 GB.
Second step is to determine the root cause of the full partition. A good strategy is to look for large consumers. The next command searches for files larger then 100 MB, only on the root partition:

# find / -xdev -type f -size +100M

In my case some interesting results:

/usr/lib/vmware-sca/wrapper/bin/wrapper.log
/usr/lib/oracle/11.2/client64/lib/libociei.so
/var/log/dnsmasq.log-20180121
/var/log/dnsmasq.log-20180128
/var/log/dnsmasq.log-20180107
/var/log/dnsmasq.log-20180114
/var/log/dnsmasq.log
/etc/vmware-vpx/docRoot/client/Vmware-viclient.exe

The most eye-catching files are: the wrapper.log and the dnsmasq.log files.

Read the rest of this entry »

Advertisements

Getting started with the vCSA 6.x – Part 3

22/01/2018

In part 1 and part 2 of this series about the vCSA, we have covered topics like; the shells, filesystem, services, health, logging, database and some extra tools. Recently I realised there a few more topics worth mentioning.

Appliance MUI

In pre 6.0 releases of the vCSA, there was a vCenter Server Appliance Management Interface, better known as the VAMI. This management interface is written in HTML5 and is now called the e Appliance Management User Interface (Appliance MUI).

You will find the new management interface in vCSA 6.0 and 6.5, however there are some differences.

You can login to this interface, using: https://<vCSA fqdn or IP>:5480. Us a local account such as the “root” account.

Fig. 1 – Summary vCSA 6.0.

Read the rest of this entry »


Writing effective scripts using VMware PowerCLI

30/12/2017

20171224-00Lately I have been busy writing some Windows PowerShell scripts for a vSphere environment. I noticed that there are some similarities between learning a spoken language and a programming language. In both cases you start by learning the grammar and vocabulary and develop your skills by a lot of practicing. But for both skills, when you have not used them for a while, the skills will fade.
While writing and testing my scripts, I realized that a good preparation and a structured way of working will help you becoming more productive and making fewer mistakes.
This post is not a full blown Windows PowerShell course, but contains some insights I would like to share with you. If this is all new, I recommend following a PowerShell Getting Started training. Pluralsight offers over 11 Windows PowerShell courses from beginner to expert level. So if you are relatively new to Windows PowerShell and the VMware PowerCLI, please read on.

Read the rest of this entry »


About Long Fat Networks and TCP tuning

08/09/2017

Recently I came about a data communications subject that was pretty unknown to me, known as the Bandwidth-delay product. Knowledge about this can help you to recognize certain network issues and ways to resolve them. It is all about two Linux hosts, a source and a destination host, communicating with each other over a high capacity network link. The question is how can you, given this scenario, reach maximum throughput over the network?

BDP

The first step is to determine the Bandwidth-delay product for this network. Bandwidth-delay product (BDP) is defined as the product of a data link’s capacity (in bits per second) and its round-trip delay time (in seconds). The result, the amount of data (in bits or bytes), is the maximum amount of data on the network at any given time, that is data that has been transmitted but not yet acknowledged.
Why is this important? The TCP protocol is designed for reliable transmission of data, acknowledgements are an essential part of the protocol. A high BDP value has impact on the efficiency of TCP, because the protocol can only achieve optimum throughput if a sender sends a sufficiently large quantity of data before being required to stop and wait until a confirming message (acknowledgement) is received from the receiver, acknowledging successful receipt of that data.

Read the rest of this entry »


Getting started with the vCSA 6.x – Part 2

30/07/2017

In the previous post we started to unravel the vCSA and discussed topics like the Appliance shell, the file system and the services. In this post we will continue with the vCSA Health.

Health

Knowing the health of your system is important. Like the Windows vCenter Server, the vCSA is also able to report its health. Most common is using the vSphere Web Client and from the main menu, choose: System Configuration and watch the “Service Health” pane. Detailed information can be found by clicking on the various Services.

Figure 1

However, from the Appliance shell, the following API command will also inform you”

Command> system.health.get

If everything is OK, it will report; Health: green

In the Bash shell, you can browse to the folder: /etc/vmware-sca/health/.
On a vCSA 6.0, you will find two files with health status information:

vmware-vpxd-health-status.xml
vmware-postgres-health-status.xml

On a vCSA 6.5, you will only find the first file.

Read the rest of this entry »


Getting started with the vCSA 6.x – Part 1

12/07/2017

The vCenter Server Appliance is the new vCenter Server. In the old days, we had a brand new Windows Server on which the vCenter Server was installed. The necessary database server was quite often an external MS SQL database and sometimes an internal database. In the those days, tweaking the Windows Server and the installed components was more or less a common practice, due to the familiarity with Windows.

But now the vCenter Server Appliance (vCSA) is the new and preferred standard. The vCSA comes as a virtual appliance and is ready to run within minutes compared to the old vCenter Server. Although a (virtual) appliance still means; (virtual) hardware, an operating system, middleware and applications, VMware likes to treat the vCSA as a “black box”. Like most appliances, the operating system is a Linux flavor and you can log in. After a successful log in, you will encounter the first discouragement; you are not welcomed by a Bash shell but with the default “Appliance Shell”. For more information read my post on the vCSA shells.
In this post, a brief introduction on the following topics; Appliance shell, File system and the Services.

Read the rest of this entry »


Get-ClusterRules

16/06/2017

I recently encountered an interesting question, maybe not the one you will see every day. A vCenter Center server runs a large number of Clusters; the VMs on those clusters are controlled by a considerable number of DRS rules. The question that raised; “How do we know if the DRS rules we once designed are still in place?” In the course of time, rules can be disabled, VM or Host groups does not match any more. Trying to answer this question by going through the vCenter Server configuration is not the way to go.

Thankfully, the VMware PowerCLI contains a useful Cmdlet Get-DrsRule that enables you to create a dump of the configured rules for each cluster. This makes checking your configuration a lot easier.

But there is another thing, now we know about the configuration, but what do we know about the actual situation? For instance, VM to Host affinity has “should” and “must” rules, but to what extent is a “should” rule fulfilled?

So time to create a PowerShell script which performs the following tasks; for each Cluster within a vCenter Server, a dump of the configured DRS rule is made. The second part of the script determines on which host a VM is running and compares it to the configured rules. The script will also report if a DRS rule is disabled and displays the power state of each VM. You will probably worry less about a powered down VM.

The script can be found here on GitHub.

I am aware that the script and my programming skills are far from perfect, so expect updated versions in the future.