VCAP5-DCA Objective 4.1 – Implement and maintain complex VMware HA solutions

Objectives

  • Calculate host failure requirements
  • Configure customized isolation response settings
  • Configure HA redundancy
    • Management Network
    • Datastore Heartbeat
    • Network partitions
  • Configure HA related alarms and monitor an HA cluster
  • Create a custom slot size configuration
  • Understand interactions between DRS and HA
  • Analyze vSphere environment to determine appropriate HA admission control policy
  • Analyze performance metrics to calculate host failure requirements
  • Analyze Virtual Machine workload to determine optimum slot size
  • Analyze HA cluster capacity to determine optimum cluster size

Reading:

Calculate host failure requirements

Official Documentation:
vSphere Availability Guide, Chapter 2, Section “Host Failures Cluster Tolerates Admission Control Policy”, page 16.

Summary:
One step back:
vCenter Server uses admission control to ensure that sufficient resources are available in a cluster to provide failover protection and to ensure that virtual machine resource reservations are respected.
Three types of admission control are available.

  • Host
    Ensures that a host has sufficient resources to satisfy the reservations of all virtual machines running on it.
  • Resource Pool
    Ensures that a resource pool has sufficient resources to satisfy the reservations, shares, and limits of all virtual machines associated with it.
  • vSphere HA
    Ensures that sufficient resources in the cluster are reserved for virtual machine recovery in the event of host failure.

NOTE: vSphere HA is the only type of admission control that can be disabled. When vSphere HA admission control is disabled, vSphere HA ensures that there are at least two powered-on hosts in the cluster even if DPM is enabled and can consolidate all virtual machines onto a single host. This is to ensure that failover is possible.

There are three options for vSphere HA admission control:

  1. Host Failures Cluster Tolerates;
  2. Percentage of Cluster Resources Reserved (preferred option)
  3. Specify a Failover Host

With the Host Failures Cluster Tolerates admission control policy, vSphere HA ensures that a specified number of hosts can fail and sufficient resources remain in the cluster to fail over all the virtual machines from those hosts.

This is how the policy works:

  1. Calculates the slot size.
    A slot is a logical representation of memory and CPU resources. By default, it is sized to satisfy the requirements for any powered-on virtual machine in the cluster.
  2. Determines how many slots each host in the cluster can hold.
  3. Determines the Current Failover Capacity of the cluster.
    This is the number of hosts that can fail and still leave enough slots to satisfy all of the powered-on virtual machines.
  4. Determines whether the Current Failover Capacity is less than the Configured Failover Capacity (provided by the user).
    If it is, admission control disallows the operation.

This leaves a few questions

  • How is the slot size calculated?
    Slot size is comprised of two components, CPU and memory.

    • vSphere HA calculates the CPU component by obtaining the CPU reservation of each powered-on virtual machine and selecting the largest value. If you have not specified a CPU reservation for a virtual machine, it is assigned a default value of 32 MHz. You can change this value by using the das.vmcpuminmhz advanced attribute.)
    • vSphere HA calculates the memory component by obtaining the memory reservation, plus memory overhead, of each powered-on virtual machine and selecting the largest value. There is no default value for the memory reservation.
    • From Slots to computing the current Failover capacity?
      • vSphere HA determines each host’s CPU and memory resources that are available for virtual machines. These amounts are those contained in the host’s root resource pool, not the total physical resources of the host. This can be found on the “Resource Allocation” Tab .

Figure 1 – Resource Allocation

  • Resources being used for virtualization purposes are not included. Only hosts that are connected, not in maintenance mode, and that have no vSphere HA errors are considered.
  • The maximum number of slots that each host can support is then determined.
    The host’s CPU resource amount is divided by the CPU component of the slot size and the result is rounded down.
    The host’s Memory resource amount is divided by the CPU component of the slot size and the result is rounded down.
  • These two numbers are compared and the smaller number is the number of slots that the host can support.
  • The Current Failover Capacity is computed by determining how many hosts (starting from the largest) can fail and still leave enough slots to satisfy the requirements of all powered-on virtual machines.

The “Advanced Runtime Info” presents a nice Summary:

Figure 2

In this example we have:

  • A cluster of 3 equally sized hosts (Total hosts in Cluster)
  • All 3 hosts are up and running (Total good hosts in Cluster)
  • Each host has 113 slots. 3 x 113 gives a total of 339 slots (Total Slots in Cluster)
  • Used slots (42), number of slots assigned to powered-on VMs.
  • Available slots (184), number of slots available to power-on additional VMs
  • Failover slots,
  • Cluster is configured to tolerate one host failure, so
    Available slots (184) = Total slots in Cluster (339) – Used Slots (42) – Failover Slots (113)

If the Host Failures Cluster Tolerates setting is used, the following apply:

  • Ensure that all cluster hosts are sized equally. An “unbalanced” cluster results in excess capacity’s being reserved to handle failure of the largest possible node.
  • Attempt to keep virtual machine resource reservations similar across all configured virtual machines. Mixing virtual machines of greatly different CPU and memory requirements will cause the slot size calculation to default to the largest possible of all virtual machines, limiting consolidation.

The second option “Percentage of Cluster Resources Reserved”, HA ensures that a specified percentage of memory and CPU resources are reserved for failover. This policy is recommended for situations where you must host virtual machines with significantly different CPU and memory reservations in the same cluster or have different-sized hosts in terms of CPU and memory capacity (vSphere 5.0 adds the ability to specify different percentages for memory and CPU through the vSphere Client). A key difference between this policy and the Host Failures Cluster Tolerates policy is that the capacity set aside for failures can be fragmented across hosts. So there is a chance that at the time of failing over a virtual machine, there might be insufficient unfragmented capacity available on a single host to power on all virtual machines. DRS, if enabled, will attempt to defragment the capacity in such situations.

Figure 3

This policy offers the most flexibility in terms of host and virtual machine sizing and is sufficient for most situations. In most cases, a simple calculation of 1/N, where N = total nodes in the cluster, will yield adequate sparing.

Figure 4 – HA section from Cluster Summary Tab

With this admission Control policy configured, The Summary tab on the Cluster level presents an overview of configured and available resource. I tend to forget how the maths is done. This example by VMware tells you how:

Figure 5 – Calculating Available Resources (VMware)

Specify a Failover Host: VMware HA designates a specific host or hosts as a failover host(s). When a host fails, HA attempts to restart its virtual machines on the specified failover host(s). The ability to specify more than one failover host is a new feature in vSphere High Availability 5.0. When a host is designated as a failover host, HA admission control disallows powering on virtual machines on that host, and DRS will not migrate virtual machines to the failover host. It effectively becomes a hot standby.

NOTE: If you use the Specify Failover Hosts admission control policy and designate multiple failover hosts, DRS does not load balance failover hosts and VM-VM affinity rules are not supported.
Other references:

  • A

Configure customized isolation response settings

Official Documentation:
vSphere Availability Guide

Summary:
In vSphere 5 HA within one HA cluster we have only one Master (Fault Domain Manager Master (FDMS)), the other Hosts are Slaves.

In a vSphere HA cluster, three types of host failure are detected:

  • A host stops functioning (that is, fails).
  • A host becomes network isolated (network partitions, due to management network failure).
  • A host loses network connectivity with the master host.

When the master host in a vSphere HA cluster cannot communicate with a slave host over the management network, the master host uses datastore heartbeating to determine whether the slave host has failed, is in a network partition, or is network isolated. If the slave host has stopped datastore heartbeating, it is considered to have failed and its virtual machines are restarted elsewhere.

A Slave host declares itself network isolated in case connectivity with the Master has been lost and also connectivity with the Isolation address (the default Isolation address is the Gateway specified for the management address, this can be modified das.isolationaddress[1-10]).
An isolated Host must determine whether it must take any action based upon the configuration settings for the isolation response for each virtual machine that is powered on. The isolation response setting provides a means to dictate the action desired for the powered-on virtual machines maintained by a host when that host is declared isolated. There are three possible isolation response values that can be configured and applied to a cluster or individually to a specific virtual machine. These are Leave Powered On, Power Off and Shut Down.

  • Leave Powered On, is the default value in vSphere 5 HA and also the recommended setting
  • Power Off, VMs are immediately and not gracefully stopped. The advantage is that HA will restart the affected VM more quickly. Recommended setting for environments that use network-based storage like iSCSI and NFS.
  • Shut Down, tries to gracefully shut down a VM with help of the VMware Tools. It will wait for 5 minutes to shut down before it will Power Off the VM.

Figure 6

The restarting by VMware HA of virtual machines on other hosts in the cluster in the event of a host isolation or host failure is dependent on the “host monitoring” setting. If host monitoring is disabled, the restart of virtual machines on other hosts following a host failure or isolation is also disabled. Disabling host monitoring also impacts VMware Fault Tolerance because it controls whether HA will restart a Fault Tolerance (FT) secondary virtual machine after an event.

Figure 7 Cluster Default and Individual settings

Host Isolation Response can be overridden on a VM level.

NOTE: If Enable Host Monitoring is selected, each host in the cluster is checked to ensure it is running. If a host failure occurs, virtual machines are restarted on another host. Host Monitoring is also required for the vSphere Fault Tolerance recovery process to work properly.

Other references:

  • A

Configure HA redundancy

Official Documentation:
vSphere Availability Guide

Summary:
Configure HA redundancy

  • Management Network
  • Datastore Heartbeat
  • Network partitions

Although VMware HA provides a lot of benefits and can protect you from disaster, the are some guidelines and considerations building your HA cluster. The vSphere Availability Guide Deployment Best Practices provides a lot of useful information

Management Network

Setting up Management Network redundancy is highly recommended and can be set up in two ways:

  • At the network adapter level (teaming);
  • At the Management Network level (second vSwitch)

To configure a network adaptor team for the management network, configure the vNICs in the vSwitch configuration for the ESXi host for active/standby configuration.

Requirements:

  • Two physical network adaptors
  • VLAN trunking
  • Two physical switches

The vSwitch (vSwitch0 in this example) should be configured as follows:

Figure 8

Port group Management Network
VLAN 2
Management Traffic is Enabled
vmk0: 192.168.2.53
vmnic0 Active / vmnic1 Standby
Load balancing: route based on the originating virtual port ID (default)
Failback: No

Port group vMotion
VLAN 21
vMotion is Enabled
vmk0: 192.168.21.53
vmnic1 Active / vmnic0 Standby
Load balancing: route based on the originating virtual port ID (default)
Failback: No

Needless to say that vmnic0 is connected to the first switch in the stack and vmnic1 is connected to the second switch in the stack.

In this example, the Management network runs on vSwitch0 as active on vmnic0 and as standby on vmnic1. The vMotion network runs on vSwitch0 as active on vmnic1 and as standby on vmnic0. Each port group has a VLAN ID assigned and runs dedicated on its own physical network adaptor. Only in the case of a failure is it switched over to the standby network adaptor. Failback is set to “no” because in the case of physical switch failure and restart, ESXi might falsely recognize that the switch is back online when its ports first come online. In reality, the switch might not be forwarding on any packets until it is fully online. However, when failback is set to “no” and an issue arises, both your management network and vMotion network will be running on the same network adaptor and will continue running until you manually intervene.

Datastore Heartbeats

HA has been build up from the ground in vSphere 5, Datastore Heartbeats is one of the new features. Storage heartbeats are used when the management network is unavailable to enable a slave to communicate with a master. This provides an additional level of redundancy for internode communication.

By default, vCenter will automatically select two datastores to use for storage heartbeats. An algorithm designed to maximize availability and redundancy of the storage heartbeats selects these datastores. This algorithm attempts to select datastores that are connected to the highest number of hosts. It also attempts to select datastores that are hosted on different storage arrays/NFS servers. A preference is given to VMware vSphere VMFS–formatted datastores, although NFS-hosted datastores can also be used.

Although users can manually select the datastores to be used for storage heartbeating, it is not recommended, because having vCenter automatically select and update the storage heartbeat datastores reduces management tasks. It also provides more flexibility for the system to adapt to unplanned storage outages.

Figure 9

Network Partitions

When a management network failure occurs for a vSphere HA cluster, a subset of the cluster’s hosts might be unable to communicate over the management network with the other hosts. Multiple partitions can occur in a cluster.

A partitioned cluster leads to degraded virtual machine protection and cluster management functionality. Correct the partitioned cluster as soon as possible.

  • Virtual machine protection. vCenter Server allows a virtual machine to be powered on, but it is protected only if it is running in the same partition as the master host that is responsible for it. The master host must be communicating with vCenter Server. A master host is responsible for a virtual machine if it has exclusively locked a system-defined file on the datastore that contains the virtual machine’s configuration file.
  • Cluster management. vCenter Server can communicate with only some of the hosts in the cluster, and it can connect to only one master host. As a result, changes in configuration that affect vSphere HA might not take effect until after the partition is resolved. This failure could result in one of the partitions operating under the old configuration, while another uses the new settin

Other references:

  • A

Configure HA related alarms and monitor an HA cluster

Official Documentation:
vSphere Availability Guide, page 30

Summary:
Several default vSphere HA alarms are available.

  • Insufficient failover resources (a cluster alarm)
  • Cannot find master (a cluster alarm)
  • Failover in progress (a cluster alarm)
  • Host HA status (a host alarm)
  • VM monitoring error (a virtual machine alarm)
  • VM monitoring action (a virtual machine alarm)
  • Failover failed (a virtual machine alarm)

NOTE The default alarms include the feature name, vSphere HA.

Figure 10 – vSphere HA Alarms

You also need the Cluster Validity. A valid cluster is one in which the admission control policy has not been violated.

A cluster enabled for vSphere HA becomes invalid (red) when the number of virtual machines powered on exceeds the failover requirements, that is, the current failover capacity is smaller than configured failover capacity. If admission control is disabled, clusters do not become invalid.

The cluster’s Summary tab in the vSphere Client displays a list of configuration issues for clusters. The list explains what has caused the cluster to become invalid or overcommitted (yellow).

Figure 11 – Cluster issues
DRS behaviour is not affected if a cluster is red because of a vSphere HA issue.

Other references:

  • A

Create a custom slot size configuration

Official Documentation:
vSphere Availability Guide, Page 17 and 28

Summary:
If your cluster contains any virtual machines that have much larger reservations than the others, they will distort slot size calculation. To avoid this, you can specify an upper bound for the CPU or memory component of the slot size by using the das.slotcpuinmhz or das.slotmeminmb advanced attributes, respectively

Figure 12

Figure 12, original

Figure 13

One powered-on VM has a memory reservation of 3072 MB.

Figure 14

Adjusting Maximum memory Slot size to 256 MB.

Figure 15

New Slot Size, unfortunately, still shortage of available slots

Other references:

  • A

Understand interactions between DRS and HA

Official Documentation:
vSphere Availability Guide, page 15

Summary:
Using vSphere HA with Distributed Resource Scheduler (DRS) combines automatic failover with load balancing. This combination can result in a more balanced cluster after vSphere HA has moved virtual machines to different hosts.

When vSphere HA performs failover and restarts virtual machines on different hosts, its first priority is the immediate availability of all virtual machines. After the virtual machines have been restarted, those hosts on which they were powered on might be heavily loaded, while other hosts are comparatively lightly loaded. vSphere HA uses the virtual machine’s CPU and memory reservation to determine if a host has enough spare capacity to accommodate the virtual machine.

If you are using VM-Host affinity rules that are required, be aware that these rules cannot be violated. vSphere HA does not perform a failover if doing so would violate such a rule.

Other references:

  • A

Analyze vSphere environment to determine appropriate HA admission control policy

Official Documentation:
vSphere Availability Guide, page 21.

Summary:
You should choose a vSphere HA admission control policy based on your availability needs and the characteristics of your cluster. When choosing an admission control policy, you should consider a number of factors.

  • Avoiding Resource Fragmentation
    Resource fragmentation occurs when there are enough resources in aggregate for a virtual machine to be failed over. However, those resources are located on multiple hosts and are unusable because a virtual machine can run on one ESXi host at a time. The Host Failures Cluster Tolerates policy avoids resource fragmentation by defining a slot as the maximum virtual machine reservation.
    The Percentage of Cluster Resources policy does not address the problem of resource fragmentation.
    With the Specify Failover Hosts policy, resources are not fragmented because hosts are reserved for failover.
  • Flexibility of Failover Resource Reservation
    Admission control policies differ in the granularity of control they give you when reserving cluster resources for failover protection.
    The Host Failures Cluster Tolerates policy allows you to set the failover level as a number of hosts.
    The Percentage of Cluster Resources policy allows you to designate up to 100% of cluster CPU or memory resources for failover.
    The Specify Failover Hosts policy allows you to specify a set (>=1) of failover hosts.
  • Heterogeneity of Cluster
    Clusters can be heterogeneous in terms of virtual machine resource reservations and host total resource capacities.
    In a heterogeneous cluster, the Host Failures Cluster Tolerates policy can be too conservative because it only considers the largest virtual machine reservations when defining slot size and assumes the largest hosts fail when computing the Current Failover Capacity.
    The other two admission control policies are not affected by cluster heterogeneity.

Other references:

  • A

Analyze performance metrics to calculate host failure requirements

Official Documentation:
vSphere Availability Guide

Summary:
Even on the Cluster level is a Performance tab available. In the Overview mode, three views are available:

  • Home (CPU and Memory metrics on the Cluster level)
  • Resource Pools & Virtual Machines (Detailed CPU and Memory metrics on RP, ESXi host and VM level)
  • Hosts (Detailed CPU and Memory metrics per ESXi host)

Figure 16

The Advanced View has a section on Cluster Services. Three metrics are available, from which two can be selected at a time:

  • Effective CPU resources, Total available CPU resources of all hosts within a cluster;
  • Effective Memory resources, Total amount of machine memory of all hosts in the cluster that is available for use for the VM memory and overhead memory;
  • Current Failover level, the vSphere HA number of failures that can be tolerated.

The last metric is a good indication.

Other references:

  • A

Analyze Virtual Machine workload to determine optimum slot size

Official Documentation:
vSphere Availability Guide

Summary:
See first topic. Slot Size is part of the “Host failure Cluster Tolerates ” admission control policy. The slot size is compromised of two components, CPU and Memory. CPU and Memory Reservations do have a huge impact on the calculated slot size. Two examples from the same cluster:

Figure 17 – before adjustment

Figure 18 – after adjustment

In Figure 16, the slot size is dominated by a single VM with a 16 GB Memory reservation. The affected VM was moved to a resource pool with a Memory reservation and the Memory reservation on the VM was removed. Figure 17 shows the new slot size calculation.

Other references:

  • A

Analyze HA cluster capacity to determine optimum cluster size

Official Documentation:
vSphere Availability Guide

Summary:
See the previous two topics. Other guidelines can be found in the vSphere Availability Guide Deployment Best Practices.

  • Cluster size, minimum # hosts is 2, maximum is 32.
  • Build cluster of identical servers, otherwise consider building multiple clusters of identical servers
  • Smaller-sized clusters require a larger relative percentage of the available cluster resources to be set aside as reserve capacity to adequately handle failures.
    For example, for a cluster of three nodes to tolerate a single host failure, about 33 percent of the cluster resources will be reserved for failover. A 10-node cluster requires that only 10 percent be reserved.
  • As cluster size increases, so does the HA management complexity of the cluster. This complexity is associated with general configuration factors as well as ongoing management tasks such as troubleshooting. This increase in management complexity, however, is overshadowed by the benefits a large cluster can provide. Features such as DRS and Distributed Power Management (DPM) become very compelling with large clusters. In general, it is recommended that creating the largest clusters possible to reap the full benefits of these solutions.
  • If possible, all hosts in a cluster run the same and the latest edition of ESXi. Otherwise read the guidelines if you have a mixed Cluster.

Other references:

  • In case you want to check your Cluster Capacity using PowerShell, read this post by Jonathan Medd.
Advertisements

2 Responses to VCAP5-DCA Objective 4.1 – Implement and maintain complex VMware HA solutions

  1. […] Create the HA cluster and add hosts (see: Objective 4.1) […]

  2. Definitely had quite a few back and forths here on using the percentage resource vs the host failure cluster tolerates for HA failover. In the end, we went with host failure cluster tolerates as it did force us to make sure we were aware of our memory and CPU reservations of all our VMs. Nice detailed post here!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: