VCAP5-DCA Objective 3.1 – Tune and Optimise vSphere performance

Objectives

  • Tune ESXi host memory configuration
  • Tune ESXi host networking configuration
  • Tune ESXi host CPU configuration
  • Tune ESXi host storage configuration
  • Configure and apply advanced ESXi host attributes
  • Configure and apply advanced Virtual Machine attributes
  • Configure advanced cluster attributes
  • Tune and optimize NUMA controls

Tune ESXi host memory configuration

Official Documentation:
vSphere Resource Management Guide,
Chapter 5, Memory Virtualization Basics, Page 25 and also
Chapter 6, Administering Memory Resources, Page 29

vSphere Monitoring and Performance Guide, Chapter 1, Monitoring Inventory Objects with Performance Charts, Section “Solutions for memory Performance Problems”, Page 19. This section contains some information on troubleshooting memory issues

Performance Best Practices for VMware vSphere 5.0, Chapter 2 ESXi and Virtual Machines, Section ESXi Memory Considerations, page 25

Summary:
Chapter 5 Memory Virtualization Basics of the vSphere Resource Management Guide explains the concepts of memory resource management. It is very useful to know what happens when turning the knobs…

Some Highlights to test your knowledge

  • Know the difference between shares, reservations and limits;
  • What is memory Over commitment
  • The principles of Software-Based Memory Virtualization vs. Hardware-Assisted Memory Virtualization;
  • Is Hardware-Assisted Memory Virtualization always better?

Chapter 6 Administering Memory Resources discusses subjects like:

  • Understanding Memory Overhead
  • How ESXi Hosts Allocate Memory
    Details on the use of Limits, Reservations, Shares and Working Set Size
  • VMX Swap files
    To avoid more confusion, in a few words:
    ESXi reserves memory per virtual machine for a variety of purposes. Memory for the needs of certain components, such as the virtual machine monitor (VMM) and virtual devices, is fully reserved when a virtual machine is powered on. However, some of the overhead memory that is reserved for the VMX process can be swapped. The VMX swap feature reduces the VMX memory reservation significantly (for example, from about 50MB or more per virtual machine to about 10MB per virtual machine).The host creates VMX swap files automatically, provided there is sufficient free disk space at the time a virtual machine is powered on.
  • Memory Tax for Idle Virtual Machines
    You can modify the idle memory tax rate with the Mem.IdleTax option. Use this option, together with the Mem.SamplePeriod advanced attribute, to control how the system determines target memory allocations for virtual machines
  • Memory Reclamation
  • Using Swap Files
    By default, the swap file is created in the same location as the virtual machine’s configuration file. However it is possible to specify a datastore stored locally on a host. This is a two step process. First, adjust the Cluster settings:

Figure 1

  • Second step for all hosts in the Cluster

Figure 2

  • Another important subject is the relation between Swap Space and Memory Overcommitment.
    You must reserve swap space for any unreserved virtual machine memory (the difference between the reservation and the configured memory size) on per-virtual machine swap files. This swap reservation is required to ensure that the ESXi host is able to preserve virtual machine memory under any circumstances. In practice, only a small fraction of the host-level swap space might be used. If you are overcommitting memory with ESXi, to support the intra-guest swapping induced by ballooning, ensure that your guest operating systems also have sufficient swap space. This guest-level swap space must be greater than or equal to the difference between the virtual machine’s configured memory size and its Reservation.
    Settings on a individual VM.

Figure 3

  • Swapping to Host Cache
    A new vSphere 5 feature
    Datastores that are created on solid state drives (SSD) can be used to allocate space for host cache.
    The host reserves a certain amount of space for swapping to host cache.

Figure 4

  • Sharing Memory Across Virtual Machines
    Use the Mem.ShareScanTime and Mem.ShareScanGHz advanced settings to control the rate at which the system scans memory to identify opportunities for sharing memory. You can also disable sharing for individual virtual machines by setting the sched.mem.pshare.enable option to FALSE (this option defaults to TRUE).
    ESXi memory sharing runs as a background activity that scans for sharing opportunities over time. The amount of memory saved varies over time. For a fairly constant workload, the amount generally increases slowly until all sharing opportunities are exploited. To determine the effectiveness of memory sharing for a given workload, try running the workload, and use resxtop or esxtop to observe the actual savings. Find the information in the PSHARE field of the interactive mode in the Memory page.
  • Memory Compression
    Memory Compression is enabled by default, but can be disabled

Figure 5

  • Mem.MemZipMaxPct. controls the size of the compression cache as a percentage of the memory size of the virtual machine
  • Measuring and Differentiating Types of Memory Usage
    Is good reading when interpreting the Memory Performance graphics and explains the difference between Memory Granted and Memory Consumed
  • Memory Reliability

Performance Best Practices for VMware vSphere 5.0, Chapter 2 ESXi and Virtual Machines, Section ESXi Memory Considerations is good reading and continues where the previous guide ends.

Subjects are:

  • Memory Overhead and Sizing.
  • Memory Overcommit Techniques presents a nice overview in which order different techniques will be used:
    • Page Sharing
    • Ballooning
    • Memory Compression
    • Swap to Host Cache
    • Regular Swapping
  • Large Memory Pages for Hypervisor and Guest Operating System
  • Hardware-Assisted MMU Virtualization

Other references:

  • A

Tune ESXi host networking configuration

Official Documentation:
vSphere Networking Guide, read Chapter 7, Networking Best Practices.

vSphere Monitoring and Performance Guide, Chapter 1, Monitoring Inventory Objects with Performance Charts, Section “Solutions for poor Network Performance ”, Page 21. This section contains some information on troubleshooting network performances issues

Performance Best Practices for VMware vSphere 5.0, Chapter 2 ESXi and Virtual Machines, Section ESXi Networking Considerations, page 34

Summary:
The Performance Best Practices for VMware vSphere 5.0 document contains a few important general considerations. One – I was not fully aware – is this one:

“In a native environment, CPU utilization plays a significant role in network throughput. To process higher levels of throughput, more CPU resources are needed. The effect of CPU resource availability on the network throughput of virtualized applications is even more significant. Because insufficient CPU resources will limit maximum throughput, it is important to monitor the CPU utilization of high-throughput workloads.”

Other subjects:

  • a section on Network I/O Control. See Objective 2.4.
  • DirectPath I/0. See Objective 1.1
  • SplitRx Mode. a new feature in ESXi 5.0, uses multiple physical CPUs to process network packets received in a single network queue. Only supported in VMXNET3 virtual network adapters
  • Running Network Latency Sensitive Applications

Other references:

  • A

Tune ESXi host CPU configuration

Official Documentation:
vSphere Resource Management Guide,
Chapter 3, CPU Virtualization Basics, Page 15 and also
Chapter 4, Administering CPU Resources, Page 17

Performance Best Practices for VMware vSphere 5.0, Chapter 2 ESXi and Virtual Machines, Section ESXi CPU Considerations, page 19

Summary:
The vSphere Resource Management Guide chapter 3, explains the basics of:

  • Software-Based CPU Virtualization
  • Hardware-Assisted CPU Virtualization

The final note in this section is very important, as we tend to hand-over more resources than often needed:
Deploy single-threaded applications on uniprocessor virtual machines, instead of on SMP virtual machines, for the best performance and resource use.

Single-threaded applications can take advantage only of a single CPU. Deploying such applications in dualprocessor virtual machines does not speed up the application. Instead, it causes the second virtual CPU to use physical resources that other virtual machines could otherwise use.

Chapter 4 goes in to detail on subjects like:

  • Multicore Processors (my simple rule of the thumb: More Cores is more CPU Cycles)
  • Hyperthreading, sometimes confusing, as Hyperthreading is not the same as Multicore Technology. Instead Hyperthreading allows a single core to process two independent applications (under certain circumstances). The best analogy I have ever read, is the comparison with a wide road. Two small vehicles can use the road alongside each other, but a wide vehicle needs the whole road, and a second vehicle has to wait for its turn.
  • VMware advises to enable Hyperthreading at all times. Check the Processor model and the BIOS setting. Certain features are often disabled on arrival.

On the Virtual Machine Level, you can set the hyperthreaded core sharing mode:

Figure 6

Possible values:

Any
Default setting for all virtual machines on a hyperthreaded system. The virtual CPUs of a virtual machine with this setting can freely share cores with other virtual CPUs from this or any other virtual machine at any time.

None
Virtual CPUs of a virtual machine should not share cores with each other or with virtual CPUs from other virtual machines. That is, each virtual CPU from this virtual machine should always get a whole core to itself, with the other logical CPU on that core being placed into the halted state.

Internal
This option is similar to none. Virtual CPUs from this virtual machine cannot share cores with virtual CPUs from other virtual machines. They can share cores with the other virtual CPUs from the same virtual machine.

Using CPU affinity
Be careful with this one. VMware presents potential issues when using this option

  • For multiprocessor systems, ESXi systems perform automatic load balancing. Avoid manual specification of virtual machine affinity to improve the scheduler’s ability to balance load across processors.
  • Affinity can interfere with the ESXi host’s ability to meet the reservation and shares specified for a virtual machine.
  • Because CPU admission control does not consider affinity, a virtual machine with manual affinity settings might not always receive its full reservation.
    Virtual machines that do not have manual affinity settings are not adversely affected by virtual machines with manual affinity settings.
  • When you move a virtual machine from one host to another, affinity might no longer apply because the new host might have a different number of processors.
  • The NUMA scheduler might not be able to manage a virtual machine that is already assigned to certain processors using affinity.
  • Affinity can affect the host’s ability to schedule virtual machines on multicore or hyperthreaded processors to take full advantage of resources shared on such processors.

The Performance Best Practices for VMware vSphere 5.0 contains some useful general considerations and goes into detail on the following subjects:

  • UP vs. SMP
  • Hyper-Threading
  • Non-Uniform Memory Access (NUMA)
    Remember while adjusting the BIOS:

    • node interleaving is disabled, ESXi detects the system as NUMA and applies NUMA optimizations.
    • node interleaving is enabled, ESXi does not detect the system as NUMA.
  • Configuring ESXi for Hardware-Assisted Virtualization, comes to this question:

Figure 7

  • Automatic allows ESXi to determine the best choice. This is the default;
    http://communities.vmware.com/docs/DOC-9882  provides a detailed list of which VMM is chosen for each combination of CPU and guest operating system.
  • Host Power Management in ESXi
    As in the Resource Mangement Guide, available Options are discussed.

    • High performance
      This power policy maximizes performance, using no power management features.
    • Balanced
      This power policy (the default in ESXi 5.0) is designed to reduce host power consumption while having little or no impact on performance.
    • Low power
      This power policy is designed to more aggressively reduce host power consumption at the risk of reduced performance.
    • Custom
      This power policy starts out the same as Balanced, but allows for the modification of individual parameters.
    • Be sure, also, that your server’s BIOS settings are configured correctly. Preferably in “OS Controlled ” mode to let ESXi take control.

Figure 8

Other references:

  • A

Tune ESXi host storage configuration

Official Documentation:
vSphere Resource Management Guide,
Chapter 7, Managing Storage I/O Control, Page 39.

vSphere Monitoring and Performance Guide, Chapter 1, Monitoring Storage Resources, Page 31. This section contains information on using Storage Reports and Storage Maps

Performance Best Practices for VMware vSphere 5.0, Chapter 2 ESXi and Virtual Machines, Section ESXi Storage Considerations, page 30.

Summary:
in a few words, Storage I/O Control is a big improvement on the shares mechanism. This is best explained in this graphic from VMware:

Figure 9 – Graphic by VMware

Requirements:

  • Datastores that are Storage I/O Control-enabled must be managed by a single vCenter Server system.
  • Storage I/O Control is supported on Fibre Channel-connected, iSCSI-connected, and NFS-connected storage. Raw Device Mapping (RDM) is not supported.
  • Storage I/O Control does not support datastores with multiple extents
  • Check whether your automated tiered storage array has been certified to be compatible with Storage I/O Control

SIOC must be enabled on the Datastore level.

Figure 10

The final step is to set the number of shares and Maximum number of IOPS per VM. Default settings are:

  • Normal, 1.000 shares;
  • Unlimited IOPS

“If the limit you want to set for a virtual machine is in terms of MB per second instead of IOPS, you can convert MB per second into IOPS based on the typical I/O size for that virtual machine. For example, to restrict a backup application with 64KB IOs to 10MB per second, set a limit of 160 IOPS.”
The Maths: 64 KB * 160 = 10240 /1024 = 10 MB/s.

Figure 11

Final step, monitor Performance

Figure 12

The Performance Best Practices for VMware vSphere 5.0 has topics on:

  • VAAI
  • LUN Access Methods, Virtual Disk Modes, and Virtual Disk Types
  • Partition Alignment
  • SAN Multipathing
  • Storage I/O Resource Allocation
  • Running Storage Latency Sensitive Applications

Other references:

Configure and apply advanced ESXi host attributes

Official Documentation:
vSphere Resource Management Guide,
Chapter 14, Advanced Attributes, Page 101 describes how to set advanced Host attributes and Virtual Machine Attributes.

Summary:
The easiest way is using the vSphere Client. Select a host and go to Configuration, Software and Advanced Settings.

Chapter 14 presents an overview of the following categories:

  • Advanced Memory Attributes
  • Advanced NUMA Attributes
  • Advanced Virtual NUMA Attributes

Advanced attributes can also be set with the vSphere CLI. Use command: esxcfg-advcfg.
To get an overview type:

# esxcfg-advcfg –l

Other references:

  • A

Configure and apply advanced Virtual Machine attributes

Official Documentation:
vSphere Resource Management Guide,
Chapter 14, Advanced Attributes, Page 101 describes how to set advanced Host attributes and Virtual Machine Attributes.

Summary:
The easiest way is using the vSphere Client. Select a VM, Edit Settings and go to Options, Advanced, General and Configuration Parameters.

Figure 13

Other references:

  • A good step-by

Configure advanced cluster attributes

Official Documentation:

Summary:
On the Cluster Level, after enabling vSphere HA and/or vSphere DRS, Advanced attributes can be configured on the vSphere HA level and on the vSphere DRS level. On both levels you will find an Advanced Potions.. button which allows you to add and edit custom parameters.

Figure 14

Other references:

  • A

Advertisements

2 Responses to VCAP5-DCA Objective 3.1 – Tune and Optimise vSphere performance

  1. […] Summary: See my notes on Objective 3.1, section “Tune ESXi host CPU configuration”. […]

  2. […] should understand how ESXi handles Memory and Memory Overcommit Techniques, see also Objective 3.1. Recommended reading is “Understanding Memory Resource Management in VMware ESX […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: