The shared storage of the VMware cluster was based on iSCSI technology, in fact a couple of Dell PowerVault MD3000i boxes. In that case Veeam recommends for data retrieval the “Direct SAN access” mode. The operating system of the backup server is Microsoft Windows 2008 R2 Standard edition. The server has two NICs available for accessing the iSCSI storage.

My first attempt to configure the Windows 2008 R2 iSCSI initiator did not match my expectations. During the first run of Veeam Backup & Replication, LUNS configured for Storage Controller 1 were moved to Storage Controller 0, causing the famous Warnings “LUN not on preferred path”.
Luckily, I found this article on the Dell Storage Wiki, which helped me into the right direction (Thank you very much for that!). The article was written for Windows Server 2003 R2 x_64. In this post, I will step through the process for Windows 2008 R2.

Step 1

Windows 2008 R2 comes with the iSCSI initiator already installed, which is fine. Do not; I repeat, Do not (as I did during my first attempt) install the Microsoft Windows MPIO. The next screenshot shows the “Add Features Wizard”. This screenshot has been taken after finishing Step 2 (that explains why the “Multipath I/O” option has been greyed out).

Step 2

Now, you will need the Dell MD300i iSCSI software, you will need the “PowerVault MD300i Resource CD”. A recent version can be found here.

Start the installation. If you also want to install the “Modular Disk Storage Manager”, choose “Typical (Full Installation)”. If you only need the MPIO drivers, choose “Host”.

Curious to see what happened? Go to the Control Panel and open the MPIO feature, it looks like this; a number of Dell devices are now supported.

Step 3

From the Control Panel, you can start the iSCSI initiator, go to the Configuration tab and copy the Initiator Name for later use.

Step 4

Now it is time to give the Windows 2008 R2 server access to the MD3000i. Start the “Dell Modular Disk Storage Manager”, go to the Configure Tab and Add the Windows host. Specify the hostname and OS type.

Add the iSCSI initiator of the host; enter the Initiator name from step 3.

In my case, there is already a Hostgroup available, to provide access to the VMware vSphere servers. Because the Windows server functions as a backup server, I added the newly created host to the same group.

Step 5

Return to the Windows server and the iSCSI initiator. Go to the Discovery tab. Click “Discover portal” and add the first of four IP addresses of the MD3000i. Repeat this step for the remaining addresses.

Step 6

Now, go to the Targets tab, the MD3000i shows up under Discovered Targets.

The status of the Discovered Target is Inactive. Click Connect, as a result the “Connect To Target” windows appear. Make sure both options are enabled.

Click the Advanced button, the “Advanced Settings“  window shows up.

In the Connect Using section, under Local adapter, choose “Microsoft iSCSI initiator” and under “Initiator IP” and “Target portal IP” supply a matching combination, for instance:  192.168.100.76 (host) and 192.168.100.11 (MD3000i).

Repeat this step for the other 3 sessions.

Step 7

Return to the Targets tab. The status of the MD3000i is now Connected.

Click the “Properties” button, it will show the four sessions.

Click the “Devices” button.

In this example the MD3000i has 2 LUNS available (LUN11 and LUN12). LUN31 is for administrative purposes and does not show up because it is Disk -1 (compare with Disk 2 and Disk 3). Select the first Device (Disk 2, LUN11) and click the MPIO button.

Out-of-the-box, the “Load balance policy” is “Least Queue Depth”. Dell recommends the “Round Robin With Subset” policy. Note that two paths corresponding to the MD3000i controller that currently owns the LUN are Active; the other ones are in Standby mode.

The Details button shows the path details. You are already familiar with that.

Repeat this step for the other Devices.

Thank you for reading, I welcome your comments.

This year Eric Sloof and Mattias Sundling presented their famous “Mythbusters goes Virtual” session. You can see a sneak preview here. In the session some old habits are presented, the question is, can you do better with vSphere 5?

For me, the most surprising fact of this session is the recommendation to configure a VM with the VMware Paravirtual SCSI (PVSCSI) adapter instead of the LSI Logic Controller. At the end of the session there was an audience question how to reconfigure a VM with the PVSCSI adapter. So let’s find out for myself.

VMware has published KB 1010398 Configuring disks to use VMware Paravirtual SCSI (PVSCSI) adapters. The KB states that “PVSCSI adapters are high-performance storage adapters that can result in greater throughput and lower CPU utilization. PVSCSI adapters are best suited for environments, especially SAN environments, where hardware or applications drive a very high amount of I/O throughput. PVSCSI adapters are not suited for DAS environments.” A matrix supported Operating Systems and instructions how to configure a PVSCSI adapter is also included in the KB.

Reconfigure a Windows 2008 R2 VM

This is the original configuration of a Windows 2008 R2 VM. Notice that the SCSI adapter is LSI Logic. In case we just changed the adapter type, the VM won’t boot. So that would not help us much. To change the adapter, we must use a trick. We will add a “helper” disk.

• Add a disk
• “Select a Disk” window, choose “Create a new Virtual Disk”,
• “Create a Disk” windows, leave everything default
• Under “Advanced Options”, under “Virtual Device Node”, default the next available ID on Controller 0 will be used. Now select Node 1:0.
  By choosing 1:0, a new SCCI controller will appear.
• leave the other options to their default and Complete the creation.
• As a result, this time, a new SCSI Controller has been added. Now we need to change the type from LSI Logic SAS to VMware Paravirtual

  

•  Select Change Type and change to “VMware Paravirtual”.

  

• Results in this…

  

• Now boot the VM and check the hardware. In the image, you see both storage controllers.
• Shut down and return to the configuration.

  
  Now change the type for SCSI controller 0 and change to “VMware Paravirtual”.

• Boot the VM

  

• Final step, from a running Windows 2008 R2 VM, you can remove the temporarily added Hard disk and SCSI controller 1.
• remove Hard Disk 3. Don’t forget to select “Remove from virtual machine and delete files from disk”.

  

• Last step is to reove SCSI Controller 1.
• Finished.

  

Creating a new Windows 2008 R2 server

The best option is to include the VMware Paravirtual SCSI (PVSCSI) adapter in a template. In case you would like a fresh Windows 2008 R2 server with the Paravirtual adapter included, follow these steps.

Create a new Windows 2008 R2 VM. After creation, Edit the hardware:

• Change the SCSI controller type to Paravirtual;
• Add  a Floppy drive

Go to the Floppy device and check “Connect at Power On”.

Select “Use existing floppy image in datastore:”, browse to folder vmimages, floppies and select the correct floppy image, in this case: pvscsi-Windows2008.flp.

Boot the newly created VM and proceed to the “Where do you want to install Windows?” page. Notice that there’s no disk available.

Select “Load Driver” and select the floppy drive.

Select the driver and…

…if everything goes well, now the install disk is present. Now you can continue the installation.

I want to thank the VMware communities forum for providing this info. As always, I hope this post was useful, thank you for reading and do not hesitate to contact me.

Sometime ago, I encountered a situation where (due to circumstances) production servers were running on old, unsupported hardware. No need to tell you that the customer and the support team were very uncomfortable with this situation. The number one priority was to get the production servers on supported hardware. The organization is also in a process of transformation, a forthcoming project is to migrate to a Windows 2008 R2 platform, new Directory Services etc.

So, it was decided to build a new infrastructure based on VMware vSphere, new servers and a SAN. To resolve priority number one, the advise was to go for a Physical-to-Virtual (P2V) conversion.

The first product that comes into mind is the VMware product, officially “VMware vCenter Converter Standalone”, (from now on, “Converter Standalone”). The official link is here.

Today the VMware Converter is a free product. If you haven’t already done so, register with VMware and you can download the VMware Converter. The VMware Converter is available in these editions: 5.0, 4.3, 4.0.1 and 4.0.

The “Resources” tab leads you to the documentation. The User’s Guide is a comprehensive document that describes every detail about the Converter Standalone and its usage. Chapter 1 also includes an interesting topic on types of Data Cloning Operations. In short, there are 2 options:

• Hot cloning, also called live- or online cloning, which requires the physical server running its operating system during the converting process.
• Cold cloning, also called offline cloning, which requires the physical server not running its operating system. When you cold clone a server, you reboot the source machine using a CD that has its own operating system and the Converter Standalone application on it.

The topic concludes with an Hot and Cold Cloning Comparison.

There is one important caveat , the before mentioned boot CD is not free, in fact you will at least need a vSphere Enterprise license to download it. What you also need to know is that Converter Standalone 4.3 and later do not support cold cloning, you must use the Boot CD of an earlier edition.

An alternative could be the MOA project on http://sanbarrow.com/moa.html which offers a replacement for the VMware Boot CD.

Hot or Cold Cloning?

How do we answer this question? I think it the answer is, “it depends”. Until this project Hot cloning was my favourite method. Very useful is the VMworld Session “VMware Converter 3.0.2. Troubleshooting” which you can download from here.

In this project we had to take into account the following topics:

• Two Windows Domain Controllers should be converted. Best practice is to create VMs and build new Domain Controllers and demote the physical one’s. Unfortunately these Domain Controllers have more roles like file and application server (not best practice, I know). In that case there is only one option: Cold Clone.
  Very important, after conversion and booting the virtualized Domain Controller, never, ever reconnect the physical Domain Controller or you will be in great trouble. Read more on this topic in this excellent blog post.
• The physical servers reside in a different VLANs than the management network of the ESXi servers and the vCenter server. Both methods require some modification. Using the hot clone method, requires a modification to allow traffic between VLANs (the network admin does not like). Another option for hot cloning is to reconfigure the switch port for the management VLAN, this also requires changing the IP address of the NIC.
• Another customer requirement was, minimal configuration changes on the physical servers and easy fall back in case the conversion does not succeed.

The final solution was: use the cold clone method, one NIC of the physical server (in most cases a spare NIC) was attached to an available switch port in the management VLAN. This method met the requirements of the customer:

• Minimal configuration changes. No additional software configuration on the server (a hot clone installs an agent during the process). No changes in the network configuration of the physical server.
• Easy fall back, just remove the connection to the NIC used for the conversion, remove the boot CD and reboot the server.

What did we learn?

As always, preparation is important.

• Prepare the extra switch ports used during the conversion.
• Check the NICs in the physical servers,  support 1 Gb/s.
• Does the physical servers have a console or remote access card? You will need a keyboard and monitor using a boot CD.
• Does the physical servers have a CD-ROM?
• If your server supports booting from an USB CD-ROM drive, have a drive available.
• Does  the drive CD-ROM works fine? One 8 year old drive succeeded to boot the CD after 8 reboots…
• Prepare the Converter Boot CDs.
• On eaxh server, make a note of the network settings. Use these commands.
  ipconfig /all > network.txt
  route print >> network.txt
• Check the health of the Windows Domain before you start the conversion process. Check again after the conversion of each Domain Controller. For instructions, see this post, or on http://www.petri.co.il
• On the domain controllers, create an extra System State backup, using NTbackup and store locally (In case things go really bad).

During the conversion process

• When the Converter reports that “No OS has been found”, changes are that the RAID controller of the physical server has not been recognized, for instance some Dell SAS controllers.
  The good news, a few moments after booting the Boot CD, you can press F6 and add additional drivers. I used an extra USB floppy drive to solve this.
• Does the physical server has a small System drive? During the conversion process make sure every partition will be converted to a separate disk. You can also expand volumes during the conversion process. Do not convert hardware vendor’s management partitions.
• One of the important steps after converting a physical server is to clean up the newly created VM. Especially remove things like RAID management software, Dell Open Manage etc. Kendrick Coleman at http://www.kendrickcoleman.com provides an excellent collection of tools for cleaning, bundled in an .iso image. Upload to your environment, mount the .iso and go. You can find it here.
• Do not panic in case something goes wrong.

To conclude, the Converter Standalone is a very good product for converting physical servers to virtual. Personally, I regret that the Boot CD has been discontinued, especially since the latest version 5.x has some very interesting features, like partition aligning. As always, if you have any questions, please respond. Happy reading.

While writing this post, I stumbled onto some BIOS settings, I did not fully understand. That makes it difficult to choose the correct settings. In this post, I will show you how to deal with some “CPU Power Management” settings on a HP ML110 G6 server.

Before you start, make sure which CPU’s are supported in the server, some BIOS settings are not relevant for your CPU. Example: the ML110 G6 supports[1] Intel Xeon X3430, X3450, X3460, In the BIOS under “Advanced Processor Options” there is the “Processor Hyper-Threading” option. The X3430 does not support Hyperthreading, which makes this option irrelevant. However the other models do support Hyperthreading.

The first question is, which processor is installed? A HP ML110 G6 will show the CPU type during the boot process. In case you have VMware ESXi already installed, you can also determine the CPU type from the vSphere Client, under Configuration.

Now it is time getting to know your processor. Today, both Intel and AMD have a wide range of CPUs with many different options. In this example, the ML110 G6 is equipped with an Intel Xeon X3430 CPU. With a little help from Google, you will find this information from Intel:

Wikipedia provides some compact background information:

Now it is time for some advanced setting. In this post, I have already explained some of the “Advanced Processor Options”. Now let’s have a look at the “CPU Power Management” settings.

BIOS => Advanced => Advanced Processor Options => CPU power Management

EIST (GV3)

EIST is the abbreviation of: Enhanced Intel SpeedStep.  According to Wikipedia, “SpeedStep is a trademark for a series of dynamic frequency scaling technologies (codenamed Geyserville and including SpeedStep, SpeedStep II, and SpeedStep III) built into some Intel microprocessors that allow the clock speed of the processor to be dynamically changed (to different P-states) by software. This allows the processor to meet the instantaneous performance needs of the operation being performed, while minimizing power draw and heat dissipation.”
By the way, I was not able to find out what GV3 means, anyone out there?
The Intel X3430 does support EIST,

Turbo Mode

Turbo Mode reflects to the Intel Turbo Boost Technology, which is available on the Intel X3430.

According to Wikipedia, “Turbo Boost is a technology that enables the processor to run above its base operating frequency via dynamic control of the CPU’s “clock rate”. It is activated when the operating system requests the highest performance state of the processor. Processor performance states are defined by the Advanced Configuration and Power Interface (ACPI) specification, an open standard supported by all major operating systems; no additional software or drivers are required to support the technology. The design concept behind Turbo Boost is commonly referred to as “dynamic overclocking”

The clock rate of an Intel X3440 CPU is 2,4 GHz. Under circumstances Turbo Boost allows a core to run above this clock speed.

CPU C1E

A good summary about ACPI, Processor and Performance States is (again) this Wiki Page. In this article, the C1 and C1E state are also mentioned. “C1 (often known as Halt) is a state where the processor is not executing instructions, but can return to an executing state essentially instantaneously. All ACPI-conformant processors must support this power state. Some processors, such as the Pentium 4, also support an Enhanced C1 state (C1E or Enhanced Halt State) for lower power consumption.”
With enabling C1 Enhanced Mode, all logical processors in the physical processor have entered the C1 state, the processor will reduce the core clock frequency to system bus ratio and VID

In this blog post Intel’s Taylor Kidd explains that C states is all about Idle states and there are a lot of different C-states, like Core C-states, Processor C-states and Logical C-states. C0 is the operational state, where the CPU is doing useful work and on the other end, C3 and C6 are “deep sleep” states.

Idle states are supported on a Intel X3440.

CPU C3 Report, as the comprehensive item specific help states, this setting reflects to the core C3 state which matches the ACPI C2 state (C2. often known as Stop-Clock,  is a state where the processor maintains all software-visible state, but may take longer to wake up).

And finally, CPU C6 Report, as the comprehensive item specific help states, this setting reflects to the core C6 state which matches the ACPI C3 state (often known as Sleep, is a state where the processor does not need to keep its cache coherent, but maintains other state. Some processors have variations on the C3 state (Deep Sleep, Deeper Sleep, etc.) that differ in how long it takes to wake the processor).
So the C3 and C6 Report options allows you to enable extra sleep states.

And finally, the question remains, what are the optimal BIOS settings?
Before we conclude let us have a look at this VMware document “Performance Best Practices for VMware vSphere 4.1”

The “Hardware BIOS Settings” section has a few interesting points related to the Intel X3440 CPU:

• Make sure you are running the latest BIOS for your hardware.
  OK, always a good idea imho.
• ESX (and I assume ESXi also…) supports Enhanced Intel Speedstep (EIST) CPU Power management technologies that can save power when a host is not fully utilized. However because these and other power-saving technologies can reduce multi-threaded performance and increase I/O latency in some situations, you should consider disabling them when these particular performance considerations outweigh power considerations.
• Enable “Turbo Mode” if your processor supports it.
• Disable C1E halt state in the BIOS if multi-threaded performance and I/O latency are important considerations. Consider enabling C1E if “Turbo Mode” is present and single-threaded performance of  certain workloads is more relevant. (See note above regarding performance considerations versus power considerations.)
  The explanation is that to enable “Turbo Mode ” for one core, the CPU has to shut off other cores which is only possible with C1E state enabled. This gives you the “single-threaded” performance. The downside of enabling C1E state is that a few CPU cores may shut off. When these cores are needed, there will be some I/O latency.
• For the highest performance, potentially at the expense of higher power consumption, set any BIOS power-saving options to high-performance mode. Depending on the manufacturer, this might involve disabling an option (for example, “Demand-Based Power Management”) or enabling an option (for example, “Static High Performance Mode”).

This seems to relate to this BIOS setting:

If we want optimal performance, I would go for these settings:

and…

Notice that the “Turbo Mode” option is only available if EIST is enabled.

Final thoughts, in my opinion, one should care for his environment and energy savings.
This post is meant as an exercise how to handle advanced CPU options. In real life, you probably have a cluster of vSphere servers. In that case I would recommend that you consider using the Distributed Power Management(DPM) feature and configure the individual servers for optimal performance. The win of a server shut down by DPM is much bigger than tweaking CPU sleep states and possible performance loss.

As always enjoy reading and I welcome your feedback.

Some time ago, I stumbled on this excellent blog post by Kendric Coleman. In this post he presents several scenarios how to design a vSphere host equipped with six physical NICs.

Recently, I have read a great book entitled “VMware vSphere 4.1 HA and DRS Technical deepdive”. This book is written by Duncan Epping and Frank Denneman and explains the principles behind HA, DRS and DPM. The book also offers very useful design principles, while configuring for HA and DRS. This book is a must read for every serious VMware Admin out there. What is more, even before the GA of vSphere 5, the successor “VMware vSphere 5 Clustering technical Deepdive” is already available (in fact is already on my iPad).
Especially in the first part, about VMware High Availability, they wrote down a few interesting design principles concerning network configuration.

In this post, I will try to combine both sources and create a design for an ESXi server with 6 NICs.

There are a few preconditions:

• I leave aside the topic how to distribute on-board NICs and NICs on expansion cards (the ESXi server in this example is virtual)
• Shared storage is iSCSI
• Try to fit-in FT
• Everything must be redundant, assume we do have two stacked switches for management, VMotion and LAN traffic and two stacked switches for iSCSI traffic. It is best practice to have separate switches for storage traffic. Switches have been configured correctly (trunk ports etc.)

The first part of the design is the Management network. There are two options to create a virtual switch, a vNetwork Standard Switch (vSS)or a vNetwork Distributed Switch (vDS). Besides the required license (Enterprise PLus), there is discussion if you should go for a 100% vDS solution or a hybrid approach (combination of vSS and vDS). However, for the Management network, we prefer a vSS. The other vSwitches can be a vSS or a vDS.

In a vSphere High Availability (HA) cluster, the “heartbeat” network plays a very important role and with that, the Management network in ESXi. NIC teaming provides redundancy. The preferred scenario is a Single Management Network with vmnics in Active/Standby configuration. It is also common practice to combine the Management network with the vMotion Network. This results in the following design.

Management Network
VLAN 2
Management Traffic is Enabled
vmk0: 192.168.2.53
vmnic0 Active / vmnic1 Standby
Load balancing: Use explicit failover order
Failback: No

vMotion
VLAN 21
vMotion is Enabled
vmk1: 192.168.21.53
vmnic1 Active / vmnic0 Standby
Load balancing: Use explicit failover order
Failback: No

Needless to say that vmnic0 is connected to the first switch in the stack and vmnic1 is connected to the second switch in the stack.

The second part of the design is the Storage network.Another recommendation from the HA and DRS Deepdive is avoiding the chances of a split-brain scenario. A split-brain situation can occur during a HA incident where a virtual machine is restarted on another host, while not being powered down on the original host. So for all network based storage, iSCSI included, it is recommended to create a secondary Management Network on the same vSwitch as the storage network to detect a storage outage.

As a Storage Adapter to connect to our iSCSI storage, we will use the iSCSI Software adapter. We configure two VMkernel NICs for redundancy and load balancing. Both VMkernel NICs will be assigned to the iSCSI Software adapter.

For iSCSI networks, it is also recommended to enable Jumbo frames. For more information on Jumbo frames see this link.
Unfortunately in vSphere 4.1 it is not possible to use the vSphere Client to create a virtual switch with Jumbo frames enabled. You will have to  use a CLI. In this example I used the vSphere Management Assistant (vMA)

iSCSI1
VLAN
vmk2: 192.168.50.53
vmnic2 Active / vmnic3 Unused

iSCSI2
VLAN
vmk3: 192.168.50.63
vmnic3 Active / vmnic2 Unused

Management Network2
VLAN
Management Traffic is Enabled
vmk4: 192.168.50.73
vmnic2 Active / vmnic3 Active
Load balancing: Use explicit failover order
Failback: No

The actual code. I suppose that the iSCSI Software adapter has been set up already, this can be done with the vSphere Client.

# vSwitch1
# -a add new vSwitch
/usr/bin/vicfg-vswitch -a vSwitch1

# -m set MTU value
/usr/bin/vicfg-vswitch vSwitch1 -m 9000

# -A add portgroup
/usr/bin/vicfg-vswitch -A iSCSI1 vSwitch1
/usr/bin/vicfg-vswitch -A iSCSI2 vSwitch1

# -a add VMkernel nic, requires -i IP address, -n Netmask and -p Portgroup. -m set MTU is optional
/usr/bin/vicfg-vmknic -a -i 192.168.50.53 -n 255.255.255.0 -m 9000 -p iSCSI1

/usr/bin/vicfg-vmknic -a -i 192.168.50.63 -n 255.255.255.0 -m 9000 -p iSCSI2

# -L bind physical NIC to vSwitch
/usr/bin/vicfg-vswitch vSwitch1 -L vmnic2
/usr/bin/vicfg-vswitch vSwitch1 -L vmnic3

# -N unlink physical NIC from portgroup
/usr/bin/vicfg-vswitch -p iSCSI1 -N vmnic3 vSwitch1
/usr/bin/vicfg-vswitch -p iSCSI2 -N vmnic2 vSwitch1

# add 2nd management portgroup
/usr/bin/vicfg-vswitch -A “Management Network2″ vSwitch1

# configure VMkernel nic
/usr/bin/vicfg-vmknic -a -i 192.168.50.73 -n 255.255.255.0 -m 9000 -p “Management Network2″

# To bind the virtual nics to the Software iSCSI adapter
esxcli –-server=<esxi host> –username=root –-password=<password> swiscsi nic add –n vmk2 -d <virtualHBA>

esxcli –-server=<esxi host> –username=root –-password=<password> swiscsi nic add –n vmk3 –d <virtualHBA>

# To check the Software iSCSI adapter
esxcli –-server=<esxi host> –username=root –-password=<password> swiscsi nic list <virtualHBA>

An extra recommendation from the HA and DRS Deepdive is to specify an additional Isolation address. In case you are using network based storage like iSCSI, a good choice is the IP address of the storage device, in this example: 192.168.50.11

The third and final part is the vSwitch for the Virtual Machine networks. In case your VMs run on VLANs, create a vSwitch and add a Port group for every VLAN. Label each portgroup with a name that reflects the VLAN. In this example 2 Port groups have been created. There are several options for the Loadbalancing policy. A  recommended way is the “Explicit failover order” instead of the default “originating virtual port id”

VM Network31
VLAN 31

VM Network32
VLAN 32

All adapter Active/Active
Load balancing: Use explicit failover order

What is missing in this design so far is Fault Tolerance (FT). It is recommended to have FT on a separate network. One possibility is to add two extra physical NICs and create an extra vSwitch. With 6 NICs, I consider this also possible

Management Network
VLAN 2
Management Traffic is Enabled
vmk0: 192.168.2.53
vmnic0 Active / vmnic1 Standby
Load balancing: Use explicit failover order
Failback: No

vMotion
VLAN 21
vMotion is Enabled
vmk0: 192.168.21.53
vmnic1 Active / vmnic0 Standby
Load balancing: Use explicit failover order
Failback: No

FaultTolerance
VLAN 22
Fault Tolerance Logging is Enabled
vmk0: 192.168.22.53
vmnic0 Active / vmnic1 Standby
Load balancing: Use explicit failover order
Failback: No

This is my design so far. I welcome your ideas, suggestions and other feedback. And please do not shoot me. Thank you very much for your attention.

Updated: 29-7-2011, minor corrections and stupid mistake concerning network redundancy, sorry.

About one year ago, I hesitated about what to purchase for my new home lab. One day I discovered Simon Seagrave’s http://www.techhead.co.uk. As a result, some time later I bought a HP ML110 G5, followed by a HP ML110 G6.

And that isn’t all, you buy a server with 1 or 2 GB Memory and a single NIC, so a hardware upgrade is inevitable. Again, TechHead was of great help.

I regularly receive new comments on blog post “HP Proliant ML110 G6 Released – Still a worthy virtualization work or home lab server?” and I notice that people often encounter problems while configuring the G6 or installing ESXi.

So in this post, I will describe:

-          The basic configuration of my G6

-          Hardware Upgrade

-          Important BIOS settings

-          Installing VMware ESXi 4.1 Update 1

The basic configuration of my G6

An actual overview of the components of my home lab can be found here. The HP ML 110 G6 has the following characteristics:

Server 2 (Make/Model): HP ML110 G6
ProLiant BIOS: 027
CPU: 1 x Intel Xeon X3430 @2,4 GHz (processor options: VT-d (Directed I/O))
Memory (Make/Model): HP 2GB 2Rx8 PC3-10600E-9 kit
Memory (Amount): 2 x 2 GB (1 x 2 GB was for free )

Hard Disk (Make/Model): WD 250 GB 7200 RPM SATA
No of Hard Disk: 1
RAID Level (if applicable): No
Disk Controller (Make/Model): On Board Ibex Peak, 4 ports SATA plus 2 ports SATA

CD/DVD Drive (Make/Model): HP DVD ROM DH20N SATA
No of Hard Disk: 1

Network Card (Make/Model): On Board NC107i PCI Express Gigabit Server Adapter
Network Speed: 1 GB/s

Hypervisor/Operating System: VMware ESXi 4.1 on SanDisk Cruzer 4 GB USB

Hardware Upgrade

The first upgrade was extra memory. I bought one pair of the specified Kingston 4 GB modules and combined the modules with the HP modules.

Memory (Make/Model): Kingston KVR1333D3E9s/4G (4GB PC3 – 10600 CL9 ECC 240 – Pin DIMM)
Memory (Amount): 2 x 4 GB

The idea was to try this combination, which would give me a total of 12 GB memory. In case of failure, the scenario was to buy another pair of these modules. The combibation works fine.

With VMware it is said you cannot have enough NICs, even in a home lab. This Intel Card is not expensive and it functions well with ESXi 4.x.

Network Card (Make/Model) : Intel EXPI9301CT = Intel 82574L
Network Speed: 1 GB/s

How about storage? Both servers are equipped with a rather small and single harddisk and lack a supported RAID controller. From the moment I decided to buy a second server, it was clear to have some sort of shared storage. There are a lot of possibilities, you can use virtual appliances like Openfiler to simulate iSCSI storage or NFS. Finally I decided to buy a relatively cheap Iomega StorCenter ix2 with 2 x 1TB disks configured in RAID-1.

Network connectivity is provided by a Cisco SLM2008 8-Port Gigabit Smart Switch, see this link for a review.

Important BIOS settings

There are a few important BIOS settings to review, I will not go into every detail, but show some relevant settings to create a nice ESX or ESXi host.

The Main window gives an overview of the BIOS version and the amount of memory.

The Advanced Window overview. I am not sure about the “Processor power Efficiency” option, it should correspond with the Power Settings in the vSphere Client (in the vSphere Client, select ESXi host, go to Configuration Tab, Hardware section, Power Management).

The “Advanced Chipset Control Section”. Make sure you enable the “Intel VT for Directed I/O (VT-d)”. See this link for a demo.

The “Advanced Processor Options”. In my case the “Intel(R) Virtualization Technology” setting was disabled. Make sure it is Enabled. The “Processor Hyper-Threading” setting does not make sense , because the Intel Xeon X3430 does not support hyper-threading.

The “CPU Power Management” settings.

The “Harddisk Configuration” window. In this blog post the reason for choosing the “SATA mode selection” AHCI is explained.

Finally, check the Boot order, we need this for the next section on Installing ESXi.
Update per 10–8-2011, more on CPU settings in this post.

Installing VMware ESXi 4.1 Update 1

Both servers in my home lab run ESXi instead of ESX. I have created a few virtual ESX servers, in case I need to do some testing on ESX. With ESXi, you have the option to install on the local hard disk or on an USB memory stick . I have decided to install ESXi on a USB memory stick (Sandisk Cruzer 4GB). The G6 has a USB connector on the mainboard. However, because I do change the memory stick on a regular basis, I use one of the external USB connectors.

Another concern is the HP DVD ROM that came with the server. You can install ESXi 4.0. But when you try to install ESXi 4.1, you will be presented this window. The DVD ROM is not supported.

I have read that people replaced the DVD ROM for a supported drive.

Another approach is to use the Virtual Media. Unlike the G5, the G6 is equipped with a Lights-Out 100 remote access card.

Out of the box it has functionality to start and stop the G6, review sensors, event logs and alerting, but lacks the remote console functionality. After obtaining an additional license, the remote console and virtual media functionality become available.

Steps to install ESXi 4.1 U1 with Virtual Media. Download the correct .ISO image. Do not download the HP customized .ISO, these images will not work on a ML110 G6.

Now, it is time to mount the .ISO image. In the menu, go to Control en Virtual Media…

The Virtual Media Windows opens, choose “Add..”

Search for the correct .ISO, in this case ESXi 4.1 Update 1, Build 348481 and press “Select”.

“Connect” the previous selected media and press “OK”.
After pressing “Connect”, the button changes to “Disconnect” to confirm that you have connected the selected media.

Make sure your media is connected when you fire up your G6. In my case it took me a few reboots before I discovered that the media was not connected at boot time

Boot the ML 110 G6, after a few moments, the boot menu shows up, choose “ESXi Installer” to start the actual installation.

In the next window, press “Enter”to continue the installation.

Press F11 to Accept the licence terms and continue.

The “Select a Disk” windows, shows both storage devices. The first one is the USB memory stick, the second device is the local SATA disk. Of course, we choose the first device.

An extra warning to tell us there is some old data on the USB memory stick, which is OK.

After this last warning the installation will start by pressing the F11 key.

Sometime later we receive a notification that the installation has been finished, the server must be rebooted.

Before reboot, first Disconnect the virtual media and remove it.

The server boots, and after a few moments…

Now you are ready to start the configuration part.

That is the whole story. In case you have any questions or remarks, please feel free to respond.

This post is dedicated to all the people that responded to the post on TechHead. I want to say, thank you to all contributors, it was invaluable and has helped me so much.

You can buy the training online for $397, it includes:

• 3 DVD set that will be shipped (within 7 workdays in my case) with no extra costs.
• The first and second DVD contain 12 video lessons. The third disc contains the same material as iPod, MP3 or WMV files and instructions how to place them on your iPod.
• Besides the video lessons, the Instructor notes (sheets) and example scripts are also included.
• Instant access to the “My Online training”. This is really the best part!
  Minutes after purchasing the training, you can start your training online. You don’t have to wait for UPS. The “My Online training” contains all the training material.

The course outline

Lesson 1 – Getting Started with VMware vSphere PowerCLI Training

Lesson 2 – Lab Setup

Lesson 3 – Introduction to PowerCLI

Lesson 4 – PowerCLI Concepts

Lesson 5 – Working with Objects in PowerShell

Lesson 6 – Managing vSphere Hosts with PowerCLI

Lesson 7 – Managing Virtual Machines with PowerCLI

Lesson 8 – Managing Virtual Networks with PowerCLI

Lesson 9 – Managing Storage with PowerCLI

Lesson 10 – Managing High Availability and Clustering with PowerCLI

Lesson 11 – Managing vCenter with PowerCLI

Lesson 12 – Reporting with PowerCLI

The total playing time of the 12 lessons is more than 11 hours.

My personal evaluation

I am rather new to Windows PowerShell and I do have more experience writing Linux Bash scripts.  The first five lessons is an introduction to PowerShell and the vSphere PowerCLI.
Mr. Rottenberg explains why PowerShell is important to you. One of the most important parts of this course are the numerous demonstrations. It does not make sense just to watch the videos. You should also practice in your own environment. Mr. Rottenberg explains what you need to setup your own lab, so you can repeat and try to solve your questions in PowerShell. Two lessons are dedicated to the introduction and the concepts of the PowerCLI.

Lesson 6 to 11 discusses Cmdlets grouped by vSphere Hosts, Virtual Machines, Virtual Networks, Storage and so on. On first glance, one might think “When I do know one Get- Cmdlet, I know them all”, this is not the case.

After a few lessons, especially during Mr. Rottenberg’s demonstrations, you will start to see the magic of PowerShell and the use of the PowerCLI commands. The Get-VMHost | Get-VM pipelines become familiar. Before starting this training, one of the things I had hoped, was to gain more insight in the concepts of PowerShell. In that aspect, this course also met my expectations. In each demonstration, Mr Rottenberg shows how Cmdlets work and explains how to use other PowerShell commands (Now I know how to use Get-Method, Get-View etc.) and custom functions (like gpm) to get your things done. The final lesson “Reporting with PowerCLI” is in my opinion one of the best lessons. In this lesson, three example scripts (to create Reports) are discussed.

My final conclusion is, when you are new to PowerShell and the vSphere PowerCLI, this course is worth considering. It will get you up to speed with the vSphere PowerCLI in no time.

You can watch a sample of this training.

Of course, there is the official VMware “vSphere PowerCLI Administration Guide”. However, this guide is not a complete vSphere PowerCLI training. After a brief introduction it gives a lot of useful examples concerning the more advanced topics. So, I recommend this guide for the more experienced users.

Disclaimer: In no way am I sponsored by TrainSignal. I bought my own copy of this training.

The essential part of this Error message:

Starting guest agent
Cannot register management service. Service name: [VeeamVssSupport].
Cannot create service. Machine: [<IP address>]. Service name: [VeeamVssSupport].
Service binary path: [C:\WINDOWS\VeeamVssSupport\VeeamGuestAgent.exe].
Service account: [].
Cannot create service.
Win32 error: The specified service has been marked for deletion.
Code: 1072

It always surprises me that this kind of messages, contain a lot of useful information and how bad I (and I assume a lot of other people) tend to read the content.

That said, it is time to have a look at the VM in question, open the Services applet and find this one:

Windows has already changed the Startup type in “Disabled” and the Service Status is “Stopped”. Investigating the folder  C:\WINDOWS, you will notice that the folder “VeeamVssSupport” and the executable “VeeamGuestAgent.exe” no longer exist. VeeamVssSupport seems some kind of helper service and  will be uninstalled after finishing its job. As the error message states “Win32 error: The specified service has been marked for deletion.”, Windows seems unable to finish the last part of the job. For the same reason, subsequent backup jobs fail to reinstall the service and the backup job will fail.

How to solve this?
My first idea was, open the Registry Editor and remove the service entry. Unfortunately that did not work. Some blog posts advice similar actions, all without result. The official Microsoft approach is to reboot the server. Because the affected service has been marked for deletion, it was gone after the reboot.

And yes, the first backup job after the reboot was successful.

If you have any comments or remarks, please feel free to respond. Happy reading.

The “vCenter Service Status” often shows all kinds of interesting alerts.

So after watching David Davis “VMware vSphere Troubleshooting Training”, which I can recommend ( I don’t get paid for this recommendation), I try whatever I can to fix my broken vCenter server. VMware also did a good job, there are a lot Knowledge Base articles that can help you solving your problems.

But sometimes you have to be rational, the cause of the problem seems a weird Tomcat issue, you have already spent too much time, but the “nerd inside” whispers; “Go on, search and fix it”.

Another approach.

• Make sure, you can afford some downtime, let us say about 30 minutes, sounds reasonable?
• Search your vCenter installation media. If your vCenter server is virtual, the ISO file will do. Make sure you have the correct Build number. In the menu, go to Help and About VMware vSphere:
  In my case this is the correct ISO: VMware-VIMSetup-all-4.1.0-345042.iso.
  Right now, I cannot explain the difference between Build 345043 and 345042. Anyway, 345042 is the official Build number according to VMware’s Support and Download section.
• If your vCenter Server is virtual, create a snapshot.
• Login to the vCenter Server, go to Services and Stop the “VMware VirtualCenter Server” service.
• Go to “Programs and Features”, select “VMware vCenter Server” and choose Uninstall.
• After some time “Uninstall completed” shows up. Now vCenter has been uninstalled. But the database is still there, even “vCenter Update Manager” is still installed.
• Now it is time to reboot the server.
• After the reboot, make sure the correct installation media is mounted and start the installation of vCenter Server. Proceed as you would do installing a new server. However, there are a few important steps.
• When it comes to “Database Options”, make sure, you select “Use an existing supported database”. Under Data Source Name (DSN), select the vCenter Server Database. If nothing shows up, you have to check this first.
• The next screen should look something like this one.
• The next dialog should show “Database re-initialization warning”. Here you must (sorry, I don’t like this word) choose “Do not overwrite, leave my existing database in place”.
• After this important dialog, continue as normal. vCenter Server will be installed with your old database available.
• If the installation has been finished, it is time to check the results. Connect to the vCenter Server with the vSphere Client. Check the “vCenter Service Status” and see if “Update Manager” is available.
• If everything is fine, do not forget to remove the snapshot.

Other issues, this part will be updated when necessary.

When trying to add users or groups from an Active Directory, the list is not populated.

All credits go to VMware Communities forum member “thinks2much” for solving the issue in this post.

When trying to change permissions, all users and groups local to the vCenter server are visible, after selecting the Active Directory domain (in my case, the name is “Virtual”), no users or groups are visible.

The ultimate solution was to run the “VMware VirtualCenter Server” and the “VMware VirtualCenter Management Webservices” under the Local System account. For some reason, during install, I have configured these services to run under a local Administrator account. This brings me to this overview of all services relevant to vCenter (Startup Type = Automatic) and the configured accounts.

I welcome your feedback, enjoy reading.

This post on a Microsoft forum set me on track on solving this issue. The root cause is that before upgrading to the Standard Edition, the Microsoft SQL Server Management Studio tools were installed in a Basic edition, where the Complete Tools should be installed.

So now, the remaining question is how to replace the Basic edition for a complete edition? In the rest of this post, I will show you how it is done. I have added some screenshots, I will only show the critical dialog windows and leave out the Next, next windows.

So mount the MS SQL server installation media. First I will show how to check which version of the Management Tools is currently installed. Select the Tools section and choose “Installed SQL Server features discovery report”

After a few moments, the result shows up in IE. We definitely have the Basic version installed.

First we must uninstall the Basic version. Go to “Programs and Features“, select “Microsoft SQL Server 2008 R2 (64-bit)” (that is the version that is installed) and choose “Uninstall/Change”.

From the next dialog, choose Remove. Do not be afraid.

Click OK in the “Setup Support Rules” dialog.

The next screen “Select Instance”. In my case, there is only one instance installed, so proceed with “Next”.

In the “Select Features” dialog, tick “Management Tools – Basic” and proceed.

Click “Next” in the “Removal Rules” dialog.

Click “Remove” in the “ready to remove” dialog.

If everything goes well, it will end with this message.

Now it is time to install the Complete version. Start the installer, go to the Installation section and choose the “New installation or add features to an existing installation” option.

Click “Install” in the “Setup Support Files” dialog.

Click “Next” in the “Setup Support Rules” dialog.

In the “Installation Type” dialog, choose “New installation or add shared features”.

Click “Next” in the “Product Key” dialog.

In the “License terms dialog”, place a tick at “I accept the license terms” and proceed with Next.

In the “Setup Role”, select “SQL Server Feature Installation” and proceed with Next.

In the “Feature Selection” dialog, make sure that “Management Tools – Complete” is slected. This wil also select “Management Tools – Basic” and proceed with Next.

In “Installation Rule” dialog, Next.

The next dialog is “Disk space requirements”, should be no problem and continue. The same goes for the “Error Reporing”, “Installation Configuration Rules” and “Ready to Install”,  dialog. The whole process should finish with this window.

So let’s see if this action was successful. Open the Microsoft SQL Server Management Studio and try to Modify an existing Manintenance Plan. Here is the result, we are now able to modify the Maintenance Plan. By the way, this plan was created using the Maintenance Plan Wizard.

Thank you very much for reading. I welcome your feedback.