VM disk consolidation fails

One morning while reviewing a vSphere 5.1 cluster at a customer location, I noticed one VM with the following message “Configuration Issues. Virtual machine disks consolidation is needed”.

VMdiskConsolidation-01Figure 1

A few days before, a snapshot was made as part of a regular update process. At the end of the day the snapshot was deleted. Apparently, something went wrong.

VMware KB 2003638 “Consolidating snapshots in vSphere 5.x” details the backgrounds for this message and also presents a solution for resolving this issue. Right-click the virtual machine and from the menu, choose Snapshot > Consolidate.

So far so good, but some moments later, a new message was received and the consolidation task was ended.

VMdiskConsolidation-02Figure 2

No panic, another KB will help you; KB 2013520 “Committing or creating snapshots fail with the error: the virtual disk is either corrupted or not a supported format”.

Time to check the vmware.log file of the virtual machine.

VMdiskConsolidation-03Figure 3

As the KB article already suggest, the tracking files (CTK files) are not accessible (Could not open/create change tracking file).

The presented resolution is to create a temporary folder, move the .CTK files to the newly created folder and try to consolidate the snapshots.

The first step is to SSH to the console of the ESXi host (in most situations, you will need to relax the security settings while working on this issue). Select the ESXi host, Configuration, Software, Security Profile, under Services, start the SSH service).

SSH to the ESXi server and CD to the folder where the affected VM resides and create a temporary folder:

# mkdir TEMP

Move the CTK files to the folder

# mv *ctk* ./TEMP

VMdiskConsolidation-04Figure 4

Now, return to the vSphere Client and restart the Consolidation. In VMware KB 1007566 “Commands to monitor snapshot deletion in ESX 2.5/3.x/4.x and ESXi 3.x/4.x/5.x (1007566)” some very useful commands are presented to watch the progress of the consolidation process, especially this command is good for the job:

# while true;do date;ls -lht *vmdk|head -10;echo ________;sleep 3;done

As a result, every 3 seconds, you will see the top 10 files which are being updated during the consolidation process.

VMdiskConsolidation-05Figure 5

As soon as the consolidation process has finished, it is time to delete the remaining snapshot(s).

VMdiskConsolidation-06Figure 6

As soon as the task “Remove all snapshots” has finished, check the folder. If everything went well, all snapshot files must be gone. Also notice that new .CTK files have been created. The files in the TEMP folder can be deleted now. See Figure 7.

VMdiskConsolidation-07Figure 7

Conclusion. First of all, do not panic. Consolidation and deleting snapshots can take quite some time. Wait patiently and watch the progress. During the whole process, the affected VM (an Exchange server) was up and running without any interruption of service.

As always, I thank you for reading and I welcome your comments.


9 Responses to VM disk consolidation fails

  1. albertwt says:

    I thought that Exchange Server is not supported with Hypervisor snapshot technology by MS ? or is this just the failed VADP backup snapshot remnants ?

  2. Nottingham says:

    Thanks,Excellent article

  3. […] I got help from: https://paulgrevink.wordpress.com/2013/12/02/vm-disk-consolidation-fails/ […]

  4. Suma G Raju says:

    Reboot required for this task ?

  5. S.N.M.A.R says:

    Thank you, you’re a life saver

  6. Thank you, you’re a life saver also for me…
    We had a similar issue, the snapshot removal failed with “A general system error occurred: vim.fault.GenericVmConfigFault”
    and consolidate failed with “msg.changetracker.NOTOPEN”
    But your procedure worked as a charm!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: