Post update 14th May 2015
VMware have released a hotfix KB2116126.
Removed information regarding old CBT workaround for Veeam is incompatible
_________________________________
So it’s come to my attention that the new and mighty ESXi 6.0 has a CBT fault as well. This time its effects are different than previous versions, but causes you backups to either fail, or take a considerably longer time to complete.
Here is the official KB from VMware: KB2114076 – Backing up a virtual machine with Change Block Tracking (CBT) enabled fails after upgrading to or installing VMware ESXi 6.0 (2114076)
The Cause
Below is VMware official note on the issue, which isn’t very promising, and will affect any backup technology using snapshots within VMware.
This issue occurs due to heap exhaustion when attempting to enable Change Block Tracking (CBT). If a virtual machine with a large number of virtual disks reached an upper threshold, enabling CBT fails because of heap exhaustion. This issue also occurs with multiple virtual machines with CBT enabled. In case of Windows virtual machines with VSS enabled, taking a quiesced snapshot creates double the amount of memory overhead. Note: The virtual disks can be spread across virtual machines or can be in a single virtual machine.
You can look in your VMKernel.log from the host the affected VM is on to see various warnings about CBT.
WARNING: CBT: 191: No memory available! Called from 0x418010db750e
Here I’ve used the System Centre Configuration Manager Trace32 Tool for viewing the VMKernel.log
Affected systems
Overall this will affect any backup systems that use change block tracking and snapshots for their backups.
On this Veeam forum post, it seems that any customer that has update their working production environment to ESXi 6.0 is affected by this issue.
I am currently working at a customer site, where we deployed brand new vCenter & hosts on ESXi 5.5 U2d, and migrated the VMs onto the hosts. We then upgraded the hosts to ESXi 6.0, crucially we didn’t backup the VMs using Veeam on the new hosts until the hosts were upgraded to ESXi 6.0
Out of 25 VM’s, it seems 3 are affected, with the following error message displayed in Veeam.
CBT data is invalid, failing over to legacy incremental backup. No action is required, next job run should start using CBT again. If CBT data remains invalid, follow KB1113 to perform CBT reset. Usual cause is power loss.
The Fix
VMware have released a hotfix for this KB2116126 as of 14th May 2015. (VMware:esx-base:6.0.0-0.6.2715440)
If you are unable to install this hot fix for what ever reason, then your only other option is below
Turn off CBT.
To disable CBT, ensure that there are no snapshots on the virtual machine. For more information, see Consolidating snapshots in ESX/ESXi 3.x and 4.x (1007849) and Consolidating snapshots in vSphere 5.x/6.0 (2003638). To disable CBT: 1. Power off the virtual machine. 2. Right-click the virtual machine and click Edit Settings. 3. Click the Options tab. 4. Click General under the Advanced section and then click Configuration Parameters. The Configuration Parameters dialog opens. 5. Set the ctkEnabled parameter to false for the corresponding SCSI disk. Alternatively, refrain on having too many virtual disks with CBT enabled on a single ESXi host. VMware recommends spreading the virtual disks across multiple ESXi hosts.
Regards
Dean
This is another bug and the mentioned workaround can not fix it. VMware Fix is available soon: http://forums.veeam.com/vmware-vsphere-f24/esxi-6-0-cbt-issue-t27816-15.html
Hi Yes, you are completely correct!!! I did write that in the text, but the heading was “workaround for veeam” which was completely misleading, I apologise, as I was testing the issue whilst writing the post. So at one point I thought I managed to get it working by Resetting CBT.
Anyway, I’ve re-wrote the header and the text underneath to reflect in clearer terms, resetting CBT does not work
Thanks for the post! Just so I’m clear, does this patch fix the CBT heap exhaustion issue described at the beginning of the post?
Yes it does!
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2114076
Hi,
This is the KB article I linked to originally.
Dean