backup error

Yet again, ESXi 6.0, CBT issues #vDM30in30

So here we are again, rounding the year off with yet another change block tracking warning.

I’ve written posts previously on the CBT bugs found in 5.5 and below and those found when vSphere 6.0 was released into the wild.

So whats the issue this time?

Basically it is the same as last time, the wrong sectors of data locations are returned by CBT when requested by your backup software. Meaning it targets the wrong data to backup.

Here is the official VMware KB 2136854. Which describes the following;

When running virtual machine backups which utilize Changed Block Tracking (CBT) in ESXi 6.0, you experience these symptoms:

 - The CBT API call QueryDiskChangedAreas() API call can sometimes      return incorrect changed sectors, which results in inconsistent  incremental virtual machine backups.

 - Inconsistent virtual machine backups

All incremental backups which utilize CBT are potentially affected.

Anton Gostev, Veeam, has released a lengthy email to Veeam customers and Gostev lovers stating the following (you can follow the veeam forum post here);

To put it short, Changed Block Tracking (CBT) data on ESXi 6 cannot be trusted, and there are no real workarounds except downgrading to vSphere 5.5. Yet again, those of you who decided to hold off jumping the new vSphere release until it stabilizes will get some pats on their shoulders from colleagues, while the rest will have a few tough weeks ahead...

Disabling CBT is essential - otherwise, even Active Full backup may contain corruption, because CBT data is used there too to determine and skip zeroed regions of virtual disks.

Below is a quick unofficial statement from Aaron Meza, who I was conversing with around the last CBT issue.

Any i/o happening during consolidation has the potential to not be recorded. consolidation is a regular occurrence so this affects all VMs, not just those where disks were resized (which the case was for previous bug). Any backup product that relies on CBT for getting in-use blocks (a "full backup" for many products) could even be corrupt not just incrementals.
Which systems are effected this time?
  • Any system’s running VMware vSphere 6.0 (ESXi 6.0) including Update 1a

When combined with any backup software that uses change block tracking as its backup mechanism.

What are the fixes/workarounds?

There is no fix currently, the options offered by VMware are;

  • Downgrade your VMware ESXi to 5.5 update 3a and VM hardware version

You can this post by VMware on how to downgrade ESXi, and this one on how to downgrade the VM hardware version

  • Shutdown the VM before running any incremental backups

Which for production systems isn’t really viable

  • Perform a full backup rather than incremental

This is viable for production machines, but there is an increase in time taken to do this and the storage capacity will quickly run out. Even using Veeam’s compression and deduplication options (other backup software is available)

Once again, will increase the time taken for a backup, and this will vary from backup product to backup product. Here is how to disable CBT via Veeam itself.

What if you haven’t moved to vSphere/ESXi 6.0?

Then don’t, leave it until the new year, wait for these bugs to be squashed. To be honest I was hoping this was the last time we’d see a CBT issue, but apparently not. The technology is great, but once again VMware have really dropped the ball on this one.



Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.