Tag Archives: Bug

VMware Change Block Tracking Issue - Header

vSphere data loss bug returns – CBT issues in vSphere ESXI 8.0 update 2

General, Veeam, VMware, , , , , , , , , ,
The Issue

I keep saying, there are no new ideas in technology, just re-hashes of old ones. That is also true for VMware and their data loss issues.

The vSphere-based change block tracking (CBT) bug is back! I think I wrote 5 articles on this back in 2014/2015 with explanations and fixes!

Veeam reported this at the start of week commencing 11th December 2023, with VMware confirming the issue by the end of the same week.

The Cause

Change block tracking is the feature used to see which blocks of data have changed since a known point in time, to enable backup software to capture only the incremental changes.

If this feature fails, you could lose data in your backups, as the backup software doesn’t know which blocks to protect.

as per VMware:

CBT's QueryChangedDiskAreas may lose some data changed on the disk after disk is hot-extended.
It only happens on ESXi 8.0u2.
The Fix/Workaround

Directly from VMware’s newly published KB, which took them only a few days to confirm this behaviour after Veeam noticed at the start of the week!

  • Resolution
    • Unfortunately, there is no fix available for this bug at this time. However, you can use the following workaround to work around the issue until a fix is released
  • Workaround
    1. Reset CBT after disk is hot-extended. Then, user need to take a full backup immediately.
      It does not fix existing backups, but it makes sure the new ones are good.
    2. Or, user extend disk in offline.

You cannot fix your existing incremental backups if they have been affected, if they missed the correct data to backup, it’s been missed! But you can run an Active Full backup to capture everything, certainly for Veeam this is the case, other backup vendors you’ll need to double check with!

How do I reset Change Block Tracking?

If you are using Veeam, you can just perform an Active Full backup, and ensure the reset CBT option is configured. This is enabled by default.

If you aren’t using Veeam, then the following will be your next steps.

To reset Change Block Tracking, as per this older VMware KB article from the last time this was an issue. VMware may update this article or produce another one now this recent bug has been found.

  • Find your VM in the vCenter Client
    • Power the VM off
    • Click the Options tab, select the Advanced section and then click Configuration Parameters.
  • Disable CBT for the virtual machine by setting the ctkEnabled value to false.
  • If you need to do this for specific virtual disks attached to your virtual machine
    • Disable CBT by configuring the scsix:x.ctkEnabled value for each attached virtual disk to false. (scsix:x is SCSI controller and SCSI device ID of your virtual disk.)
  • Ensure there are no snapshot files (.delta.vmdk) present in the virtual machine’s working directory. For more information, see Determining if there are leftover delta files or snapshots that VMware vSphere or Infrastructure Client cannot detect (1005049).
  • Delete any -CTK.VMDK files within the virtual machine’s working directory.

Now power on your virtual machine.

Depending on your backup software vendor, you may need to manually re-enable Change Block Tracking, you can find a full list of steps and considerations in this VMware KB article. It’s essentially power down the VM, enable in value again in configuration parameters.

Summary

Let’s hope VMware produces a fix for this quickly, I remember they had this issue in vSphere 5.5 and 6.0 and some fixes didn’t resolved the issue, it was a pain being a consultant having to install fixes at customers sites.

It’s good that VMware have only taken a short amount of time to validate this bug and publish something officially about it!

 

Regards

Dean Lewis

Paris Tuileries Garden Facepalm statue

Further ESXi 6.0 CBT bug info – Reset your CBT!!!

VMware, , , , , , , , , ,

Following on from the recent (November 2015) ESXi 6.0 CBT bug, which has now been fixed in the latest released patch ESXi600-201511401-BG, some further information has come to light, provided by Anton Gostev, of Veeam.

You can read the snippet of important information from the Veeam forum post following the issue (Official Veeam KB2075);

All, we have completed the first day of testing in the same exact lab and using the same heavy write I/O test that made the original issue easily reproducible. After a few TB of increments, the above-mention patch appears to fully resolve the original issue when installed on ESX 6.0 Update 1a build 3073146.

However, we found that simply installing the patch is not sufficient, and CBT reset is required for all of your VMs. This is because existing CBT map files may contain issues created earlier due to the original bug, which may result in inconsistent full backups in future. Having CBT reset will also force the following job run use "full scan" incremental pass, thus fixing any existing inconsistencies in backups and replicas, as discussed earlier in this topic.

Provided CBT reset has been performed, Active Full backups is not required.

Performing Active Full backups by itself cannot be considered as a substitute to CBT reset with this particular CBT issue.

Thanks!

You can either follow the CBT Reset instructions from Veeam or look over to Chris Wahl’s latest blog post “Resetting VMware’s Changed Block Tracking (CBT) File with PowerCLI”.

Regards

Dean

c03601945

HP 2920 Switch – Reboot issue on firmware ver 15.18.0006 #vDM30in30

HP, Networking, , , , , , , , , , ,

Updated 25.11.15

Firmware WB.15.18.0007 resolves the issue, see below

A colleague of mine found an issue with the latest HP 2920 switch firmware.
If you create VLANs using the CLI Menu, the switch reboots and the configuration is not saved.

We have reported this to HP, but is currently being treated as a non critical issue as when creating a VLAN via the web interface or native CLI, the issue doesn’t happen.

We have also noticed on this firmware the switch seems to be less responsive. Luckily we had a few units in stock that we could replicate this issue on, and can confirm downgrading to the previous firmware version removes the issue.

A quick cheers to my colleague Marco for finding and researching this issue.

The issue

Switch: HP 2920-48G-POE+

Primary Image    :    12852982 08/12/15 WB.15.18.0006

Software revision  : WB.15.18.0006

  1.        Go to the Main Menu
  2.        Select (2) Switch Configuration…
  3.        Select (8) VLAN Menu…
  4.        Select (3) VLAN Port Assignment
  5.        Select Edit
  6.        Modify the tagging mode for a port
  7.        Select Save
  8.        Switch reboots and doesn’t save configuration

Hopefully HP will release a fix for this firmware soon, as mentioned we have recreated this issue in production and test.

The Fix

The following information was provided by HP Support. Continue reading HP 2920 Switch – Reboot issue on firmware ver 15.18.0006 #vDM30in30 →

backup error

Yet again, ESXi 6.0, CBT issues #vDM30in30

VMware, , , , ,

So here we are again, rounding the year off with yet another change block tracking warning.

I’ve written posts previously on the CBT bugs found in 5.5 and below and those found when vSphere 6.0 was released into the wild.

So whats the issue this time?

Basically it is the same as last time, the wrong sectors of data locations are returned by CBT when requested by your backup software. Meaning it targets the wrong data to backup.

Here is the official VMware KB 2136854. Which describes the following;

When running virtual machine backups which utilize Changed Block Tracking (CBT) in ESXi 6.0, you experience these symptoms:

 - The CBT API call QueryDiskChangedAreas() API call can sometimes      return incorrect changed sectors, which results in inconsistent  incremental virtual machine backups.

 - Inconsistent virtual machine backups

All incremental backups which utilize CBT are potentially affected.

Anton Gostev, Veeam, has released a lengthy email to Veeam customers and Gostev lovers stating the following (you can follow the veeam forum post here); Continue reading Yet again, ESXi 6.0, CBT issues #vDM30in30 →

2015 05 11 15 17 57

vCenter 6.0 Client Application GUI bug – VM’s showing Snapshots?

VMware, , ,

Today I’ve noticed that when using the vCenter C# client, each VM shows up as allowing for “Revert to current snapshot” and “Consolidation” however there is no open snapshot on the virtual machine.

2015-05-11_15-03-40

Above we can see clearly the options to “Revert …” and “Consolidate” are available, but when looking at Snapshot Manager, they are is nothing apparent.

2015-05-11_15-03-53

Looking into the configuration of the VM, there is no number string appended to the file name for the hard drive, i.e -00000001.vmdk as you would usually see for hard drives of a VM that is running on a snapshot

2015-05-11_15-04-17

If I bite the bullet and try to revert the snapshot or consolidate the VM, the vCenter task comes straight away, but the options are still available.2015-05-11_15-05-25 2015-05-11_15-05-52

Logging into the Web Client, you can see these options are not available for the VM.

2015-05-11_15-12-00

This means that it must be a GUI based bug in the application. As VMware are trying to phase out the application, I am not sure if they will resolve it.

Update – it is a known issue

2015-05-12_10-54-43

 

Frank Buchsel is a Technical Support Engineer for VMware, and his blog is brilliant! I urge you to check it out.

Regards

Dean