Category Archives: Veeam

VMware Change Block Tracking Issue - Header

vSphere data loss bug returns – CBT issues in vSphere ESXI 8.0 update 2

The Issue

I keep saying, there are no new ideas in technology, just re-hashes of old ones. That is also true for VMware and their data loss issues.

The vSphere-based change block tracking (CBT) bug is back! I think I wrote 5 articles on this back in 2014/2015 with explanations and fixes!

Veeam reported this at the start of week commencing 11th December 2023, with VMware confirming the issue by the end of the same week.

The Cause

Change block tracking is the feature used to see which blocks of data have changed since a known point in time, to enable backup software to capture only the incremental changes.

If this feature fails, you could lose data in your backups, as the backup software doesn’t know which blocks to protect.

as per VMware:

CBT's QueryChangedDiskAreas may lose some data changed on the disk after disk is hot-extended.
It only happens on ESXi 8.0u2.
The Fix/Workaround

Directly from VMware’s newly published KB, which took them only a few days to confirm this behaviour after Veeam noticed at the start of the week!

  • Resolution
    • Unfortunately, there is no fix available for this bug at this time. However, you can use the following workaround to work around the issue until a fix is released
  • Workaround
    1. Reset CBT after disk is hot-extended. Then, user need to take a full backup immediately.
      It does not fix existing backups, but it makes sure the new ones are good.
    2. Or, user extend disk in offline.

You cannot fix your existing incremental backups if they have been affected, if they missed the correct data to backup, it’s been missed! But you can run an Active Full backup to capture everything, certainly for Veeam this is the case, other backup vendors you’ll need to double check with!

How do I reset Change Block Tracking?

If you are using Veeam, you can just perform an Active Full backup, and ensure the reset CBT option is configured. This is enabled by default.

If you aren’t using Veeam, then the following will be your next steps.

To reset Change Block Tracking, as per this older VMware KB article from the last time this was an issue. VMware may update this article or produce another one now this recent bug has been found.

  • Find your VM in the vCenter Client
    • Power the VM off
    • Click the Options tab, select the Advanced section and then click Configuration Parameters.
  • Disable CBT for the virtual machine by setting the ctkEnabled value to false.
  • If you need to do this for specific virtual disks attached to your virtual machine
    • Disable CBT by configuring the scsix:x.ctkEnabled value for each attached virtual disk to false. (scsix:x is SCSI controller and SCSI device ID of your virtual disk.)
  • Ensure there are no snapshot files (.delta.vmdk) present in the virtual machine’s working directory. For more information, see Determining if there are leftover delta files or snapshots that VMware vSphere or Infrastructure Client cannot detect (1005049).
  • Delete any -CTK.VMDK files within the virtual machine’s working directory.

Now power on your virtual machine.

Depending on your backup software vendor, you may need to manually re-enable Change Block Tracking, you can find a full list of steps and considerations in this VMware KB article. It’s essentially power down the VM, enable in value again in configuration parameters.


Let’s hope VMware produces a fix for this quickly, I remember they had this issue in vSphere 5.5 and 6.0 and some fixes didn’t resolved the issue, it was a pain being a consultant having to install fixes at customers sites.

It’s good that VMware have only taken a short amount of time to validate this bug and publish something officially about it!



Dean Lewis

kasten vmware red hat openshift header

How to install and configure Kasten to protect container workloads on Red Hat OpenShift and VMware vSphere

In this blog post I’m going to cover deploying and configuring Kasten, the container based enterprise backup software now owned by Veeam Software.

This deployment will be inside my Red Hat OpenShift Environment which is running on top of VMware vSphere.

I’ll be protecting a cool gaming application that has data persistence using MongoDB.

Installing Kasten on Red Hat OpenShift

In this guide, I am going to use Helm, you can learn how to install it here.

Create a OpenShift project (Kubernetes namespace) called “kasten-io”

oc new-project kasten-io

oc new project kasten-io

Next we are going to use Helm to install the Kasten software into our OpenShift cluster.

helm install k10 kasten/k10 --namespace=kasten-io --set scc.create=true --set route.enabled=true --set route.path="/k10" --set auth.tokenAuth.enabled=true

Breaking down the command arguments;

  • –set scc.create=true
    • This creates the correct Security Contexts against the users created by the install. This is needed in OpenShift as the security context stance is higher OOTB than that of a vanilla Kubernetes install.
  • –set route.enabled=true
    • This creates a route in OpenShift using the default ingress, so that the Kasten dashboard is accessible externally. This will use the default cluster ID domain name.
  • –set route.path=”/k10″
    • This sets the route path for the redirection of the dashboard. Without this, your users will need to go to http://{FQDN}/ and append the path to the end (k10).
  • –set auth.tokenAuth.enabled=true

helm install k10 kasten kasten-io Continue reading How to install and configure Kasten to protect container workloads on Red Hat OpenShift and VMware vSphere


Interview with Nikola Pejkova – Technical Analyst & Community Manager @Veeam

For those of you who know or follow me, I’m delighted to be part of the Veeam Vanguard program, created to bring together some of the top individuals focused on technology and yes, the Veeam ecosystem to help guide Veeam with feedback and perspective.

But to be honest, for me it’s just a fantastic group of individuals who are experts in their various fields. They will bend over backwards to help and support you, and we all love to learn and share with the community. If you are involved with Veeam products in any way in your daily life/job role, I highly recommend you look into the program and apply if you feel like it suits you.

Moving onto this blog post, I’ve had the pleasure of interviewing Nikola Pejkova (twitter) from Veeam, Technical Analyst and Community Manager. Nikola has been the driving force behind the Veeam Vanguard program and events for the past few years.

So, let’s dive in!

Nikola, tell us a bit about yourself, and what led to you taking the role as the community manager for Veeam.

Thanks for having me, Dean! I live in Prague, beautiful capital of Czech Republic where Veeam have its offices. I’ve been working in IT industry for more than 6 years now and I’ve experienced different roles in corporates as well as in startups. This mix of experiences have been probably the right fit when Veeam was searching for a community manager and I’ve been searching for a role where I can build something valuable and long-term.

It's safe to say Nikola has made her mark working for Veeam
It’s safe to say Nikola joined Veeam with the goal of making her mark at the company and in the technology industry. And I think many will agree, she is doing just that.

So, the Veeam Vanguard community manager role was the perfect match! I’ve been impressed by Veeam’s history, maintenance, and development of their own software products and also the culture that is so far from the experience I gained in other worldwide corporate companies.  

Please can you define what community means to you? And what goals you had coming into this role?

Community is something like a secondary family when it’s treated in a good way. There are many kinds of communities out there, but specially in case of IT ones, they’re kind of a special case. As they are uniting literally people from all the world, they are incredibly diverse and let us overcome the physical distance through technologies we got available those days.

Nikola meeting Vanny the Veeam Vanguard mascot
Nikola meeting Vanny the Veeam Vanguard mascot

Power of the community comes from its members, who can share together their experiences, have conversations about issues they’re solving at the moment or just support each other in hard times. Another crucial role in the community success is having the company that has to offer not just great products, but also continuous and various ways of engagement that keep the community live and up-to-date with all the innovations that are coming so fast nowadays! It’s like a group of friends who are having the same passion, in our case,  saving and protecting the data with products that just work.  

What makes an excellent community program versus just a good community program?

In my opinion there’s a crucial management of the community that needs to be done in a way that members understand the purpose of being part of the community, what benefits they are getting from interactions with others and what is the added value that they take out of being a part of any community.

Since COVID, one of the regular weekly get togethers in the calendar is the Veeam Vanguard Social call.
Since COVID, one of the regular weekly get-together in the calendar is the Veeam Vanguard Social call.

I’d say that difference between excellent and just a good community program is a passion of its members – when the community program is managed in a great way, members are happy, passionate, willing to participate and do extra steps for the community. When you got an average one, it can fulfill its purpose as a platform for social interactions and sharing experiences, but members won’t be so proactive to dedicate their own time out for the community. 

Are there areas from other community programs you are looking to replicate or even avoid? Which activities to you think have lost their shine?

What I’d like to generally avoid is taking the community members as “anonymous” numbers without any personal touch. What is incredible on Veeam Vanguard community is that we know each other not just by name, in most cases also personally. We do care about Vanguards as a group of unique and very special and talented individuals, which I’d love to keep for the future, as it’s one of the aspects that differentiate our community from others. 

What’s been the highlights so far leading the community programs at Veeam?

It’s hard to name just few of them, but for sure the highlight has been Vanguard Summit in Prague last October, where we met all together – Veeam Product Strategy team accompanied also with members of R&D team was spending almost the week in Prague together with Vanguards from all over the world to share latest product updates, which was super fun and we all had incredible time!

Addressing the Veeam Vanguard members at the annual summit in Prague, Nikolas home city.
Addressing the Veeam Vanguard members at the annual summit in Prague, Nikolas home city. (L-R, David Hill, Nikola Pejkova, Andrew Zhelezko, Kirsten Stoner, Karinne Bessette)

Also presenting about the Vanguard program during VMworld in Barcelona last fall was incredible experience. And I cannot forget how proud I was when so many Vanguards were answering during VeeamON Virtual in Expert Lounge to attendees from across the globe or participated on presentations with other Veeamers during VeeamON this year. Generally, the biggest highlight for me is to see the passion that Vanguards have towards Veeam products and willingness to constantly helping improve them. 

You are launching a new program, the Veeam Legends program, can you tell me about the process behind getting this ready for release to the public? 

Well, there were a lot of thoughts and efforts behind this project! First, we wanted to foster even deeper engagement that rewards our customers and partners for the role they have played in our success. We also wanted to introduce a new tier in our community program enabling some of our most active community participants to be recognized as leaders in the data protection community through content creation and user group and event participation.

Second, we wanted to keep conversation going and to provide users with a single place where they can share best practices, participate in virtual Veeam User Groups, grow as a Veeam pro by completing free, on-demand training through Veeam University and have fun by recognition and rewards.

That’s how we launched Veeam Community Resource Hub and Veeam Legends, the new community programs!

Veeam Legends part of the interview

Diversity and inclusion have become a recent focal topic in the 2020 climate, what is Veeam doing around this?

Veeam is very proactive when it comes to inclusion and equal opportunities. There are various initiatives inside the company that everybody can join, we also have an option to be active inside our local communities as volunteers. As Veeam is a global company with offices across the world, diversity is one of the key values as well as respect to others. 

And as a woman in the technology industry, what do you think of the state of D&I in technology today? And what barriers do we need to overcome?

Since time I’ve started to work in IT industry, I can see growing number of women not just in leading managerial roles, but also in development, graphics, and analytics, which is great!

Presenting at VMworld 2019 on the vBrownbag stage, encouraging others to be part of the Veeam communities
Presenting at VMworld 2019 on the vBrownbag stage, encouraging others to be part of the Veeam communities

I don’t see any reason why IT should be just for men, but I believe that it’s kind of stigmatised from the past that those technical areas are for men and others for women. I believe that we’ve overcome this as a society and much more girls are applying now to technical fields in universities, so I am expecting continual growth of numbers of girls in the IT. Also, for the reason that technologies are everywhere today, so younger generation is growing surrounded by all this since they were born, and It feels generally much more natural for them to be in contact with all the technology that is new for previous generations. 

Obviously with the current global pandemic we find ourselves suffering some disruption in our lives, but looking towards the rest of 2020 and into 2021, what personal goals and plans do you have?

After being locked due to COVID quarantine over 4 months in Canarias, my main goal to get home was fortunately finally met ?.

Nikola returning home during covid.

Regarding the current travel restrictions and turning the world as we knew it before upside down, I’m thinking about some minivan reconstruction to rebuild it into the house on 4 wheels. We are having unlimited number of beautiful spots to visit in Czech Republic and surrounding countries, so having the van and independence to go anywhere I want gives me enough satisfaction. And finally put together my home office, as I think I might be seeing it a little more often!

Wrapping up

Dean here again.

I’d like to thank Nikola for agreeing to be interviewed. Working as a community manager in the technology industry is a unique challenge. It’s more than running events and asking customers/partners/advocates to help promote your brand. There are many community programs out there, but not all are successful, and it’s a fine line in getting it right.

It’s taken me a while to finalise this article and writing during a tumultuous year has been interesting. Since Nikola and I first exchanged messages on this blog post, so much has changed worldwide.

Hopefully in the future when we find ourselves to travel, myself and Nikola will be able to sit down and conduct a follow up interview, we can review this time, and how the community has changed during this time.


Veeam vRA Header

How to backup vRealize Automation 8.x using Veeam

In this blog post I am going to dissect backing up vRealize Automation 8.x using Veeam Backup and Replication.

- Understanding the backup methods
- Performing an online backup
- Performing an offline backup

Understanding the Backup Methods

Reading the VMware documentation around this subject can be somewhat confusing at times. And if you pay attention, there are subtle changes between the documents as well. Lets break this down.

  • vRealize Automation 8.0
    • As part of the backup job, you need to run a script to stop the services
    • This is known as an offline backup
    • Depending on your backup software, you can either do this by running a script located on the vRealize Automation appliance or by triggering using the pre-freeze/post-freeze scripts when a snapshot is taken of the VM.
    • The snapshot must not include the virtual machines memory.
    • If you environment is a cluster, you only need to run the script on a single node.
    • All nodes in the cluster must be backed up at the same time.
  • vRealize Automation 8.0.1 and 8.1 (and higher)
    • It is supported to run an online backup
      • No script is needed to shut down the services
    • Snapshot taken as part of the backup must quiesce the virtual machine.
    • The snapshot must not include the virtual machines memory.
    • It is recommended to run the script to stop all services and perform an offline backup.
      • You may also find your backup runs faster, as the virtual machine will become less busy.

Performing an Online Backup

Let’s start with the easier of the two options. Again, this will be supported for vRealize Automation 8.0.1 and higher. Continue reading How to backup vRealize Automation 8.x using Veeam

Veeam new logo featured image

Veeam – Script finished execution with unexpected exit code: 126

This Issue

When using Veeam if you use Pre/Post freeze scripts for application aware processing, these scripts must reside on the VBR server, which uploads to the virtual machine you are protecting.

I hit an issue when running a job that I received the error;

Error: Script finished execution with unexpected exit code: 126

Veeam Script finished execution with unexpected exit code 126

The Cause

I reviewed the task logs on the VBR server;

  • C:\ProgramData\Veeam\Backup\{Job_Name}\task_{VMName}.txt

You are looking for the sections [ScriptInvoker] for troubleshooting. And I found that my script was uploaded successfully, however when it was ran the Guest OS was throwing an error;

[ScriptInvoker] Failed to execute script in lin guest machine over SSH. Script path: C:\Backup\Scripts\

bash: /tmp/ /bin/bash^M: bad interpreter: No such file or directory

I also connected to the linux machine and ran “journalctl -xe -f” whilst running the backup job and seen the same error flash up too.

The script indicates that it must be executed by a shell located at /bin/bash^M. 
There is no such file: it's called /bin/bash. The ^M is a carriage return character. 
Linux uses the line feed character to mark the end of a line, whereas Windows uses the two-character sequence CR LF. Your file has Windows line endings, which is confusing Linux. 

You can see the full part of the task logs relating to this below.

[ScriptInvoker] Scripting mode is FailJobOnError.
[ScriptInvoker] Script enabled
[ScriptInvoker] Creating Linux invoker.
[ScriptInvoker] Starting pre-freeze script execution 
[ScriptInvoker] Running Linux script (SSH) 'C:\Backup\Scripts\'
[Ssh] SSH connection ae2f5df0-1d28-4dfc-a8a5-7ceb952af2a9 to server created successfully
[ScriptInvoker] SSH connection is established (
[ScriptInvoker] Exception thrown during script execution (SSH).
[Ssh] Connection ae2f5df0-1d28-4dfc-a8a5-7ceb952af2a9 - [host: '', port: 22, elevation to root: 'no', autoSudo: no, use su if sudo fails: no, host name: sc-dc1-vra001.simon.local, IPs: [], AuthenticationData: [UserName: root, AuthTypes: [KeyboardInteractive, Password]]] is disposing.
[ScriptInvoker] Failed to execute script in lin guest machine over SSH. Script path: C:\Backup\Scripts\
bash: /tmp/ /bin/bash^M: bad interpreter: No such file or directory
at Veeam.Backup.SSH.CSshCommandResult.GetAnswer(Boolean trimAnswer, String failText, Boolean checkStdErr)
at Veeam.Backup.Core.CSshScriptInvoker.RunScript(CSshScriptFile scriptFile, TimeSpan timeout, Boolean collectLogs, String stdOutFilePath, String stdErrFilePath, Boolean checkStdErr, Int32& exitCode)
at Veeam.Backup.Core.CSshScriptInvoker.ExecScriptInner(String localPath, TimeSpan timeout, Boolean collectLogs, String stdOutFilePath, String stdErrFilePath, Boolean checkStdErr, Int32& exitCode)

[ScriptInvoker] Failed to execute script over SSH, failing over to VIX.
bash: /tmp/ /bin/bash^M: bad interpreter: No such file or directory

[ScriptInvoker] Running Linux script (VIX) 'C:\Backup\Scripts\'
[ScriptInvoker] Linux script exit code = '126'

The Fix

The fix is quite easy, either change your text editor to show the correct line endings.

Or create the script on a Linux machine and copy it to your VBR Server.

To fix this issue in your text editor

Edit --> EOL Conversion --> UNIX/OSX Format

File > Convert Line Delimiters To > Unix (LF, \n, 0Α, ¶)

Or change the New text file line delimiter to Other: Unix on Window > Preferences > General > Workspace

Sublime Text Editor
View > Line Endings > Unix 

See this guide.