Change block tracking is the feature used to see which blocks of data have changed since a known point in time, to enable backup software to capture only the incremental changes.
If this feature fails, you could lose data in your backups, as the backup software doesn’t know which blocks to protect.
CBT's QueryChangedDiskAreas may lose some data changed on the disk after disk is hot-extended.
It only happens on ESXi 8.0u2.
The Fix/Workaround
Directly from VMware’s newly published KB, which took them only a few days to confirm this behaviour after Veeam noticed at the start of the week!
Resolution
Unfortunately, there is no fix available for this bug at this time. However, you can use the following workaround to work around the issue until a fix is released
Workaround
Reset CBT after disk is hot-extended. Then, user need to take a full backup immediately.
It does not fix existing backups, but it makes sure the new ones are good.
Or, user extend disk in offline.
You cannot fix your existing incremental backups if they have been affected, if they missed the correct data to backup, it’s been missed! But you can run an Active Full backup to capture everything, certainly for Veeam this is the case, other backup vendors you’ll need to double check with!
How do I reset Change Block Tracking?
If you are using Veeam, you can just perform an Active Full backup, and ensure the reset CBT option is configured. This is enabled by default.
If you aren’t using Veeam, then the following will be your next steps.
To reset Change Block Tracking, as per this older VMware KB article from the last time this was an issue. VMware may update this article or produce another one now this recent bug has been found.
Find your VM in the vCenter Client
Power the VM off
Click the Options tab, select the Advanced section and then click Configuration Parameters.
Disable CBT for the virtual machine by setting the ctkEnabled value to false.
If you need to do this for specific virtual disks attached to your virtual machine
Disable CBT by configuring the scsix:x.ctkEnabled value for each attached virtual disk to false. (scsix:x is SCSI controller and SCSI device ID of your virtual disk.)
Delete any -CTK.VMDK files within the virtual machine’s working directory.
Now power on your virtual machine.
Depending on your backup software vendor, you may need to manually re-enable Change Block Tracking, you can find a full list of steps and considerations in this VMware KB article. It’s essentially power down the VM, enable in value again in configuration parameters.
Summary
Let’s hope VMware produces a fix for this quickly, I remember they had this issue in vSphere 5.5 and 6.0 and some fixes didn’t resolved the issue, it was a pain being a consultant having to install fixes at customers sites.
It’s good that VMware have only taken a short amount of time to validate this bug and publish something officially about it!
Well, it’s exactly as the name suggests, a tool for building your own vRealize Operations Management Pack, to bring data into vROPs whereby there is no existing Management Pack today.
How do I get access to it?
You can sign up for the BETA here. Currently VMware is taking feedback from customers to help shape the future of this product.
You can find documentation and videos on the product on this page.
Note: VMware does not commit to delivering features discussed in this program in any generally available product.
Installing the Appliance
I’m not going to go into detail here, it’s a simple appliance that you deploy as an OVA file and provide the networking configuration as either DHCP or Static IP.
Setup the Management Pack Builder
Log into the appliance using “admin/admin” and you will be prompted to change your password.
You will need to licence the product (using the beta key).
Click the little person icon in the top right
Select “Licence”
Apply the licence
Next, we need to create a connection to our vRealize Operations environment. While this step is not necessary at all to get started. If you want to create relationships for data objects with existing vROPs objects, you need to configure this connection.
For those of you who know or follow me, I’m delighted to be part of the Veeam Vanguard program, created to bring together some of the top individuals focused on technology and yes, the Veeam ecosystem to help guide Veeam with feedback and perspective.
But to be honest, for me it’s just a fantastic group of individuals who are experts in their various fields. They will bend over backwards to help and support you, and we all love to learn and share with the community. If you are involved with Veeam products in any way in your daily life/job role, I highly recommend you look into the program and apply if you feel like it suits you.
Moving onto this blog post, I’ve had the pleasure of interviewing Nikola Pejkova (twitter) from Veeam, Technical Analyst and Community Manager. Nikola has been the driving force behind the Veeam Vanguard program and events for the past few years.
So, let’s dive in!
Nikola, tell us a bit about yourself, and what led to you taking the role as the community manager for Veeam.
Thanks for having me, Dean! I live in Prague, beautiful capital of Czech Republic where Veeam have its offices. I’ve been working in IT industry for more than 6 years now and I’ve experienced different roles in corporates as well as in startups. This mix of experiences have been probably the right fit when Veeam was searching for a community manager and I’ve been searching for a role where I can build something valuable and long-term.
It’s safe to say Nikola joined Veeam with the goal of making her mark at the company and in the technology industry. And I think many will agree, she is doing just that.
So, the Veeam Vanguard community manager role was the perfect match! I’ve been impressed by Veeam’s history, maintenance, and development of their own software products and also the culture that is so far from the experience I gained in other worldwide corporate companies.
Please can you define what community means to you? And what goals you had coming into this role?
Community is something like a secondary family when it’s treated in a good way. There are many kinds of communities out there, but specially in case of IT ones, they’re kind of a special case. As they are uniting literally people from all the world, they are incredibly diverse and let us overcome the physical distance through technologies we got available those days.
Nikola meeting Vanny the Veeam Vanguard mascot
Power of the community comes from its members, who can share together their experiences, have conversations about issues they’re solving at the moment or just support each other in hard times. Another crucial role in the community success is having the company that has to offer not just great products, but also continuous and various ways of engagement that keep the community live and up-to-date with all the innovations that are coming so fast nowadays! It’s like a group of friends who are having the same passion, in our case, saving and protecting the data with products that just work.
What makes an excellent community program versus just a good community program?
In my opinion there’s a crucial management of the community that needs to be done in a way that members understand the purpose of being part of the community, what benefits they are getting from interactions with others and what is the added value that they take out of being a part of any community.
Since COVID, one of the regular weekly get-together in the calendar is the Veeam Vanguard Social call.
I’d say that difference between excellent and just a good community program is a passion of its members – when the community program is managed in a great way, members are happy, passionate, willing to participate and do extra steps for the community. When you got an average one, it can fulfill its purpose as a platform for social interactions and sharing experiences, but members won’t be so proactive to dedicate their own time out for the community.
Are there areas from other community programs you are looking to replicate or even avoid? Which activities to you think have lost their shine?
What I’d like to generally avoid is taking the community members as “anonymous” numbers without any personal touch. What is incredible on Veeam Vanguard community is that we know each other not just by name, in most cases also personally. We do care about Vanguards as a group of unique and very special and talented individuals, which I’d love to keep for the future, as it’s one of the aspects that differentiate our community from others.
What’s been the highlights so far leading the community programs at Veeam?
It’s hard to name just few of them, but for sure the highlight has been Vanguard Summit in Prague last October, where we met all together – Veeam Product Strategy team accompanied also with members of R&D team was spending almost the week in Prague together with Vanguards from all over the world to share latest product updates, which was super fun and we all had incredible time!
Addressing the Veeam Vanguard members at the annual summit in Prague, Nikolas home city. (L-R, David Hill, Nikola Pejkova, Andrew Zhelezko, Kirsten Stoner, Karinne Bessette)
Also presenting about the Vanguard program during VMworld in Barcelona last fall was incredible experience. And I cannot forget how proud I was when so many Vanguards were answering during VeeamON Virtual in Expert Lounge to attendees from across the globe or participated on presentations with other Veeamers during VeeamON this year. Generally, the biggest highlight for me is to see the passion that Vanguards have towards Veeam products and willingness to constantly helping improve them.
You are launching a new program, the Veeam Legends program, can you tell me about the process behind getting this ready for release to the public?
Well, there were a lot of thoughts and efforts behind this project! First, we wanted to foster even deeper engagement that rewards our customers and partners for the role they have played in our success. We also wanted to introduce a new tier in our community program enabling some of our most active community participants to be recognized as leaders in the data protection community through content creation and user group and event participation.
Second, we wanted to keep conversation going and to provide users with a single place where they can share best practices, participate in virtual Veeam User Groups, grow as a Veeam pro by completing free, on-demand training through Veeam University and have fun by recognition and rewards.
Diversity and inclusion have become a recent focal topic in the 2020 climate, what is Veeam doing around this?
Veeam is very proactive when it comes to inclusion and equal opportunities. There are various initiatives inside the company that everybody can join, we also have an option to be active inside our local communities as volunteers. As Veeam is a global company with offices across the world, diversity is one of the key values as well as respect to others.
And as a woman in the technology industry, what do you think of the state of D&I in technology today? And what barriers do we need to overcome?
Since time I’ve started to work in IT industry, I can see growing number of women not just in leading managerial roles, but also in development, graphics, and analytics, which is great!
Presenting at VMworld 2019 on the vBrownbag stage, encouraging others to be part of the Veeam communities
I don’t see any reason why IT should be just for men, but I believe that it’s kind of stigmatised from the past that those technical areas are for men and others for women. I believe that we’ve overcome this as a society and much more girls are applying now to technical fields in universities, so I am expecting continual growth of numbers of girls in the IT. Also, for the reason that technologies are everywhere today, so younger generation is growing surrounded by all this since they were born, and It feels generally much more natural for them to be in contact with all the technology that is new for previous generations.
Obviously with the current global pandemic we find ourselves suffering some disruption in our lives, but looking towards the rest of 2020 and into 2021, what personal goals and plans do you have?
After being locked due to COVID quarantine over 4 months in Canarias, my main goal to get home was fortunately finally met ?.
Regarding the current travel restrictions and turning the world as we knew it before upside down, I’m thinking about some minivan reconstruction to rebuild it into the house on 4 wheels. We are having unlimited number of beautiful spots to visit in Czech Republic and surrounding countries, so having the van and independence to go anywhere I want gives me enough satisfaction. And finally put together my home office, as I think I might be seeing it a little more often!
Wrapping up
Dean here again.
I’d like to thank Nikola for agreeing to be interviewed. Working as a community manager in the technology industry is a unique challenge. It’s more than running events and asking customers/partners/advocates to help promote your brand. There are many community programs out there, but not all are successful, and it’s a fine line in getting it right.
It’s taken me a while to finalise this article and writing during a tumultuous year has been interesting. Since Nikola and I first exchanged messages on this blog post, so much has changed worldwide.
Hopefully in the future when we find ourselves to travel, myself and Nikola will be able to sit down and conduct a follow up interview, we can review this time, and how the community has changed during this time.
In this blog post I am going to dissect backing up vRealize Automation 8.x using Veeam Backup and Replication.
- Understanding the backup methods
- Performing an online backup
- Performing an offline backup
Understanding the Backup Methods
Reading the VMware documentation around this subject can be somewhat confusing at times. And if you pay attention, there are subtle changes between the documents as well. Lets break this down.
vRealize Automation 8.0
As part of the backup job, you need to run a script to stop the services
This is known as an offline backup
Depending on your backup software, you can either do this by running a script located on the vRealize Automation appliance or by triggering using the pre-freeze/post-freeze scripts when a snapshot is taken of the VM.
The snapshot must not include the virtual machines memory.
If you environment is a cluster, you only need to run the script on a single node.
All nodes in the cluster must be backed up at the same time.
vRealize Automation 8.0.1 and 8.1 (and higher)
It is supported to run an online backup
No script is needed to shut down the services
Snapshot taken as part of the backup must quiesce the virtual machine.
The snapshot must not include the virtual machines memory.
It is recommended to run the script to stop all services and perform an offline backup.
You may also find your backup runs faster, as the virtual machine will become less busy.
When using Veeam if you use Pre/Post freeze scripts for application aware processing, these scripts must reside on the VBR server, which uploads to the virtual machine you are protecting.
I hit an issue when running a job that I received the error;
Error: Script finished execution with unexpected exit code: 126
You are looking for the sections [ScriptInvoker] for troubleshooting. And I found that my script was uploaded successfully, however when it was ran the Guest OS was throwing an error;
[ScriptInvoker] Failed to execute script in lin guest machine over SSH. Script path: C:\Backup\Scripts\stop-vra.sh.
bash: /tmp/d6791b89-e0b8-4cce-acec-45d682ce1f2c_stop-vra.sh: /bin/bash^M: bad interpreter: No such file or directory
I also connected to the linux machine and ran “journalctl -xe -f” whilst running the backup job and seen the same error flash up too.
The script indicates that it must be executed by a shell located at /bin/bash^M.
There is no such file: it's called /bin/bash. The ^M is a carriage return character.
Linux uses the line feed character to mark the end of a line, whereas Windows uses the two-character sequence CR LF. Your file has Windows line endings, which is confusing Linux.
(Source)
You can see the full part of the task logs relating to this below.
[ScriptInvoker] Scripting mode is FailJobOnError.
[ScriptInvoker] Script enabled
[ScriptInvoker] Creating Linux invoker.
[ScriptInvoker] Starting pre-freeze script execution
[ScriptInvoker] Running Linux script (SSH) 'C:\Backup\Scripts\stop-vra.sh'
[Ssh] SSH connection ae2f5df0-1d28-4dfc-a8a5-7ceb952af2a9 to server 192.168.200.39 created successfully
...
[ScriptInvoker] SSH connection is established (192.168.200.39).
[ScriptInvoker] Exception thrown during script execution (SSH).
[Ssh] Connection ae2f5df0-1d28-4dfc-a8a5-7ceb952af2a9 - [host: '192.168.200.39', port: 22, elevation to root: 'no', autoSudo: no, use su if sudo fails: no, host name: sc-dc1-vra001.simon.local, IPs: [192.168.200.39], AuthenticationData: [UserName: root, AuthTypes: [KeyboardInteractive, Password]]] is disposing.
[ScriptInvoker] Failed to execute script in lin guest machine over SSH. Script path: C:\Backup\Scripts\stop-vra.sh.
bash: /tmp/d6791b89-e0b8-4cce-acec-45d682ce1f2c_stop-vra.sh: /bin/bash^M: bad interpreter: No such file or directory
(System.Exception)
at Veeam.Backup.SSH.CSshCommandResult.GetAnswer(Boolean trimAnswer, String failText, Boolean checkStdErr)
at Veeam.Backup.Core.CSshScriptInvoker.RunScript(CSshScriptFile scriptFile, TimeSpan timeout, Boolean collectLogs, String stdOutFilePath, String stdErrFilePath, Boolean checkStdErr, Int32& exitCode)
at Veeam.Backup.Core.CSshScriptInvoker.ExecScriptInner(String localPath, TimeSpan timeout, Boolean collectLogs, String stdOutFilePath, String stdErrFilePath, Boolean checkStdErr, Int32& exitCode)
[ScriptInvoker] Failed to execute script over SSH, failing over to VIX.
bash: /tmp/d6791b89-e0b8-4cce-acec-45d682ce1f2c_stop-vra.sh: /bin/bash^M: bad interpreter: No such file or directory
...
[ScriptInvoker] Running Linux script (VIX) 'C:\Backup\Scripts\stop-vra.sh'
[ScriptInvoker] Linux script exit code = '126'
The Fix
The fix is quite easy, either change your text editor to show the correct line endings.
Or create the script on a Linux machine and copy it to your VBR Server.
To fix this issue in your text editor
Notepad++
Edit --> EOL Conversion --> UNIX/OSX Format
Eclipse
File > Convert Line Delimiters To > Unix (LF, \n, 0Α, ¶)
Or change the New text file line delimiter to Other: Unix on Window > Preferences > General > Workspace
Sublime Text Editor
View > Line Endings > Unix
Atom
See this guide.
(Source)
