In this blog post I am going to dissect backing up vRealize Automation 8.x using Veeam Backup and Replication.

- Understanding the backup methods
- Performing an online backup
- Performing an offline backup

Understanding the Backup Methods

Reading the VMware documentation around this subject can be somewhat confusing at times. And if you pay attention, there are subtle changes between the documents as well. Lets break this down.

• vRealize Automation 8.0
  • As part of the backup job, you need to run a script to stop the services
  • This is known as an offline backup
  • Depending on your backup software, you can either do this by running a script located on the vRealize Automation appliance or by triggering using the pre-freeze/post-freeze scripts when a snapshot is taken of the VM.
  • The snapshot must not include the virtual machines memory.
  • If you environment is a cluster, you only need to run the script on a single node.
  • All nodes in the cluster must be backed up at the same time.
• vRealize Automation 8.0.1 and 8.1 (and higher)
  • It is supported to run an online backup
    • No script is needed to shut down the services
  • Snapshot taken as part of the backup must quiesce the virtual machine.
  • The snapshot must not include the virtual machines memory.
  • It is recommended to run the script to stop all services and perform an offline backup.
    • You may also find your backup runs faster, as the virtual machine will become less busy.

Performing an Online Backup

Let’s start with the easier of the two options. Again, this will be supported for vRealize Automation 8.0.1 and higher.

When creating your backup job, include all vRealize Automation cluster node virtual machines.

• On the Storage page, click for the advanced settings.

• Select the vSphere Tab
• Tick to Enable VMware Tools quiescence
• Click OK, return to the Storage dialog box and click next.

• On the Guest Processing page, untick all options and click next.

• Click complete the configuration of your backup job by setting a schedule as needed.

When running your backup job, you will see the Veeam Temporary Backup job will not create a snapshot of the virtual machine’s memory. This meets the requirements as stated in the official documentation.

And just to finish this section on creating an online backup, with a screenshot of a successful backup run. In my lab environment I am only running a single machine.

Performing an Offline Backup

Setting up to perform an offline backup is a bit trickier. We need to follow some of the steps in the documentation for vRealize Suite Backup and Restore by Using EMC Avamar – Backing Up the vRealize Automation 8.x Appliance and adapt for Veeam.

Note: you must be running Veeam Backup and Replication 10a (build 10.0.1.4854)

This version contains a bugfix to correctly identify support for PhotonOS guest OS.

You need to create two scripts on the VBR Server, which will bring the vRA Kubernetes cluster offline within the guest OS. VBR will upload these scrips to the vRA node as part of the application aware processing settings discussed below. Please note creating these bash files on Windows may cause issues. See this blog post if you get errors in the job stating; Script finished execution with unexpected exit code: 126.

Create the following files on your VBR Server;

• start-vra.sh

#!/bin/bash
 # Clear services state
 /opt/scripts/deploy.sh --onlyClean>/dev/null 2>&1
 sleep 60
 # Start services
 /opt/scripts/deploy.sh >/dev/null 2>&1
 RETURN_CODE=$?
 sleep 60
 if [[ "$RETURN_CODE" -eq 0 ]]
  then
   echo "$LOG_PREFIX vRealize Automation services successfully started"
   exit 0
  else
   echo "$LOG_PREFIX vRealize Automation services failed to start"
   exit 1
 fi

• stop-vra.sh

#!/bin/bash
 # Stop application services
 /opt/scripts/svc-stop.sh >/dev/null2>&1
 sleep 60
 # Stop infrastructure services
 /opt/scripts/deploy.sh --onlyClean >/dev/null 2>&1
 RETURN_CODE=$?
 sleep 60
 if [[ "$RETURN_CODE" -eq 0 ]]
  then
   echo "$LOG_PREFIX vRealize Automation services successfully stopped"
   exit 0
  else
   echo "$LOG_PREFIX vRealize Automation services failed to stop"
   exit 1
 fi

Next before we setup the Veeam Backup job, you need to setup the following registry key on your VBR server. VMware recommends that the star-vra.sh can take up to 30 minutes to run, which exceeds the default timeout for scripts in Veeam.

Set the below registry key and restart the “Veeam Backup Service” in services.msc.

HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
MaxGuestScriptTimeoutSec
Type: REG_DWORD
Default value: 600
Description: Timeout for pre-freeze and post-thaw scripts, in    seconds.

I recommend setting the value to 2100 seconds, which is 35 minutes. Below is a screenshot of the setting in my lab environment.

Creating the backup Job, you select all your vRA nodes in the cluster. As they must be backed up at the same time as per the VMware documentation.

On your Storage Tab > Advanced > vSphere Tab. Leave “Guest quiescence” unticked. (We are not mirroring the settings for an online backup discussed above in this blog).

On the Guest Processing page > Tick “Enable application aware processing”. Select the “Applications” button.

Add one of the vRA cluster nodes, highlight the object and click to edit this virtual machines settings.

Select the “Scripts” tab.

Select “Require successful script execution and enter the location of the Linux bash scripts on your Windows VBR server. Click ok and save the settings.

Next enter your Guest OS credentials so the VBR Server can copy the scripts across, this needs to be the vRA node root account. Testing this account will fail, as the Mlocate package is not installed on this node. However, you can proceed anyway.

Move onto configuring your Job schedule and completing the backup job configuration.

Running the backup job, on the vRA cluster node where you configured the application aware processing settings, you will see the following lines in the session window;

Running pre-freeze script in the guest OS
Creating VM snapshot 
Running post-thaw script in the guest OS

Finally, if you check the logs, you will see the following;

“C:\ProgramData\Veeam\Backup\{Job_Name}\Task.{VM_Name}.{VM_vCenter_ID}.log”

[ScriptInvoker] Running Linux script (VIX) 'C:\Backup\Scripts\stop-vra.sh'
[VssGAConn('SC-DC1-VRA001')] Initializing VIX connection.
...
Uploading file [C:\Backup\Scripts\stop-vra.sh] to guest [vm-10003] destination [/tmp/vmware-root_405-1849560565/stop-vra.sh]. Ok.
Setting "execute" permission for file [/tmp/vmware-root_405-1849560565/stop-vra.sh]
Changing guest [vm-10003] file [/tmp/vmware-root_405-1849560565/stop-vra.sh] attributes
Changing guest [vm-10003] file [/tmp/vmware-root_405-1849560565/stop-vra.sh] attributes. Ok.
Setting "execute" permission for file [/tmp/vmware-root_405-1849560565/stop-vra.sh]. Ok.
Running guest [vm-10003] program [/tmp/vmware-root_405-1849560565/stop-vra.sh] with arguments []
Waiting for guest [vm-10003] process with PID [28363] to finish, timeout: 2700000, tryRate: 1000
Getting status of the guest [vm-10003] process with PID [28363], timeout: 2700000
Listing processes with PIDs [28363,] in guest [vm-10003]
Listing processes with PIDs [28363,] in guest [vm-10003]. Ok.
Exec script in guest: temp directory found: true, need to be removed: false. Path: /tmp/vmware-root_405-1849560565
[ScriptInvoker] Linux script exit code = '0'
[VssGAConn] Agent should not be uninstalled.
[VssGAConn] Disconnecting
...
[VimApi] Create snapshot, ref "vm-10003", name "VEEAM BACKUP TEMPORARY SNAPSHOT", description "Please do not delete this snapshot. It is being used by Veeam Backup.", memory "False", quiesce "False"
[Soap] Snapshot VM 'vm-10003' was created. Ref: 'snapshot-10400'
Snapshot was created with frozen guest.
...
[ScriptInvoker] Starting post-thaw script execution 
[ScriptInvoker] Running Linux script (SSH) 'C:\Backup\Scripts\start-vra.sh'
...
[ScriptInvoker] Linux script exit code = '0'
[ScriptInvoker] Script finished with exit code = '0'

Just a final note as a shout to OffsiteDataSync, winner of the innovation award at this years VeeamON 2020 conference, whom have provided me Veeam Cloud Connect Backup storage for my lab environment and testing.

Regards

Follow @Saintdle