Tag Archives: VMware

VMware Change Block Tracking Issue - Header

vSphere data loss bug returns – CBT issues in vSphere ESXI 8.0 update 2

General, Veeam, VMware, , , , , , , , , ,
The Issue

I keep saying, there are no new ideas in technology, just re-hashes of old ones. That is also true for VMware and their data loss issues.

The vSphere-based change block tracking (CBT) bug is back! I think I wrote 5 articles on this back in 2014/2015 with explanations and fixes!

Veeam reported this at the start of week commencing 11th December 2023, with VMware confirming the issue by the end of the same week.

The Cause

Change block tracking is the feature used to see which blocks of data have changed since a known point in time, to enable backup software to capture only the incremental changes.

If this feature fails, you could lose data in your backups, as the backup software doesn’t know which blocks to protect.

as per VMware:

CBT's QueryChangedDiskAreas may lose some data changed on the disk after disk is hot-extended.
It only happens on ESXi 8.0u2.
The Fix/Workaround

Directly from VMware’s newly published KB, which took them only a few days to confirm this behaviour after Veeam noticed at the start of the week!

  • Resolution
    • Unfortunately, there is no fix available for this bug at this time. However, you can use the following workaround to work around the issue until a fix is released
  • Workaround
    1. Reset CBT after disk is hot-extended. Then, user need to take a full backup immediately.
      It does not fix existing backups, but it makes sure the new ones are good.
    2. Or, user extend disk in offline.

You cannot fix your existing incremental backups if they have been affected, if they missed the correct data to backup, it’s been missed! But you can run an Active Full backup to capture everything, certainly for Veeam this is the case, other backup vendors you’ll need to double check with!

How do I reset Change Block Tracking?

If you are using Veeam, you can just perform an Active Full backup, and ensure the reset CBT option is configured. This is enabled by default.

If you aren’t using Veeam, then the following will be your next steps.

To reset Change Block Tracking, as per this older VMware KB article from the last time this was an issue. VMware may update this article or produce another one now this recent bug has been found.

  • Find your VM in the vCenter Client
    • Power the VM off
    • Click the Options tab, select the Advanced section and then click Configuration Parameters.
  • Disable CBT for the virtual machine by setting the ctkEnabled value to false.
  • If you need to do this for specific virtual disks attached to your virtual machine
    • Disable CBT by configuring the scsix:x.ctkEnabled value for each attached virtual disk to false. (scsix:x is SCSI controller and SCSI device ID of your virtual disk.)
  • Ensure there are no snapshot files (.delta.vmdk) present in the virtual machine’s working directory. For more information, see Determining if there are leftover delta files or snapshots that VMware vSphere or Infrastructure Client cannot detect (1005049).
  • Delete any -CTK.VMDK files within the virtual machine’s working directory.

Now power on your virtual machine.

Depending on your backup software vendor, you may need to manually re-enable Change Block Tracking, you can find a full list of steps and considerations in this VMware KB article. It’s essentially power down the VM, enable in value again in configuration parameters.

Summary

Let’s hope VMware produces a fix for this quickly, I remember they had this issue in vSphere 5.5 and 6.0 and some fixes didn’t resolved the issue, it was a pain being a consultant having to install fixes at customers sites.

It’s good that VMware have only taken a short amount of time to validate this bug and publish something officially about it!

 

Regards

Dean Lewis

VMware Aria Hub Header

VMware Aria Hub and AWS Setup: A Guide to Getting Started

VMware, , , , ,

In this blog post, I am going to take you through how to get started with VMware Aria Hub, and connect your first public cloud account, in this example, AWS.

What is VMware Aria Hub?

Before we dive into the technical pieces, what is VMware Aria Hub?

If we take the marketing definition:

VMware Aria Hub is a transformational multi-cloud management solution unifying cost, performance, and config and delivery automation in a single platform with a common control plane and data model for any cloud, any platform, any tool, and every persona

To make this simple, VMware Aria Hub is one of the key SaaS based services which sits at the center of the new VMware Aria Cloud Management platform. In which it gives you a single control plane to be able to access and interrogate data across the previously named vRealize Suite of products, now rebranded as Aria [insert product name], store metadata from all of your Infrastructure platforms (VMware, AWS, Azure, Google) and in the future, bring in data from third party systems.

This centralization of data is key. That part in VMware Aria, is called “Aria Graph”, which uses an Entity Datastore, a component derived from an existing VMware product, CloudHealth SecureState product (now VMware Aria Automation for Secure Clouds). This unique component, which is based on GraphQL, provides the product a unique way to store data, query into other products, and enable the consumer to write new data into the platform as well.

Let’s take this practical example, I have my application which is made up of the typical three tier app standards:

  • Load Balancer – AWS
  • 2 x Web Servers – AWS
  • App Server – AWS
  • Database Server – On-Prem DC – vSphere

All these components are deployed by Aria Automation (vRealize Automation), monitored by Aria Operations (vRealize Operations), with application logs sent to Aria Operations for Logs (vRealize Log Insight). The AWS environment is further secured by Aria Automation for Secure Clouds (CloudHealth SecureState), which ensures a number of specific resource tags exist, and they conform to the appropriate CIS benchmark.

Now If I need to query the following information for my application; App owner (who deployed it), Cost Centre, Resource Sizing, and active security alerts. I will need to pretty much either browse the UI or query the API for each of the products mentioned.

By leveraging the new capabilities of VMware Aria Hub, I can browse a single interface to reference all the components of my application, and where this data is stored into the other Aria products, it will pull that data through for me. This would be the same if I am querying for information via the VMware Aria Graph as well, for my programmatic access.

Watch the recording!

As a growing trend is video content, I’ve also produced a recording of the same content of this blog post! So, you can follow along below!

Getting Started with Aria Hub

First, you should have an email from VMware welcoming you to the VMware Aria Hub Free Tier. Below I’ve provided a sample email, there are three things to note;

  • You need to click on the links in step 1 + 2 to activate the VMware Aria Hub product within the VMware Cloud Services Portal, and enable the Free Tier for VMware Aria Automation for Secure Clouds, which provides the Public Cloud Security Features into the Aria Hub UI
  • To setup your VMware Cloud Services Portal organisation and enable the product, there is a PDF attached to the email showing the step-by-step instructions and screenshots if needed (shown in the green box).

VMware Aria Hub - Getting Started with AWS - Welcome Email

Once enabled, in the VMware Cloud Services Portal, click the VMware Aria Hub tile (as in the above email screenshot, step 3).

This will present you with the below opening page.

To get started, you only have one option here:

  • Click the “Connect your first data source” blue button.

Continue reading VMware Aria Hub and AWS Setup: A Guide to Getting Started →

Red Hat OpenShift + VMware Header

OpenShift on VMware – Integrating with vSphere Storage, Networking and Monitoring.

VMware, Kubernetes, , ,

I was honoured to be a guest on the “Ask an OpenShift Admin” webinar recently. Where I had the chance to talk about OpenShift on VMware, always a hot topic, and how we co-innovate and work together on solutions.

You can watch the full session below. Keep reading to see the content I didn’t get to cover on a separate recording I’ve produced.

Ask an OpenShift Admin (Ep 54): OpenShift on VMware and the vSphere Kubernetes Drivers Operator

However, I had a number of topics and demo’s planned, that we never got time to visit. So here is the full content I had prepared.

Some of the areas in this webinar and my additional session we covered were:

  • Answering questions live from the views (anything on the table)
  • OpenShift together with VMware
  • Common issues and best practices for deploying OpenShift on VMware vSphere
  • Consuming your vSphere Storage in OpenShift
  • Integrating with the VMware Network stack
  • Infrastructure Up Monitoring
OpenShift on VMware – Integrating with vSphere Storage, Networking and Monitoring

Resources

Regards

Dean Lewis

OpenShift

Red Hat OpenShift on VMware vSphere – How to Scale and Edit your cluster deployments

Kubernetes, VMware, , , ,

Working with Red Hat OpenShift on vSphere, I’m really starting to understand the main infrastructure components and how everything fits together.

Next up was understanding how to control the cluster size after initial deployment. So, with Red Hat OpenShift, there are some basic concepts we need to understand first, before we jump into the technical how-to’s below in this blog.

In this blog I will cover the following;

- Understanding the concepts behind controlling Machines in OpenShift
- Editing your MachineSet to control your Virtual Machine Resources
- Editing your MachineSet to scale your cluster manually
- Deleting a node
- Configuring ClusterAutoscaler to automatically scale your environment

Machine API

The Machine API is a combination of primary resources that are based on the upstream Cluster API project and custom OpenShift Container Platform resources.

The Machine API performs all node host provisioning management actions as a post cluster installation method, providing you dynamic provisioning on top of your VMware vSphere platform (and other public/private cloud platforms).

The two primary resources are:

Machines
An object that describes the host for a Node. A machine has a providerSpec, which describes the types of compute nodes that are offered for different cloud platforms. For example, a machine type for a worker node on Amazon Web Services (AWS) might define a specific machine type and required metadata.
MachineSets
Groups of machines. MachineSets are to machines as ReplicaSets are to Pods. If you need more machines or must scale them down, you change the replicas field on the MachineSet to meet your compute need.

These custom resources add capabilities to your OpenShift cluster:

MachineAutoscaler
This resource automatically scales machines in a cloud. You can set the minimum and maximum scaling boundaries for nodes in a specified MachineSet, and the MachineAutoscaler maintains that range of nodes. The MachineAutoscaler object takes effect after a ClusterAutoscaler object exists. Both ClusterAutoscaler and MachineAutoscaler resources are made available by the ClusterAutoscalerOperator.
ClusterAutoscaler
This resource is based on the upstream ClusterAutoscaler project. In the OpenShift Container Platform implementation, this is integrated with the Machine API by extending the MachineSet API. You can set cluster-wide scaling limits for resources such as cores, nodes, memory, GPU, etc. You can configure priorities so that the cluster prioritizes pods so that new nodes are not brought online for less important pods. You can also set the ScalingPolicy, so that for example, you can scale up nodes but not scale down the node count.

MachineHealthCheck

This resource detects when a machine is unhealthy, deletes it, and, on supported platforms, creates a new machine. You can read more here about this technology preview feature in OCP 4.6.

Editing your MachineSet to control your Virtual Machine Resources

To view the current MachineSet objects available run; Continue reading Red Hat OpenShift on VMware vSphere – How to Scale and Edit your cluster deployments →

VMware.cloud .logo

vCenter patching failed to update the VAMI build “Got exception while trying to save metadata to a file: Unexpected content in /etc/issue file”

VMware, , , , , ,

The issue

After patching/upgrading your vCenter 6.7 appliance, the vCenter UI shows the latest build number, but in VAMI you see the older VAMI build number.

To troubleshoot upgrade issues, you can look at the following file;

  • /var/log/vmware/software-packages.log

In the log, you see the following error;

INFO:vmware.vherd.base.software_update:Setting appliance version to 6.7.0.31000 build 13643870

ERROR:vmware.vherd.base.software_update:Got exception while trying to save metadata to a file: Unexpected content in /etc/issue file. Data: {Unique_Data}

The cause

This issue is thrown when a custom login banner is set by configuring the advanced setting “config.etc.issue” and the default values which include the version number and deployment type have been removed.

Default lines example;

VMware vCenter Server Appliance 6.7.0.31000
Type: vCenter Server with an external Platform Services Controller
  • William Lam documents how to configure custom banners in this blog post.

The Fix

To workaround this issue follow the steps below:

  • Modify the /etc/issue file to the original before patching.

The file ‘/etc/issue’ contents can be customized but the defaults lines which has the version number and deployment type must be kept for patching to succeed.

  • Check the VAMI page for product version and type and update the /etc/issue file accordingly.
Example: /etc/issue  :: (Original Content from a LAB).
Note line 1 and 3 should be blank. Line 2 will have the version and line 4 will have the deployment type, as shown in the below example:

root@vcsa1 [ ~ ]# less -N /etc/issue
      1
      2 VMware vCenter Server Appliance 6.7.0.31000
      3
      4 Type: vCenter Server with an external Platform Services Controller
      5
/etc/issue (END)

This issue will be fixed in a future release.

Note: Since I originally drafted this blog post, VMware have now produced an external KB.
https://kb.vmware.com/s/article/76024

Regards

Dean Lewis