vRealize Automation vRealize Orchestrator Dynamic Types Header

How to create vRO Dynamic Types for vRA Custom Resources

This follow on blog post, diving into how we created the vRA integration with DMS comes from Katherine Skilling, who kindly offered to guest spot and provide the additional content regarding the work we have done internally. You can find her details at the end of this blog post.

In an earlier blog post Dean covered the use of vRA (vRealize Automation) Custom Resources in the context of using vRA to create Databases in DMS (Data Management for VMware Tanzu) and how to create custom day 2 actions. In this post, we will look at how we created the Dynamic Types in vRO (vRealize Orchestrator) to facilitate the creation of the custom resources in vRA.

Introduction – What are Dynamic Types?

Dynamic Types are custom objects in vRO created to extend the schema so that you can create and manage 3rd party objects within vRO. Each type has a definition that contains the object’s properties as well as its relationship within the overall namespace which is the top level in the Dynamic Types hierarchy.

As we started working on our use case, we looked at a tool (published on VMware Code) that would generate Dynamic Types based on an API Swagger specification. The problem we encountered was the tool was quite complex and our API Swagger for Data Management for VMware Tanzu (DMS) didn’t seem to quite fit with the expected format.

This meant we ended up with lots of orphaned entries after running the tool and hoping it would do all the heavy lifting for us. After spending some time investigating and troubleshooting it become clear we didn’t understand Dynamic Types, and how they are created sufficiently well enough, to be able to resolve all our issues. Instead, we decided to scale back on our plans and focus on just the database object we really needed initially. We could use it as a learning exercise, and then revisit the generator tool later once we had a more solid foundation.

To get a better understanding of how Dynamic Types work I recommend this blog from Mike Bombard. He walks through a theoretical example using a zoo and animals to show you how objects are related, as well as how to create the required workflows. I like this particular blog as you don’t need to consider how you are going to get values from a 3rd party system, so its easy to follow along and see the places where you would be making an external connection to retrieve data. It also helped me to understand the relationships between objects without getting mixed up in the properties provided by technical objects.

After reading Mike’s post I realised that we only had a single object for our use case, a database within DMS. We didn’t have any other objects related to it, it didn’t have a parent object and it didn’t have any children. So, when we created a Dynamic Type we would need to generate a placeholder object to act as the parent for the database. I choose to name this databaseFolder just for simplicity and because I’m a visual person and like to organise things inside folders. These databaseFolders would not exist in DMS, they are just an object I created within vRO, they have no real purpose or properties to them other than that the DMS databases are their children in the Dynamic Types inventory.

Stub Workflows

When you define a new Dynamic Type, you must create or associate four workflows to it, which are known as stubs:

  1. Find By Id
  2. Find All
  3. Has Children in Relation
  4. Find Relation

These workflows tell vRO how it can find the Dynamic Type and what its position is in the hierarchy in relation to other types. You can create one set of workflows to share across all Types or you can create a set of workflows per Type. For our use case we only needed one set of the workflows, so we created our code such that the workflows would be dedicated to just the database and databaseFolder objects.

It’s important to know that vRO will run these workflows automatically when administrators browse the vRO inventory, or when using Custom Resources within vRA. They are not started manually by administrators, if you do test them by running them manually you may struggle to populate the input values correctly.

I’ll give you a bit of background to the different workflows next.

Find By Id Workflow

This workflow is automatically run whenever vRO needs to locate a particular instance of a Dynamic Type, such as when used with Custom Resources in vRA for self-service provisioning.  The workflow follows these high-level steps:

  1. Check if the object being processed is the parent object (databaseFolder) or the child object (database).
  2. If it is a databaseFolder creates a new Dynamic Type for a databaseFolder.
  3. If it is a database perform the activity required to locate the object using its id value, in our case, this is a REST API call to DMS to retrieve a single database.
  4. Perform any activities required to create the object and set its properties, in our case, this is extracting the database details from the REST API call results as DMS returns values such as the id and the name in a JSON object formatted as a string.

Find All Workflow

This workflow is automatically run whenever vRO needs to locate all instances of a Dynamic Type, such as when the Dynamic Types namespace is browsed in the vRO client when it is called as a sub workflow of the Find Relation workflow. The workflow follows these high-level steps:

  1. Check if the object being processed is the parent object (databaseFolder) or the child object (database).
  2. If it is a databaseFolder creates a new Dynamic Type for a databaseFolder.
  3. If it is a database perform the activity required to locate all instances of the objects, in our case, this is a REST API call to DMS to retrieve all databases.
  4. Perform any activities required to loop through each of the instances found. For each instance create an object and set its properties. In our case, this is extracting all of the database details from the REST API call results, looping through each one, and extracting values such as the id and the name.

Has Children in Relation Workflow

This workflow is used by vRO to determine whether it should expand the hierarchy when an object is selected in the Dynamic Types namespace within the vRO client. If an object has children objects these would be displayed underneath it in the namespace, in the same way as the databases are displayed under the databaseFolders.  The workflow follows these high-level steps:

  1. Check if the object being processed is the parent object (databaseFolder) or the child object (database) by checking its parentType and relationName values which are provided as workflow inputs.
  2. If it is a databaseFolder call the Find Relation workflow to retrieve all related objects.
  3. If it is any other object type set the result to false to indicate that there are no child objects related to the selected object to display in the hierarchy.

Find Relation Workflow

This workflow is used by vRO when an object is selected in the Dynamic Types namespace within the vRO client. If an object has children objects these would be displayed underneath it in the namespace, in the same way as the databases are displayed under the databaseFolders. vRO automatically runs this workflow each time the Dynamic Types namespace is browsed by an administrator to find any related objects it needs to display. The workflow follows these high-level steps:

  1. Check if the object being processed is the parent object (databaseFolder) or the child object (database) by checking its parentType and relationName values which are provided as workflow inputs.
  2. If it is a databaseFolder and the relationName value is “namespace-children” which is a special value assigned to the very top level in the selected namespace, then create a new Dynamic Type for a databaseFolder.
  3. If it is a database set the type to DMS.database and then call the Find All workflow to retrieve the Dynamic Type objects for all database instances

Creating a Dynamic Type

Defining a Namespace

The first stage in creating our Dynamic Type is to define a new Namespace. Continue reading How to create vRO Dynamic Types for vRA Custom Resources

Kubernetes

Kubernetes – Kubelet Unable to attach or mount volumes – timed out waiting for the condition

The Issue

When I updated my Kasten application in my Kubernetes cluster, I found that one of the pods was stuck in “init” status.

[email protected] [ ~ ] (⎈ |[email protected]:default) # k get pods -n kasten-io -w
NAME READY STATUS RESTARTS AGE
aggregatedapis-svc-78564d4697-wl9wg 1/1 Running 0 3m9s
auth-svc-7977b9684b-zph27 1/1 Running 0 3m11s
catalog-svc-7ff7779b75-kmvsr 0/2 Init:0/2 0 2m43s

kubectl get pods - status init

Running a describe on that pod pointed to the fact the volume could not be attached.

Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m58s default-scheduler Successfully assigned kasten-io/catalog-svc-7ff7779b75-kmvsr to tkg-wld-01-md-0-54598b8d99-rpqjf
Warning FailedMount 55s kubelet Unable to attach or mount volumes: unmounted volumes=[catalog-persistent-storage], unattached volumes=[k10-k10-token-lbqpw catalog-persistent-storage]: timed out waiting for the condition
kubelet Unable to attach or mount volumes- unmounted volumes=[catalog-persistent-storage], unattached volumes=[k10-k10-token-lbqpw catalog-persistent-storage]- timed out waiting for the condition
The Cause

Some where along the line I found some stale volumeattachments linked to Kubernetes node that no longer exist in my cluster. This looks to be causing some confusion in the cluster who should be attaching the volume

The image below shows:

  • Find the Persistent Volume name linked to the associated claim for the failure in the pod events
  • Map this to the available VolumeAttachments
  • Reference VolumeAttachments for each node to available nodes in the cluster
    • I’ve highlighted the missing node in the red box

kubectl get pv - get volumeattachment - get nodes

The Fix

The fix is to remove the stale VolumeAttachment.

kubectl delete volumeattachment [volumeattachment_name]

kubectl delete volumeattachment

After this your pod should eventually pick up and retry, or you could remove the pod and let Kubernetes replace it for you (so long as it’s part of a deployment or other configuration managing your application).

Regards

Dean Lewis

VMware Tanzu Header

vSphere with Tanzu – Creating cluster fails with “storage class is not valid”

The Issue

When you have attached a vSphere Storage Policy to your vSphere Namespace, and tried to create a cluster using the Storage Policy Name, you find it will fail with an error such as:

Error from server (storage class is not valid for control plane VM: StorageClass 'Tanzu Storage Policy' is not assigned for namespace 'deanl', 

storage class is not valid for worker VMs: StorageClass 'Tanzu Storage Policy' is not assigned for namespace 'deanl', 

storage class Tanzu Storage Policy under spec.settings.storage.defaultClass is not valid: StorageClass 'Tanzu Storage Policy' is not assigned for namespace 'deanl'): error when creating "cluster.yaml": 

admission webhook "default.validating.tanzukubernetescluster.run.tanzu.vmware.com" denied the request: storage class is not valid for control plane VM: StorageClass 'Tanzu Storage Policy' is not assigned for namespace 'deanl', 

storage class is not valid for worker VMs: StorageClass 'Tanzu Storage Policy' is not assigned for namespace 'deanl', 

storage class Tanzu Storage Policy under spec.settings.storage.defaultClass is not valid: StorageClass 'Tanzu Storage Policy' is not assigned for namespace 'deanl'

When you look at the vSphere namespace, the Storage Policy is attached.

vSphere Namespace Storage Policy

And example of the erroneous Tanzu Cluster definition YAML:

apiVersion: run.tanzu.vmware.com/v1alpha1
kind: TanzuKubernetesCluster
metadata:
  name: deanl-cluster
  namespace: deanl
spec:
  distribution:
    version: v1.18.5
  topology:
    controlPlane:
      class: best-effort-small
      count: 1
      storageClass: "Tanzu Storage Policy"
    workers:
      class: best-effort-small
      count: 1
      storageClass: "Tanzu Storage Policy"
  settings:
    network:
      cni:
        name: calico
    storage:
      defaultClass: "Tanzu Storage Policy"
The Cause

Continue reading vSphere with Tanzu – Creating cluster fails with “storage class is not valid”

DMS - vRA Header

Data Management for VMware Tanzu with vRealize Automation as Custom Resources

In this blog post, we will cover the technical configuration to import the packages that myself and Katherine Skilling (Twitter, LinkedIn, Blog) have created.

This work is to show the possibility of creating custom workflows to integrate other products that are not natively supported within vRA, by exploiting Dynamic Types. A further write-up will detail the technical configurations of how this integration was created.

You can read this blog post on how to create Dynamic Types in vRealize Orchestrator to be used as custom resources in vRealize Automation:

High-Level Overview

This blog post focuses on integrating “Data Management for VMware Tanzu”, you can read more here about this product:

These packages offer the following capabilities:

  • vRA Cloud Assembly Custom Resource for Data Management with VMware Tanzu
    • Create a database instance
    • Delete a database instance (clean up when a deployment is deleted)
    • Day 2 actions for database instance
      • Scale database instance resources
      • Point in Time Backup of database instance
      • Power-On database instance
      • Power-Off database instance
Pre-Requisites
  • Data Management for VMware Tanzu platform deployed and configured
    • Agent appliance deployed and environment configured.
    • Organisation configured with Org Admin user account.
  • vRealize Automation deployed and configured
    • Using embedded vRO will be fine
    • vRA needs to be able to connect to the DMS system over HTTPs, so appropriate routes and firewalls configured.
  • Grab the files from this location
Recording

Below is a 25 minute recording showing you how to implement the documented steps that follow in this blog post.

Importing & Configuring the vRealize Orchestrator packages

From the downloaded files under the folder “vRealize Orchestrator” there is two files:

  • com.vmware.dms.backup.package
  • dms-dynamictypes-config.package

Open the vRealize Orchestrator UI (https://{vro-url}/orchestration-ui)

  • Left-hand navigation pane > Assets > Packages > Import

DMS - vRO import package

  • Select the file name “com.vmware.dms.backup.package”
  • Select to trust the package and click import

Continue reading Data Management for VMware Tanzu with vRealize Automation as Custom Resources

vRealize Automation - Code Stream Header

vRealize Automation 8.5 – Code Stream – Unexpected character (‘\\’ (code 92))

The Issue

When running certain REST Tasks in Code Stream, the pipeline execution fails on this task with:

Unexpected character ('\\' (code 92)): was expecting double-quote to start field name \n at [Source: (io.netty.buffer.ByteBufInputStream); line: X, colume: X]"
vRA 8.5 Code Stream - Unexpected character was expecting double-quote to start field name - Error
The Workaround

Continue reading vRealize Automation 8.5 – Code Stream – Unexpected character (‘\\’ (code 92))