When deploying Red Hat OpenShift to VMware vSphere platform, there are two methods:
User Provisioned Infrastructure (UPI)
Installer Provisioned Infrastructure (IPI)
There are several great blogs covering both options and deployment methods.
In this blog, we are going to use the IPI method but customize the settings of our Virtual Machines that are deployed setting CPU and Memory that is different from the default settings.
Getting Started
Setting up your Jump host Machine
I’ll be using an Ubuntu Machine as my jumphost for the deployment.
Download the OpenShift-Install tool and OC command line tool. (I’ve used version 4.6.4 in my install)
You will need an account to connect to vCenter with the correct permissions for the OpenShift-Install to deploy the cluster. If you do not want to use an existing account and permissions, you can use this PowerCLI script to create the roles with the correct privileges based on the Red Hat documentation.
If you are installing into VMware Cloud on AWS, like myself, you will also need to allow connectivity from your segments as follows:
Compute gateway
OCP Cluster network to the internet
OCP Cluster network to your SDDC Management Network
Management gateway
OCP Cluster network to ESXi – HTTPs traffic
DNS Records – You will need the two following records to be available on your OCP Cluster network in the same IP address space that your nodes will be deployed to.
Working with Red Hat OpenShift on vSphere, I’m really starting to understand the main infrastructure components and how everything fits together.
Next up was understanding how to control the cluster size after initial deployment. So, with Red Hat OpenShift, there are some basic concepts we need to understand first, before we jump into the technical how-to’s below in this blog.
In this blog I will cover the following;
- Understanding the concepts behind controlling Machines in OpenShift
- Editing your MachineSet to control your Virtual Machine Resources
- Editing your MachineSet to scale your cluster manually
- Deleting a node
- Configuring ClusterAutoscaler to automatically scale your environment
Machine API
The Machine API is a combination of primary resources that are based on the upstream Cluster API project and custom OpenShift Container Platform resources.
The Machine API performs all node host provisioning management actions as a post cluster installation method, providing you dynamic provisioning on top of your VMware vSphere platform (and other public/private cloud platforms).
The two primary resources are:
Machines
An object that describes the host for a Node. A machine has a providerSpec, which describes the types of compute nodes that are offered for different cloud platforms. For example, a machine type for a worker node on Amazon Web Services (AWS) might define a specific machine type and required metadata.
MachineSets
Groups of machines. MachineSets are to machines as ReplicaSets are to Pods. If you need more machines or must scale them down, you change the replicas field on the MachineSet to meet your compute need.
These custom resources add capabilities to your OpenShift cluster:
MachineAutoscaler
This resource automatically scales machines in a cloud. You can set the minimum and maximum scaling boundaries for nodes in a specified MachineSet, and the MachineAutoscaler maintains that range of nodes. The MachineAutoscaler object takes effect after a ClusterAutoscaler object exists. Both ClusterAutoscaler and MachineAutoscaler resources are made available by the ClusterAutoscalerOperator.
ClusterAutoscaler
This resource is based on the upstream ClusterAutoscaler project. In the OpenShift Container Platform implementation, this is integrated with the Machine API by extending the MachineSet API. You can set cluster-wide scaling limits for resources such as cores, nodes, memory, GPU, etc. You can configure priorities so that the cluster prioritizes pods so that new nodes are not brought online for less important pods. You can also set the ScalingPolicy, so that for example, you can scale up nodes but not scale down the node count.
MachineHealthCheck
This resource detects when a machine is unhealthy, deletes it, and, on supported platforms, creates a new machine. You can read more here about this technology preview feature in OCP 4.6.
Editing your MachineSet to control your Virtual Machine Resources
This blog post is an accompaniment to the session “vSphere with Tanzu Kubernetes – Day 2 Operations for the VI Admin” created by myself and Simon Conyard, with special thanks to the VMware LiveFire Team for allowing us access to their lab environments to create the technical demo recordings.
You can see the full video with technical demos below (1hr 4 minutes). This blog post acts a supplement to the recording.
This session is 44 minutes long (and is a little shorter than the one above).
The basic premise of the presentation was set at around a level-100/150 introduction to the Kubernetes world and marrying that to your knowledge of VMware vSphere as a VI Admin. Giving you an insight into most of the common areas you will need to think about when all of a sudden you are asked to deploy Tanzu Kubernetes and support a team of developers.
Scene Setting
So why are we talking about VMware and Kubernetes? Isn’t VMware the place where I run those legacy things called virtual machines?
Essentially the definition of an application has changed. On the left of the below image, we have the typical Application, we usually talk about the three tier model (Web, App, DB).
However, the landscape is moving towards the right hand side, applications running more like distributed systems. Where the data your need to function is being served, serviced, recorded, and presented not only by virtual machines, but Kubernetes services as well. Kubernetes introduces its own architectures and frameworks, and finally this new buzzword, serverless and functions.
Although you may not be seeing this change happen immediately in your workplace and infrastructure today. It is the direction of the industry.
Within vSphere there are two types of Kubernetes clusters that run natively within ESXi.
Supervisor Kubernetes cluster control plane for vSphere
Tanzu Kubernetes Cluster, sometimes also referred to as a “Guest Cluster.”
Supervisor Kubernetes Cluster
This is a special Kubernetes cluster that uses ESXi as its worker nodes instead of Linux.
This is achieved by integrating the Kubernetes worker agents, Spherelets, directly into the ESXi hypervisor. This cluster uses vSphere Pod Service to run container workloads natively on the vSphere host, taking advantage of the security, availability, and performance of the ESXi hypervisor.
To learn more about how vSphere delivers Kubernetes natively
The supervisor cluster is not a conformant Kubernetes cluster, by design, using Kubernetes to enhance vSphere. This ultimately provides you the ability to run pods directly on the ESXi host alongside virtual machines, and as the management of Tanzu Kubernetes Clusters.
Tanzu Kubernetes Cluster
To deliver Kubernetes clusters to your developers, that are standards aligned and fully conformant with upstream Kubernetes, you can use Tanzu Kubernetes Clusters (also referred to as “Guest” clusters.)
A Tanzu Kubernetes Cluster is a Kubernetes cluster that runs inside virtual machines on the Supervisor layer and not on vSphere Pods.
As a fully upstream-compliant Kubernetes it is guaranteed to work with all your Kubernetes applications and tools. Tanzu Kubernetes Clusters in vSphere use the open source Cluster API project for lifecycle management, which in turn uses the VM Operator to manage the VMs that make up the cluster.
Supervisor Cluster or Tanzu Kubernetes Cluster, which one should I choose to run my application?
Supervisor Cluster:
Has additional capabilities that are inherent in the vSphere environment and are available to Kubernetes via the kubectl command
Provides the ability to manage containers just as you would manage virtual machines
Provides stronger security and resource isolation due to the use of vSphere Pods
VMware Cloud Foundation is an integrated full stack solution, delivering customers a validated architecture bringing together vSphere, NSX for software defined networking, vSAN for software defined storage, and the vRealize Suite for Cloud Management automation and operation capabilities.
Deploying the vSphere Tanzu Kubernetes solution is as simple as a few clicks in a deployment wizard, providing you a fully integrated Kubernetes deployment into the VMware solutions.
Don’t have VCF? Then you can still enable Kubernetes yourself in your vSphere environment using vSphere 7.0 U1 and beyond. There will be extra steps for you to do this, and some of the integrations to the VMware software stack will not be automatic.
The below graphic summarises the deployment steps between both options discussed.
Building on top of the explanation of Tanzu Kubernetes Cluster explained earlier, Tanzu Kubernetes Grid (TKG) is the same easy-to-upgrade, conformant Kubernetes, with pre-integrated and validated components. This multi-cloud Kubernetes offering that you can run both on-premises in vSphere and in the public cloud on Amazon and Microsoft Azure, fully supported by VMware.
Tanzu Kubernetes Grid (TKG) is the name used for the deployment option which is multi-cloud focused.
Tanzu Kubernetes Cluster (TKC) is the name used for a Tanzu Kubernetes deployment deployed and managed by vSphere Namespace.
Introducing vSphere Namespaces
When enabling Kubernetes within a vSphere environment a supervisor cluster is created within the VMware Data Center. This supervisor cluster is responsible for managing all Kubernetes objects within the VMware Data Center, including vSphere Namespaces. The supervisor cluster communicating with ESXi forms the Kubernetes control plane, for enabled clusters.
A vSphere Namespace is a logical object that is created on the vSphere Kubernetes supervisor cluster. This object tracks and provides a mechanism to edit the assignment of resources (Compute, Memory, Storage & Network) and access control to Kubernetes resources, such as containers or virtual machines.
You can provide the URL of the Kubernetes control plane to developers as required, where they can then deploy containers to the vSphere Namespaces for which they have permissions.
Resources and permissions are defined on a vSphere Namespace for both Kubernetes containers, consuming resources directly via vSphere, or Virtual Machines configured and provisioned to operate Tanzu Kubernetes Grid (TKG).
For a Virtual Administrator the way access can be assigned to various Tanzu elements within the Virtual Infrastructure is very similar to any other logical object.
Create Roles
Assign Permissions to the Role
Allocate the Role to Groups or Individuals
Link the Group or Individual to inventory objects
With Tanzu those inventory objects include Namespaces’ and Resources.
What I also wanted to highlight was if a Virtual Administrator gave administrative permissions to a Kubernetes cluster, then this has similarities to granting ‘root’ or ‘administrator’ access to a virtual machine. An individual with these permissions could create and grant permissions themselves, outside of the virtual infrastructure.
Whilst working on a Kubernetes demo for a customer, I was cleaning up my environment and deleting persistent volume claims (PVC) that were no longer need.
I noticed that one PVC was stuck in “terminating” status for quite a while.
Note: I am using the OC commands in place of kubectl due to this being a Openshift environment
The Cause
I had a quick google and found I needed to verify if the PVC is still attached to a node in the cluster.
kubectl get volumeattachment
I could see it was, and the reason behind this was the configuration for the PVC was not fully updated during the delete process.
Here I updated the vSphere CSI driver to work the additional security constraints that are baked into OpenShift 4.x.
Since then, once of the things that has been on my list to test is file volumes backed by vSAN File shares. This feature is available in vSphere 7.0.
Well I’m glad to report it does in fact work, by using my CSI driver (see above blog or my github), you can simply deploy consume VSAN File services, as per the documentation here.
I’ve updated my examples in my github repository to get this working.
OK just tell me what to do…
First and foremost, you need to add additional configuration to the csi conf file (csi-vsphere-for-ocp.conf).
If you do not, the defaults will be assumed which is full read-write access from any IP to the file shares created.
[Global]
# run the following on your OCP cluster to get the ID
# oc get clusterversion -o jsonpath='{.items[].spec.clusterID}{"\n"}'
cluster-id = c6d41ba1-3b67-4ae4-ab1e-3cd2e730e1f2
[NetPermissions "A"]
ips = "*"
permissions = "READ_WRITE"
rootsquash = false
[VirtualCenter "10.198.17.253"]
insecure-flag = "true"
user = "administrator@vsphere.local"
password = "Admin!23"
port = "443"
datacenters = "vSAN-DC"
targetvSANFileShareDatastoreURLs = "ds:///vmfs/volumes/vsan:52c229eaf3afcda6-7c4116754aded2de/"
Next, create a storage class which is configured to consume VSAN File services.
