Category Archives: VMware

OpenShift

How to specify your vSphere virtual machine resources when deploying Red Hat OpenShift

When deploying Red Hat OpenShift to VMware vSphere platform, there are two methods:

  • User Provisioned Infrastructure (UPI)
  • Installer Provisioned Infrastructure (IPI)

There are several great blogs covering both options and deployment methods.

In this blog, we are going to use the IPI method but customize the settings of our Virtual Machines that are deployed setting CPU and Memory that is different from the default settings.

Getting Started
Setting up your Jump host Machine

I’ll be using an Ubuntu Machine as my jumphost for the deployment.

Download the OpenShift-Install tool and OC command line tool. (I’ve used version 4.6.4 in my install)

Extract the files and copy to your /usr/bin/local directory

tar -zxvf openshift-client-linux.tar.gz
tar -zxvf openshift-install-linux.tar.gz

Have an available SSH key from your jump box, so that you can connect to your CoreOS VMs one they are deployed for troubleshooting purposes.

You need to download the vCenter trusted root certificates from your instance and import them to your Jump Host.

Curl -O https://{vCenter_FQDN}/certs/download.zip

Then the following to import (ubuntu uses the .crt files, hence importing the win folder);

unzip download.zip
cp certs/win/* /usr/local/share/ca-certificates
update-ca-certificates

You will need an account to connect to vCenter with the correct permissions for the OpenShift-Install to deploy the cluster. If you do not want to use an existing account and permissions, you can use this PowerCLI script to create the roles with the correct privileges based on the Red Hat documentation.

If you are installing into VMware Cloud on AWS, like myself, you will also need to allow connectivity from your segments as follows:

  • Compute gateway
    • OCP Cluster network to the internet
    • OCP Cluster network to your SDDC Management Network
  • Management gateway
    • OCP Cluster network to ESXi – HTTPs traffic

DNS Records – You will need the two following records to be available on your OCP Cluster network in the same IP address space that your nodes will be deployed to.

  • {clusterID}.{domain_name}
    • example: ocp46.veducate.local
  • *.apps.{clusterID}.{domain_name}
    • example: *.apps.ocp46.veducate.local

If your DNS is a Windows server, you can use this script here. Continue reading How to specify your vSphere virtual machine resources when deploying Red Hat OpenShift

OpenShift

Red Hat OpenShift on VMware vSphere – How to Scale and Edit your cluster deployments

Working with Red Hat OpenShift on vSphere, I’m really starting to understand the main infrastructure components and how everything fits together.

Next up was understanding how to control the cluster size after initial deployment. So, with Red Hat OpenShift, there are some basic concepts we need to understand first, before we jump into the technical how-to’s below in this blog.

In this blog I will cover the following;

- Understanding the concepts behind controlling Machines in OpenShift
- Editing your MachineSet to control your Virtual Machine Resources
- Editing your MachineSet to scale your cluster manually
- Deleting a node
- Configuring ClusterAutoscaler to automatically scale your environment

Machine API

The Machine API is a combination of primary resources that are based on the upstream Cluster API project and custom OpenShift Container Platform resources.

The Machine API performs all node host provisioning management actions as a post cluster installation method, providing you dynamic provisioning on top of your VMware vSphere platform (and other public/private cloud platforms).

The two primary resources are:

Machines
An object that describes the host for a Node. A machine has a providerSpec, which describes the types of compute nodes that are offered for different cloud platforms. For example, a machine type for a worker node on Amazon Web Services (AWS) might define a specific machine type and required metadata.
MachineSets
Groups of machines. MachineSets are to machines as ReplicaSets are to Pods. If you need more machines or must scale them down, you change the replicas field on the MachineSet to meet your compute need.

These custom resources add capabilities to your OpenShift cluster:

MachineAutoscaler
This resource automatically scales machines in a cloud. You can set the minimum and maximum scaling boundaries for nodes in a specified MachineSet, and the MachineAutoscaler maintains that range of nodes. The MachineAutoscaler object takes effect after a ClusterAutoscaler object exists. Both ClusterAutoscaler and MachineAutoscaler resources are made available by the ClusterAutoscalerOperator.
ClusterAutoscaler
This resource is based on the upstream ClusterAutoscaler project. In the OpenShift Container Platform implementation, this is integrated with the Machine API by extending the MachineSet API. You can set cluster-wide scaling limits for resources such as cores, nodes, memory, GPU, etc. You can configure priorities so that the cluster prioritizes pods so that new nodes are not brought online for less important pods. You can also set the ScalingPolicy, so that for example, you can scale up nodes but not scale down the node count.

MachineHealthCheck

This resource detects when a machine is unhealthy, deletes it, and, on supported platforms, creates a new machine. You can read more here about this technology preview feature in OCP 4.6.

Editing your MachineSet to control your Virtual Machine Resources

To view the current MachineSet objects available run; Continue reading Red Hat OpenShift on VMware vSphere – How to Scale and Edit your cluster deployments

VMware Tanzu Header

A guide to vSphere with Tanzu Kubernetes – Day 2 Operations for the VI Admin

Intro

This blog post is an accompaniment to the session “vSphere with Tanzu Kubernetes – Day 2 Operations for the VI Admin” created by myself and Simon Conyard, with special thanks to the VMware LiveFire Team for allowing us access to their lab environments to create the technical demo recordings.

You can see the full video with technical demos below (1hr 4 minutes). This blog post acts a supplement to the recording.


This session recording was first shown at the Canada VMUG Usercon.

  • You can watch the VMUG session on-demand here.
  • This session is 44 minutes long (and is a little shorter than the one above).

The basic premise of the presentation was set at around a level-100/150 introduction to the Kubernetes world and marrying that to your knowledge of VMware vSphere as a VI Admin. Giving you an insight into most of the common areas you will need to think about when all of a sudden you are asked to deploy Tanzu Kubernetes and support a team of developers.

Scene Setting

So why are we talking about VMware and Kubernetes? Isn’t VMware the place where I run those legacy things called virtual machines?

Essentially the definition of an application has changed. On the left of the below image, we have the typical Application, we usually talk about the three tier model (Web, App, DB).

However, the landscape is moving towards the right hand side, applications running more like distributed systems. Where the data your need to function is being served, serviced, recorded, and presented not only by virtual machines, but Kubernetes services as well. Kubernetes introduces its own architectures and frameworks, and finally this new buzzword, serverless and functions.

Although you may not be seeing this change happen immediately in your workplace and infrastructure today. It is the direction of the industry.

Did you know, vRealize Automation 8 is built on a modern container-based micro-services architecture.

The defintion of an application has changed

VMware’s Kubernetes offerings

VMware has two core offerings;

  • vSphere Native
  • Multi-Cloud Aligned

Within vSphere there are two types of Kubernetes clusters that run natively within ESXi.

  • Supervisor Kubernetes cluster control plane for vSphere
  • Tanzu Kubernetes Cluster, sometimes also referred to as a “Guest Cluster.”

Supervisor Kubernetes Cluster

This is a special Kubernetes cluster that uses ESXi as its worker nodes instead of Linux.

This is achieved by integrating the Kubernetes worker agents, Spherelets, directly into the ESXi hypervisor. This cluster uses vSphere Pod Service to run container workloads natively on the vSphere host, taking advantage of the security, availability, and performance of the ESXi hypervisor.

The supervisor cluster is not a conformant Kubernetes cluster, by design, using Kubernetes to enhance vSphere. This ultimately provides you the ability to run pods directly on the ESXi host alongside virtual machines, and as the management of Tanzu Kubernetes Clusters.

Tanzu Kubernetes Cluster

To deliver Kubernetes clusters to your developers, that are standards aligned and fully conformant with upstream Kubernetes, you can use Tanzu Kubernetes Clusters (also referred to as “Guest” clusters.)

A Tanzu Kubernetes Cluster is a Kubernetes cluster that runs inside virtual machines on the Supervisor layer and not on vSphere Pods.

As a fully upstream-compliant Kubernetes it is guaranteed to work with all your Kubernetes applications and tools. Tanzu Kubernetes Clusters in vSphere use the open source Cluster API project for lifecycle management, which in turn uses the VM Operator to manage the VMs that make up the cluster.

Supervisor Cluster or Tanzu Kubernetes Cluster, which one should I choose to run my application?

Supervisor Cluster:

Tanzu Kubernetes Cluster:

  • Kubernetes clusters that are fully conformant with upstream Kubernetes
  • Flexible cluster lifecycle management independent of vSphere, including upgrades
  • Ability to add or customize open source & ecosystem tools like Helm Charts
  • Broad support for open-source networking technologies such as Antrea

For further information check out the Whitepaper – VMware vSphere with Kubernetes 101

vSphere Native Deployment Options

The above information covers running Kubernetes on your vSphere platform natively. You can deploy as follows;

VMware Cloud Foundation is an integrated full stack solution, delivering customers a validated architecture bringing together vSphere, NSX for software defined networking, vSAN for software defined storage, and the vRealize Suite for Cloud Management automation and operation capabilities.

Deploying the vSphere Tanzu Kubernetes solution is as simple as a few clicks in a deployment wizard, providing you a fully integrated Kubernetes deployment into the VMware solutions.

Don’t have VCF? Then you can still enable Kubernetes yourself in your vSphere environment using vSphere 7.0 U1 and beyond. There will be extra steps for you to do this, and some of the integrations to the VMware software stack will not be automatic.

The below graphic summarises the deployment steps between both options discussed.

Multi-cloud Deployment Options

Building on top of the explanation of Tanzu Kubernetes Cluster explained earlier, Tanzu Kubernetes Grid (TKG) is the same easy-to-upgrade, conformant Kubernetes, with pre-integrated and validated components. This multi-cloud Kubernetes offering that you can run both on-premises in vSphere and in the public cloud on Amazon and Microsoft Azure, fully supported by VMware.

  • Tanzu Kubernetes Grid (TKG) is the name used for the deployment option which is multi-cloud focused.
  • Tanzu Kubernetes Cluster (TKC) is the name used for a Tanzu Kubernetes deployment deployed and managed by vSphere Namespace.

Introducing vSphere Namespaces

When enabling Kubernetes within a vSphere environment a supervisor cluster is created within the VMware Data Center.  This supervisor cluster is responsible for managing all Kubernetes objects within the VMware Data Center, including vSphere Namespaces.  The supervisor cluster communicating with ESXi forms the Kubernetes control plane, for enabled clusters.

A vSphere Namespace is a logical object that is created on the vSphere Kubernetes supervisor cluster.  This object tracks and provides a mechanism to edit the assignment of resources (Compute, Memory, Storage & Network) and access control to Kubernetes resources, such as containers or virtual machines.

You can provide the URL of the Kubernetes control plane to developers as required, where they can then deploy containers to the vSphere Namespaces for which they have permissions.

Resources and permissions are defined on a vSphere Namespace for both Kubernetes containers, consuming resources directly via vSphere, or Virtual Machines configured and provisioned to operate Tanzu Kubernetes Grid (TKG).

Access control

For a Virtual Administrator the way access can be assigned to various Tanzu elements within the Virtual Infrastructure is very similar to any other logical object.

  • Create Roles
  • Assign Permissions to the Role
  • Allocate the Role to Groups or Individuals
  • Link the Group or Individual to inventory objects

With Tanzu those inventory objects include Namespaces’ and Resources.

What I also wanted to highlight was if a Virtual Administrator gave administrative permissions to a Kubernetes cluster, then this has similarities to granting ‘root’ or ‘administrator’ access to a virtual machine.  An individual with these permissions could create and grant permissions themselves, outside of the virtual infrastructure.

Documentation

Continue reading A guide to vSphere with Tanzu Kubernetes – Day 2 Operations for the VI Admin

vROPs

vRealize Operations – What is the Guest|Page In/Out Rate Metric?

In vRealize Operations 6.3, we added the following Guest Metrics, some of which we require VMware Tools 10.3.X or higher to be present for us to pull the data.

  • Guest|Active File Cache Memory (KB)
  • Guest|Context Swap Rate per second
  • Guest|Free Memory (KB)
  • Guest|Huge Page Size (KB)
  • Guest|Needed Memory (KB)
  • Guest|Page In Rate per second
  • Guest|Page Out Rate per second
  • Guest|Page Size (KB)
  • Guest|Physically Usable Memory (KB)
  • Guest|Remaning Swap Space (KB)
  • Guest|Total Huge Pages

I had someone query the below metrics, and the answer although easy to assume, is not clearly written down and within vROPs you don’t get a description either, so I thought I’d also publish it, in case any inquisitive minds go googling.

vRealize Operations page in rate metric

Guest|Page In Rate

The Rate the Guest OS brings memory back from disk to DIMM per second. Basically, the rate of reads going through paging/cache system.

It includes not just swapfile I/O, but cacheable reads as well (double pages/s). A page that was paged out earlier, has to be brought back first before it can be used. This creates performance issue as the application is waiting longer, as disk is much slower than RAM.
The unit is in number of pages, not MB. It’s not possible to convert due to mix use of Large Page (2 MB) and Page (4 KB).

A process can have concurrent mixed usage of Large and non-Large page in Windows. The page size isn’t a system-wide setting that all processes use. The same is likely true for Linux Huge Pages.

Windows

  • Page Input/sec counter
    • Pages Input/sec is the rate at which pages are read from disk to resolve hard page faults. Hard page faults occur when a process refers to a page in virtual memory that is not in its working set or elsewhere in physical memory, and must be retrieved from disk. When a page is faulted, the system tries to read multiple contiguous pages into memory to maximize the benefit of the read operation. Compare the value of Memory\\Pages Input/sec to the value of Memory\\Page Reads/sec to determine the average number of pages read into memory during each read operation.
    • Windows: Win32_PerfFormattedData_PerfOS_Memory::PagesInputPersec
      https://msdn.microsoft.com/en-us/ie/aa394268(v=vs.94)

Linux

  • Pages Swapped In counter
$ cat /proc/vmstat | grep pgpgin

pgpgin 604222959257
Guest|Page Out Rate

The opposite of the above. This is not as important as the above. Just because a block of memory is moved to disk that does not mean the application experiences memory problem. In many cases, the page that was moved out is the idle page. Windows does not page out any Large Pages.

Windows

  • Page Output/sec counter
    • Pages Output/sec is the rate at which pages are written to disk to free up space in physical memory. Pages are written back to disk only if they are changed in physical memory, so they are likely to hold data, not code. A high rate of pages output might indicate a memory shortage. Windows writes more pages back to disk to free up space when physical memory is in short supply.  This counter shows the number of pages, and can be compared to other counts of pages, without conversion.

Linux

  • Pages Swapped Out counter

Final notes

Page in/out rate includes pages written/read to/from swap file as well as other system files.

It is important to remember these metrics are populated by pulling the data from the performance counters of the Guest OS, hence the need for VMTools. These metrics should not be confused with virtual machine metrics, which are based on the activity of the VM at the vSphere level. Therefore not taking into account what is going on inside the guest itself.

Thanks to Iwan “E1” Rahbook blog post here for helping me figure this out as well.

Regards

vROPs

How to build vROPs dashboard for tracking VM Growth over X days

I came across an interesting query on Reddit regarding vRealize Operations Manager (vROPs), summarised the query;

“Can I have a vROPs report/dashboard which shows me the storage usage by VMs over the past 3 days”

The short answer is yes, and I produced the following dashboard, views and report and uploaded to code.vmware.com for the post author to use.

My dashboard has two elements to keep things simple;

  • A sparkline widget of VM’s storage used, the time frame show can be controlled in the dashboard view in the top-right hand corner
  • A list view of the VMs storage used covering a few metrics
    • Current Disk used
    • Disk Used (3 days ago)
    • Change of disk used (in GB)
    • Change of disk used (in %)

So, let’s look at how I created this.

Creating a Sparkline widget in the Dashboard

Create your dashboard, which will show you a blank canvas. Set the Dashboard name.

  • Drag the Sparkline Chart widget onto the canvas and resize as needed, you’ll see this option appear by hovering over the edge of the widget.
  • Click the pencil icon to edit the widget settings.

Configure the Widget configuration. The most important options here are;

  • Self Provider – On
  • Show Object Name – On
  • Column Sequence – Label First

This means the widget will provide its own metric data to be displayed. It is not linked to other objects on the dashboard, as we are keeping this as a simple view. Continue reading How to build vROPs dashboard for tracking VM Growth over X days