Category Archives: VMware

VMware Tanzu Header

A guide to vSphere with Tanzu Kubernetes – Day 2 Operations for the VI Admin

VMware, Kubernetes, , , ,
Intro

This blog post is an accompaniment to the session “vSphere with Tanzu Kubernetes – Day 2 Operations for the VI Admin” created by myself and Simon Conyard, with special thanks to the VMware LiveFire Team for allowing us access to their lab environments to create the technical demo recordings.

You can see the full video with technical demos below (1hr 4 minutes). This blog post acts a supplement to the recording.


This session recording was first shown at the Canada VMUG Usercon.

  • You can watch the VMUG session on-demand here.
  • This session is 44 minutes long (and is a little shorter than the one above).

The basic premise of the presentation was set at around a level-100/150 introduction to the Kubernetes world and marrying that to your knowledge of VMware vSphere as a VI Admin. Giving you an insight into most of the common areas you will need to think about when all of a sudden you are asked to deploy Tanzu Kubernetes and support a team of developers.

Help I need somebody I am a Tanzu Admin

Scene Setting

So why are we talking about VMware and Kubernetes? Isn’t VMware the place where I run those legacy things called virtual machines?

Essentially the definition of an application has changed. On the left of the below image, we have the typical Application, we usually talk about the three tier model (Web, App, DB).

However, the landscape is moving towards the right hand side, applications running more like distributed systems. Where the data your need to function is being served, serviced, recorded, and presented not only by virtual machines, but Kubernetes services as well. Kubernetes introduces its own architectures and frameworks, and finally this new buzzword, serverless and functions.

Although you may not be seeing this change happen immediately in your workplace and infrastructure today. It is the direction of the industry.

Did you know, vRealize Automation 8 is built on a modern container-based micro-services architecture.

The defintion of an application has changed

VMware’s Kubernetes offerings

VMware has two core offerings;

  • vSphere Native
  • Multi-Cloud Aligned

Within vSphere there are two types of Kubernetes clusters that run natively within ESXi.

  • Supervisor Kubernetes cluster control plane for vSphere
  • Tanzu Kubernetes Cluster, sometimes also referred to as a “Guest Cluster.”

Supervisor Kubernetes Cluster

This is a special Kubernetes cluster that uses ESXi as its worker nodes instead of Linux.

This is achieved by integrating the Kubernetes worker agents, Spherelets, directly into the ESXi hypervisor. This cluster uses vSphere Pod Service to run container workloads natively on the vSphere host, taking advantage of the security, availability, and performance of the ESXi hypervisor.

The supervisor cluster is not a conformant Kubernetes cluster, by design, using Kubernetes to enhance vSphere. This ultimately provides you the ability to run pods directly on the ESXi host alongside virtual machines, and as the management of Tanzu Kubernetes Clusters.

Tanzu Kubernetes Cluster

To deliver Kubernetes clusters to your developers, that are standards aligned and fully conformant with upstream Kubernetes, you can use Tanzu Kubernetes Clusters (also referred to as “Guest” clusters.)

A Tanzu Kubernetes Cluster is a Kubernetes cluster that runs inside virtual machines on the Supervisor layer and not on vSphere Pods.

As a fully upstream-compliant Kubernetes it is guaranteed to work with all your Kubernetes applications and tools. Tanzu Kubernetes Clusters in vSphere use the open source Cluster API project for lifecycle management, which in turn uses the VM Operator to manage the VMs that make up the cluster.

Supervisor Cluster or Tanzu Kubernetes Cluster, which one should I choose to run my application?

Supervisor Cluster:

Tanzu Kubernetes Cluster:

  • Kubernetes clusters that are fully conformant with upstream Kubernetes
  • Flexible cluster lifecycle management independent of vSphere, including upgrades
  • Ability to add or customize open source & ecosystem tools like Helm Charts
  • Broad support for open-source networking technologies such as Antrea

For further information check out the Whitepaper – VMware vSphere with Kubernetes 101

vSphere Native Deployment Options

The above information covers running Kubernetes on your vSphere platform natively. You can deploy as follows;

VMware Cloud Foundation is an integrated full stack solution, delivering customers a validated architecture bringing together vSphere, NSX for software defined networking, vSAN for software defined storage, and the vRealize Suite for Cloud Management automation and operation capabilities.

Deploying the vSphere Tanzu Kubernetes solution is as simple as a few clicks in a deployment wizard, providing you a fully integrated Kubernetes deployment into the VMware solutions.

Don’t have VCF? Then you can still enable Kubernetes yourself in your vSphere environment using vSphere 7.0 U1 and beyond. There will be extra steps for you to do this, and some of the integrations to the VMware software stack will not be automatic.

The below graphic summarises the deployment steps between both options discussed.

Enabling vSphere with Kubernetes

Multi-cloud Deployment Options

Building on top of the explanation of Tanzu Kubernetes Cluster explained earlier, Tanzu Kubernetes Grid (TKG) is the same easy-to-upgrade, conformant Kubernetes, with pre-integrated and validated components. This multi-cloud Kubernetes offering that you can run both on-premises in vSphere and in the public cloud on Amazon and Microsoft Azure, fully supported by VMware.

  • Tanzu Kubernetes Grid (TKG) is the name used for the deployment option which is multi-cloud focused.
  • Tanzu Kubernetes Cluster (TKC) is the name used for a Tanzu Kubernetes deployment deployed and managed by vSphere Namespace.

tkg platforms

Introducing vSphere Namespaces

When enabling Kubernetes within a vSphere environment a supervisor cluster is created within the VMware Data Center.  This supervisor cluster is responsible for managing all Kubernetes objects within the VMware Data Center, including vSphere Namespaces.  The supervisor cluster communicating with ESXi forms the Kubernetes control plane, for enabled clusters.

sddc running vsphere with kubernetes

A vSphere Namespace is a logical object that is created on the vSphere Kubernetes supervisor cluster.  This object tracks and provides a mechanism to edit the assignment of resources (Compute, Memory, Storage & Network) and access control to Kubernetes resources, such as containers or virtual machines.

You can provide the URL of the Kubernetes control plane to developers as required, where they can then deploy containers to the vSphere Namespaces for which they have permissions.

Resources and permissions are defined on a vSphere Namespace for both Kubernetes containers, consuming resources directly via vSphere, or Virtual Machines configured and provisioned to operate Tanzu Kubernetes Grid (TKG).

Access control

For a Virtual Administrator the way access can be assigned to various Tanzu elements within the Virtual Infrastructure is very similar to any other logical object.

  • Create Roles
  • Assign Permissions to the Role
  • Allocate the Role to Groups or Individuals
  • Link the Group or Individual to inventory objects

With Tanzu those inventory objects include Namespaces’ and Resources.

What I also wanted to highlight was if a Virtual Administrator gave administrative permissions to a Kubernetes cluster, then this has similarities to granting ‘root’ or ‘administrator’ access to a virtual machine.  An individual with these permissions could create and grant permissions themselves, outside of the virtual infrastructure.

Documentation

Access and RBAC Continue reading A guide to vSphere with Tanzu Kubernetes – Day 2 Operations for the VI Admin →

vRealize Operations Header

vRealize Operations – What is the Guest|Page In/Out Rate Metric?

VMware, , , ,

In vRealize Operations 6.3, we added the following Guest Metrics, some of which we require VMware Tools 10.3.X or higher to be present for us to pull the data.

  • Guest|Active File Cache Memory (KB)
  • Guest|Context Swap Rate per second
  • Guest|Free Memory (KB)
  • Guest|Huge Page Size (KB)
  • Guest|Needed Memory (KB)
  • Guest|Page In Rate per second
  • Guest|Page Out Rate per second
  • Guest|Page Size (KB)
  • Guest|Physically Usable Memory (KB)
  • Guest|Remaning Swap Space (KB)
  • Guest|Total Huge Pages

I had someone query the below metrics, and the answer although easy to assume, is not clearly written down and within vROPs you don’t get a description either, so I thought I’d also publish it, in case any inquisitive minds go googling.

vRealize Operations page in rate metric

Guest|Page In Rate

The Rate the Guest OS brings memory back from disk to DIMM per second. Basically, the rate of reads going through paging/cache system.

It includes not just swapfile I/O, but cacheable reads as well (double pages/s). A page that was paged out earlier, has to be brought back first before it can be used. This creates performance issue as the application is waiting longer, as disk is much slower than RAM.
The unit is in number of pages, not MB. It’s not possible to convert due to mix use of Large Page (2 MB) and Page (4 KB).

A process can have concurrent mixed usage of Large and non-Large page in Windows. The page size isn’t a system-wide setting that all processes use. The same is likely true for Linux Huge Pages.

Windows

  • Page Input/sec counter
    • Pages Input/sec is the rate at which pages are read from disk to resolve hard page faults. Hard page faults occur when a process refers to a page in virtual memory that is not in its working set or elsewhere in physical memory, and must be retrieved from disk. When a page is faulted, the system tries to read multiple contiguous pages into memory to maximize the benefit of the read operation. Compare the value of Memory\\Pages Input/sec to the value of Memory\\Page Reads/sec to determine the average number of pages read into memory during each read operation.
    • Windows: Win32_PerfFormattedData_PerfOS_Memory::PagesInputPersec
      https://msdn.microsoft.com/en-us/ie/aa394268(v=vs.94)

Linux

  • Pages Swapped In counter
$ cat /proc/vmstat | grep pgpgin

pgpgin 604222959257
Guest|Page Out Rate

The opposite of the above. This is not as important as the above. Just because a block of memory is moved to disk that does not mean the application experiences memory problem. In many cases, the page that was moved out is the idle page. Windows does not page out any Large Pages.

Windows

  • Page Output/sec counter
    • Pages Output/sec is the rate at which pages are written to disk to free up space in physical memory. Pages are written back to disk only if they are changed in physical memory, so they are likely to hold data, not code. A high rate of pages output might indicate a memory shortage. Windows writes more pages back to disk to free up space when physical memory is in short supply.  This counter shows the number of pages, and can be compared to other counts of pages, without conversion.

Linux

  • Pages Swapped Out counter

Final notes

Page in/out rate includes pages written/read to/from swap file as well as other system files.

It is important to remember these metrics are populated by pulling the data from the performance counters of the Guest OS, hence the need for VMTools. These metrics should not be confused with virtual machine metrics, which are based on the activity of the VM at the vSphere level. Therefore not taking into account what is going on inside the guest itself.

Thanks to Iwan “E1” Rahbook blog post here for helping me figure this out as well.

Regards

Dean Lewis
vRealize Operations Header

How to build vROPs dashboard for tracking VM Growth over X days

VMware, , , , , ,

I came across an interesting query on Reddit regarding vRealize Operations Manager (vROPs), summarised the query;

“Can I have a vROPs report/dashboard which shows me the storage usage by VMs over the past 3 days”

The short answer is yes, and I produced the following dashboard, views and report and uploaded to code.vmware.com for the post author to use.

Basic VM Growth Sparkline Dashboard VM Growth List

My dashboard has two elements to keep things simple;

  • A sparkline widget of VM’s storage used, the time frame show can be controlled in the dashboard view in the top-right hand corner
  • A list view of the VMs storage used covering a few metrics
    • Current Disk used
    • Disk Used (3 days ago)
    • Change of disk used (in GB)
    • Change of disk used (in %)

So, let’s look at how I created this.

Creating a Sparkline widget in the Dashboard

Create your dashboard, which will show you a blank canvas. Set the Dashboard name.

  • Drag the Sparkline Chart widget onto the canvas and resize as needed, you’ll see this option appear by hovering over the edge of the widget.
  • Click the pencil icon to edit the widget settings.

VM Growth Sparkline Chart Widget

Configure the Widget configuration. The most important options here are;

  • Self Provider – On
  • Show Object Name – On
  • Column Sequence – Label First

This means the widget will provide its own metric data to be displayed. It is not linked to other objects on the dashboard, as we are keeping this as a simple view. Continue reading How to build vROPs dashboard for tracking VM Growth over X days →

AVI Networks Header

How to generate self-signed certificates in AVI Networks for use with Windows Services

Networking, VMware, , 
Note: AVI Networks is also referenced as NSX Advanced Load Balancer as the product is absorbed into the VMware Solutions

The AVI Vantage controller has the ability to generate self-signed certificates for use with your services. As Self-Signed will not be trusted by your browser, it is recommended that you only use these for testing your environments.

Generating a Self-Signed certificate with an exportable private key in the UI is quite simple.

1. Go to the Templates view

2. Select the Security tab

3. Select the SSL/TLS Certificates tab

4. Click the green Create button, and select your type of certificate. (in my example I am creating an application certificate)

AVI Networks Create SSL Certificate

Filling out the forms, the options are as you would expect when working with generating a CSR and CAs.

Fill in your details and click Save.

AVI Networks Add Certificate

Once you have a SSL generated, the next step is downloading for use with your servers.

AVI Networks - Download SSL Certificate

As you can see, clicking download doesn’t give you a usable SSL file with the private key, so you need to use a tool such as OpenSSL to take the output and convert it into a usable format. Continue reading for the steps.

AVI Networks Self Signed Certificate

Note: Viewing or exporting the private key generates a system event, tracking the administrator’s export action. Avi Networks recommends using role-based access to control which users are allowed to use a certificate versus export the key.

You can see this below.

AVI Networks SSL Export logged in events

Using OpenSSL to convert the CSR into a PFX file

Download OpenSSL if you do not already have it.

Save the above SSL outputs from AVI Networks interface to two files;

  • Key > certficate.key
  • Certificate > certificate.crt

Next we will run the following command which will create a single PFX file that contains the exportable key.

openssl pkcs12 -export -out {file_name.pfx} -inkey {private key file} -in {CRT file}

After entering the command, you will be prompted to enter and verify an export password to protect the PFX file. You will use this to export the certificates and key.

Breakdown of the command;

openssl – the command for executing OpenSSL
pkcs12 – the file utility for PKCS#12 files in OpenSSL

-export -out certificate.pfx – export and save the PFX file as certificate.pfx

-inkey privateKey.key – use the private key file privateKey.key as the private key to combine with the certificate.

-in certificate.crt – use certificate.crt as the certificate the private key will be combined with.

(Source)

For me, I needed this for us with VMware Horizon, if you hadn’t already realised from the earlier screenshots. So to complete my use case, its a case of opening up the MMC and Certificates snap-in, import the PFX file and input the password.

Regards

Dean Lewis
VMware.cloud .logo

vCenter patching failed to update the VAMI build “Got exception while trying to save metadata to a file: Unexpected content in /etc/issue file”

VMware, , , , , ,

The issue

After patching/upgrading your vCenter 6.7 appliance, the vCenter UI shows the latest build number, but in VAMI you see the older VAMI build number.

To troubleshoot upgrade issues, you can look at the following file;

  • /var/log/vmware/software-packages.log

In the log, you see the following error;

INFO:vmware.vherd.base.software_update:Setting appliance version to 6.7.0.31000 build 13643870

ERROR:vmware.vherd.base.software_update:Got exception while trying to save metadata to a file: Unexpected content in /etc/issue file. Data: {Unique_Data}

The cause

This issue is thrown when a custom login banner is set by configuring the advanced setting “config.etc.issue” and the default values which include the version number and deployment type have been removed.

Default lines example;

VMware vCenter Server Appliance 6.7.0.31000
Type: vCenter Server with an external Platform Services Controller
  • William Lam documents how to configure custom banners in this blog post.

The Fix

To workaround this issue follow the steps below:

  • Modify the /etc/issue file to the original before patching.

The file ‘/etc/issue’ contents can be customized but the defaults lines which has the version number and deployment type must be kept for patching to succeed.

  • Check the VAMI page for product version and type and update the /etc/issue file accordingly.
Example: /etc/issue  :: (Original Content from a LAB).
Note line 1 and 3 should be blank. Line 2 will have the version and line 4 will have the deployment type, as shown in the below example:

root@vcsa1 [ ~ ]# less -N /etc/issue
      1
      2 VMware vCenter Server Appliance 6.7.0.31000
      3
      4 Type: vCenter Server with an external Platform Services Controller
      5
/etc/issue (END)

This issue will be fixed in a future release.

Note: Since I originally drafted this blog post, VMware have now produced an external KB.
https://kb.vmware.com/s/article/76024

Regards

Dean Lewis