Category Archives: Kubernetes

VMware Tanzu Mission Control – Getting started with your first cluster

In this blog post we will cover the following topics

- What is Tanzu Mission Control?
- So, this isn't just for VMware environments?
- Getting Started Tanzu Mission Control
- - TMC Resource Hierarchy
- - Creating a Cluster Group
- - Attaching a cluster to Tanzu Mission Control
- - Viewing your Cluster Objects
- - - Overview
- - - Nodes
- - - Namespaces
- - - Workloads
- Where can I demo/test/trial this myself?

The follow up blog posts are;

- Tanzu Mission Control 
- - Cluster Inspections
- - - What Inspections are available 
- - - Performing Inspections 
- - - Viewing Inspections
- - Workspaces and Policies
- - - Creating a workspace 
- - - Creating a managed Namespace 
- - - Policy Driven Cluster Management 
- - - Creating Policies

What is Tanzu Mission Control?

Tanzu Mission control is a cloud offering, which gives you a single point of control, monitoring and management, regardless of the Kubernetes deployment and their location (e.g Tanzu Kubernetes Grid, OpenShift Container Platform, Azure Kubernetes to name but a few).

Key Capabilities;

  • Manage Kubernetes Cluster Lifecycle through the deployment and day 2 operations
  • Attach Clusters for centralized operations and management
  • Centralized policy management
    • Apply access, network and container registry policies consistently across your Kubernetes clusters and namespaces
  • Global visibility for diagnosing and troubleshooting issues with your Kubernetes clusters
  • Inspection runbooks to validate the configuration of your clusters
    • Current offerings are;
      • Conformance; validating binaries running in your cluster to ensure proper configuration and running.
      • CIS benchmark; evaluation against the CIS Benchmark for Kubernetes published by the Center for Internet Security.
      • Lite; node conformance test to validate your nodes meet the Kubernetes requirements.

So, this isn’t just for VMware environments?

Nope, this is a cloud and Kubernetes neutral offering. You can attach CNCF conformant Kubernetes clusters to Tanzu Mission Control no matter where they are running: on vSphere, in any public clouds, or through other Kubernetes vendors.

Getting Started Tanzu Mission Control

TMC Resource Hierarchy

In the Tanzu Mission Control resource hierarchy, there are three levels at which you can specify policies.

  • Organization
  • Object groups (Cluster groups and Workspaces)
  • Kubernetes objects (Clusters and Namespaces)

You can set direct policies for a given object, but each object can also inherit based on the parent objects. So pretty much what you’ve been used to in the past with policies and hierarchies.

Creating a Cluster Group

A Cluster Group is a logical object to bring together multiple Kubernetes clusters. You can set user access policies to be able to view/edit/control cluster group objects and their child objects (clusters).

Cluster groups provide an infrastructure view, and all clusters must be attached to a group.

To create a Cluster Group;

  • Select the Cluster Group from the navigation
  • Click New Cluster Group
  • Supply a name, description and labels are optional and can be edited after creation

Continue reading VMware Tanzu Mission Control – Getting started with your first cluster

VMware Tanzu Mission Control – Workspaces and Policies

In this blog post we will cover the following topics

- Tanzu Mission Control 
- - Workspaces 
- - - Creating a workspace
- - - Creating a managed Namespace
- - - Viewing a managed Namespace
- - Policy Driven Cluster Management
- - - Creating a Image Registry Policy
- - - Creating a Network Policy

The follow up blog posts are;

- Getting Started Tanzu Mission Control
- - TMC Resource Hierarchy
- - Creating a Cluster Group
- - Attaching a cluster to Tanzu Mission Control
- - Viewing your Cluster Objects
- Cluster Inspections
- - Cluster Inspections Overview 
- - What Inspections are available 
- - Performing Inspections 
- - Viewing Inspections

Workspaces

Workspaces provide an application view, where you logically group Kubernetes Namespaces together, regardless of the cluster to which they are attached.

This is in contrast to Cluster Groups, which are focused on the infrastructure view.

These Workspaces can be created to align to your projects or applications, from a hierarchy point of view, you would then authorize your users to these Workspaces, so that they can monitor and manage the namespaces related to their function.

Creating a Workspace

Click the Workspace navigation view on the left-hand side, and then New Workspace.

Specify your Workspace name, and provide the optional description and labels, these can be added after creation if needed.

Now you have a Workspace, it’s no good without any associated Namespaces, so let’s continue.

Creating a managed Namespace

All Namespaces attached to a Workspace will be managed Namespaces under TMC.

To create a managed Namespace, you can do this in one of four places;

  • Within the Workspace Navigation view
  • Inside the Workspace Object itself
  • On the Namespace Navigation view
  • On the Cluster Object > Navigation Tab

Continue reading VMware Tanzu Mission Control – Workspaces and Policies

VMware Tanzu Mission Control – Cluster Inspections

In this blog post we will cover the following topics

- Tanzu Mission Control 
- - Cluster Inspections Overview
- - What Inspections are available
- - Performing Inspections
- - Viewing Inspections

The follow up blog posts are;

- Getting Started Tanzu Mission Control
- - TMC Resource Hierarchy
- - Creating a Cluster Group
- - Attaching a cluster to Tanzu Mission Control
- - Viewing your Cluster Objects
- Workspaces and Policies
- - Creating a workspace 
- - - Creating a managed Namespace 
- - - Viewing a managed Namespace 
- - Policy Driven Cluster Management 
- - - Creating an Image Registry Policy 
- - - Creating a Network Policy

Cluster Inspections Overview

This for me is one of the best features of Tanzu Mission Control, and an area which I expected will be developed further in the future.

Cluster inspections provide a point-in-time report of the condition of the cluster, you might want to run them periodically (to avoid drifting out of conformance) and any time you make significant alterations, such as after you patch or upgrade a cluster.

This capability is achieved by using Sonobuoy, an open source community standard, which provides diagnostics of your Kubernetes environments through conformance testing and additional plugins.

What Inspections are available?

The following cluster inspections are available from the Overview and Inspection tabs of the cluster detail page in the Tanzu Mission Control console.

  • Conformance inspection;

Validates the binaries running on your cluster and ensures that your cluster is properly installed, configured, and working. You can view the generated report from within Tanzu Mission Control to assess and address any issues that arise. For more information, see the Kubernetes Conformance documentation at https://github.com/cncf/k8s-conformance/tree/master/docs.

  • CIS benchmark inspection;

Evaluates your cluster against the CIS Benchmark for Kubernetes published by the Center for Internet Security.

  • Lite inspection;

Is a node conformance test that validates whether nodes meet requirements for Kubernetes. For more information, see Validate node setup in the Kubernetes documentation.

Performing Inspections

To perform an inspection, there are two ways; from the inspections tab when view a cluster object (as in the above screenshot).

Or you can do this from the Inspections navigation page, as below.

Continue reading VMware Tanzu Mission Control – Cluster Inspections

vRealize Operations – Monitoring OpenShift Container Platform environments

The latest release of  vRealize Operations (the “manager” part of the product name has now been dropped), brings the ability to manage your Kubernetes environments from the vSphere infrastructure up.

The Kubernetes integration in vRealize Operations 8.1;

  • vSphere with Kubernetes integration:
    • Ability to discover vSphere with Kubernetes objects as part of the vCenter Server inventory.
    • New summary pages for Supervisor Cluster, Namespaces, Tanzu Kubernetes cluster, and vSphere Pods.
    • ​Out-of-the-box dashboards, alerts, reports, and views for vSphere with Kubernetes.
  • The VMware Management Packs that are new and those that are updated for vRealize Operations Manager 8.1 are:
    • VMware vRealize Operations Management Pack for Container Monitoring 1.4.3

Where does OpenShift Container Platform fit in?

All though the above highlighted release notes point towards vSphere with Kubernetes (aka project pacific), the Container monitoring management pack has been available for a while and has received a number of updates.

This management pack can be used with any of your Kubernetes setups. Bringing components into your infrastructure monitoring view;

  • Kubernetes;
    • Clusters
    • Nodes
    • Pods
    • Containers
    • Services

So this means you can add in your OCP environment for monitoring.

Configuring vRealize Operations to monitor your OpenShift Clusters

Grab the latest Container monitoring management pack to be installed in your vRealize Operations environment.

  1. Log in to the vRealize Operations Manager with administrator privileges.
  2. In the menu, select Administration and in the left pane select Solutions > Repository.
  3. On the Repository tab, click Add/Upgrade.
  4. Browse to locate the temporary folder and select the PAK file.
  5. Click Upload. The upload might take several minutes.
  6. Read and accept the EULA,and click Next.
  7. When the vRealize Operations Management Pack for Container Monitoring is installed, click Finish.

To link any Kubernetes to your environment for monitoring, you need to install the cAdvisor Daemon.  For OCP I used the cAdvisor YAML Definition on HostPort, secondly you need to create some credentials to authenticate to your cluster from your connection in vROPs.

Below is my token created from following the KB above, ensure you copy just the token itself, and when pasted there are no line breaks etc.

Through testing, I’ve found that token based authentication works well, and I followed this KB 75169, which gives you a sample YAML, creating this secret against Kube-System was fine in my lab environment, but for production usage, like all access and authorization creation you must understand the configuration and document.

If you want to review all your Kubernetes authentication options, see here.

Finally configure your Kubernetes Adapter in your vRealize Operations interface;

  1. From the main menu of vRealize Operations Manager, click Administration, and then in the left pane, click Solutions.
  2. From the Solutions list, select VMware vRealize Operations Management Pack for Container Monitoring.
  3. Click the Configure icon to edit an object.
  4. Enter the display name of the adapter.
  5. Enter the http URL of the Kubernetes master node in the Master URL text box.
  6. Select DaemonSet as the cAdvisor Service.
  7. Enter the port number of cAdvisor (Default is 31194)
  8. Enter the Credential details of the Master URL.
  9. Under advanced settings if the OCP cluster is running on vCenter Server which is monitored by vRealize Operations, you can view a link from the Kubernetes node to the vSphere Virtual Machine. To view the link, enter the IP address of the vCenter Server instance.

New Environment Views

You will find this management pack will create a lot of new objects in vRealize operations, one of them is a new K8S-World, which will hold the data of all the Kubernetes adapters configured in your environment, this grouping sits under the new Containers World.

In the below environment view, I am monitoring the memory utilisation of the three linked components.

  • Kubernetes Cluster Node > The Virtual Machine in vSphere > The VMware ESXi host the VM is running on.

Looking at the Dashboards

By default, you only get the Kubernetes overview dashboard, which is built upon some of the imported views.

However I also highly recommend you try out the Kubernetes Namespace dashboard created by my colleague Simon Conyard.

The Overview Dashboard is split into three main areas;

  • Kubernetes Cluster Summary showing cluster alerts figures
  • Performance of Cluster nodes
  • Performance of Pods & Containers

Select your Kubernetes cluster, which will relate to the name provided when configuring your Kubernetes adapter earlier in the configuration. As you can see here, this dashboard is Kubernetes platform agnostic, and I took this screenshot just as a VMware Tanzu Kubernetes Grid platform was added to vRealize Operations.

By selecting the active alerts, I will get a summary of which objects have triggered in the cluster, which I can select and view in the environment section.

Continuing down the dashboard, we then focus on the nodes, where I’ve selected my lowest health node.

We can see the node properties, pods relationship, key metrics and even pick our own metrics and properties to view for that node as well.

Below we can see my node health is reduced due to high memory consumption.

Finally, the last part of the dashboard looks at pods and containers. In the below we can see my API server pod availability has been all over the place. So I really need to look at this if I want to continue giving demos with this environment.

Finally I just want to highlight my favourite piece of the dashboard, which is the associated components, as below it shows you everything that is brought together as part of an app deployment in Kubernetes, but as a VI admin it can really help you visualise how everything is brought together, even if you’ve never used Kubernetes before.

Summary

Although this blog post points towards OpenShift Container Platform. There is nothing which is exclusive to OCP. Which for me shows the power of the vRealize Operations platform, the ability to be agnostic of the Kubernetes platform which it monitors. We could easily do a Find all + replace on this blog post and change OpenShift/OCP to Azure Kubernetes Service, and everything would remain the same. Alternatively you could read this blog post.

Regards

Kubernetes basics – kubeadm token create –print-join-command

Recently I’ve been using the fantastic resources to start learning Kubernetes (it’s the next big thing don’t you know). The course I’ve been following;

When running the command below, I lost connectivity to my master, which means I missed the print out of my join command to run on my woker nodes;

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

I was a little stumped how I get access to the print out again, and running the Kubeadm init, failed as initiation had already been done.

The fix for this was rather simple it seems by running the below;

kubeadmin token create --print-join-command

I was then given a new output to copy and paste to into my worker nodes. Don’t worry about the previous token that was issued, this will expire after 24 hours, which is the default timer.