Category Archives: Kubernetes

VMware Tanzu Mission Control Red Hat OpenShift header

Enabling Tanzu Mission Control Data Protection on Red Hat OpenShift

VMware, Kubernetes, , , ,

Just a quick blog on how to get the Data Protection feature of Tanzu Mission Control on Red Hat OpenShift. By default you will find that once the data protection feature is enabled, the pods for Restic component of Velero error.

  • Enable the Data Protection Feature on your Openshift cluster

TMC Cluster Overview enable data protection

  • You will see the UI change to show it’s enabling the feature.

TMC Enabling Data Protection 2

  • You will see the Velero namespace created in your cluster.

TMC oc get projects velero vmware system tmc

However the “Data Protection is being enabled” message in the TMC UI will continue to show without user intervention. If you show the pods for the Velero namespace you will see they error.

This is because OpenShift has a higher security context out of the box for containers than a vanilla Kubernetes environment.

TMC oc get pods restic error crashloopbackoff

The steps to resolve this are the same for a native install of the Project Velero opensource install to your cluster.

  • First we need to add the velero service account to the privileged SCC.
oc adm policy add-scc-to-user privileged -z velero -n velero

TMC oc adm policy add scc to user privileged velero

  • Secondly we need to patch the DaemonSet to allow the containers for Restic run in a privileged mode.
oc patch ds/restic \
--namespace velero \
--type json \
-p '[{"op":"add","path":"/spec/template/spec/containers/0/securityContext","value": { "privileged": true}}]'

After this, if we run the command to get all pods under the Velero namespace again, we’ll see that they are replaced with the new configuration and running.

TMC oc get pods restic running

Going back to our TMC Console, we’ll see the Data Protection feature is now enabled.

TMC data protection enabled

Regards

Dean Lewis
VMware vRealize Log Insight Cloud Red Hat OpenShift header

How to configure Red Hat OpenShift to forward logs to VMware vRealize Log Insight Cloud

Kubernetes, VMware, , , , ,

In this blog post we will cover how to configure Red Hat OpenShift to forward logs from the ClusterLogging instance to an external 3rd party system, in this case, VMware vRealize Log Insight Cloud.

Architecture

The Openshift Cluster Logging will have to be configured for accessing the logs and forwarding to 3rd party logging tools. You can deploy the full suite;

  • Visualization: Kibana
  • Collection: FluentD
  • Log Store: Elasticsearch
  • Curation: Curator

However, to ship the logs to an external system, you will only need to configure the FluentD service.

To forward the logs from the internal trusted services, we will use the new Log Forwarding API, which is GA in OpenShift 4.6 and later (it was a tech preview in earlier releases, and the configuration YAMLs are slightly different, so read the relevant documentation version).

This setup will provide us the architecture below. We will deploy the trusted namespace “OpenShift-Logging” and use the Operator to provide a Log Forwarding API configuration which sends the logs to a 3rd party service.

For vRealize Log Insight Cloud, we will run a standalone FluentD instance inside of the cluster to forward to the cloud service.

Openshift cluster logging to vmware log insight architecture

The log types are one of the following:

  • application. Container logs generated by user applications running in the cluster, except infrastructure container applications.
  • infrastructure. Container logs from pods that run in the openshift*, kube*, or default projects and journal logs sourced from node file system.
  • audit. Logs generated by the node audit system (auditd) and the audit logs from the Kubernetes API server and the OpenShift API server.
Prerequisites
  • VMware vRealize Log Insight Cloud instance setup with Administrator access.
  • Red Hat OpenShift Cluster deployed
    • with outbound connectivity for containers
  • Download this Github Repository for the configuration files
git clone https://github.com/saintdle/openshift_vrealize_loginsight_cloud.git
Deploy the standalone FluentD instance to forward logs to vRealize Log Insight Cloud

As per the above diagram, we’ll create a namespace and deploy a FluentD service inside the cluster, this will handle the logs forwarded from the OpenShift Logging instance and send the to the Log Insight Cloud instance.

Creating a vRealize Log Insight Cloud API Key

First, we will create an API key for sending data to our cloud instance.

  1. Expand Configuration on the left-hand navigation pane
  2. Select “API Keys”
  3. Click the “New API Key” button

vRealize Log Insight Cloud API Key

Give your API key a suitable name and click Create.

vRealize Log Insight Cloud New API Key Continue reading How to configure Red Hat OpenShift to forward logs to VMware vRealize Log Insight Cloud →

RH OCP Header

OpenShift – Cluster-Monitoring-Operator Pod Error – cannot verify user is non-root

Kubernetes, , ,

The issue

After building a brand new OpenShift 4.6.9 cluster, I noticed one of the pods was not running correctly

oc get pods -n openshift-monitoring
.....
NAME READY   STATUS                       RESTARTS   AGE
cluster-monitoring-operator-f85f7bcb5-84jw5 1/2 CreateContainerConfigError 0 112m

Upon inspection of the pod;

oc describe pod cluster-monitoring-operator-XXX -n openshift-
monitoring

I could see the following error message;

Error: container has runAsNonRoot and image has non-numeric user
(nobody), cannot verify user is non-root

The Cause

There is a Red Hat article about this, but it is gated. The reason is cluster-monitoring-operator gets wrongly the non-root SCC assigned.

The Fix

Currently there is no permanent provided fix from Red Hat, but you can track this bug.

However the workaround is to simply delete the pod. This should recreate and load with the correct permissions.

Regards

Dean Lewis
OpenShift

How to specify your vSphere virtual machine resources when deploying Red Hat OpenShift

Kubernetes, VMware, ,

When deploying Red Hat OpenShift to VMware vSphere platform, there are two methods:

  • User Provisioned Infrastructure (UPI)
  • Installer Provisioned Infrastructure (IPI)

There are several great blogs covering both options and deployment methods.

In this blog, we are going to use the IPI method but customize the settings of our Virtual Machines that are deployed setting CPU and Memory that is different from the default settings.

Getting Started
Setting up your Jump host Machine

I’ll be using an Ubuntu Machine as my jumphost for the deployment.

Download the OpenShift-Install tool and OC command line tool. (I’ve used version 4.6.4 in my install)

Extract the files and copy to your /usr/bin/local directory

tar -zxvf openshift-client-linux.tar.gz
tar -zxvf openshift-install-linux.tar.gz

Have an available SSH key from your jump box, so that you can connect to your CoreOS VMs one they are deployed for troubleshooting purposes.

You need to download the vCenter trusted root certificates from your instance and import them to your Jump Host.

curl -O https://{vCenter_FQDN}/certs/download.zip

Then the following to import (ubuntu uses the .crt files, hence importing the win folder);

unzip download.zip
cp certs/win/* /usr/local/share/ca-certificates
update-ca-certificates

You will need an account to connect to vCenter with the correct permissions for the OpenShift-Install to deploy the cluster. If you do not want to use an existing account and permissions, you can use this PowerCLI script to create the roles with the correct privileges based on the Red Hat documentation.

If you are installing into VMware Cloud on AWS, like myself, you will also need to allow connectivity from your segments as follows:

  • Compute gateway
    • OCP Cluster network to the internet
    • OCP Cluster network to your SDDC Management Network
  • Management gateway
    • OCP Cluster network to ESXi – HTTPs traffic

DNS Records – You will need the two following records to be available on your OCP Cluster network in the same IP address space that your nodes will be deployed to.

  • {clusterID}.{domain_name}
    • example: ocp46.veducate.local
  • *.apps.{clusterID}.{domain_name}
    • example: *.apps.ocp46.veducate.local

If your DNS is a Windows server, you can use this script here. Continue reading How to specify your vSphere virtual machine resources when deploying Red Hat OpenShift →

OpenShift

Red Hat OpenShift on VMware vSphere – How to Scale and Edit your cluster deployments

Kubernetes, VMware, , , ,

Working with Red Hat OpenShift on vSphere, I’m really starting to understand the main infrastructure components and how everything fits together.

Next up was understanding how to control the cluster size after initial deployment. So, with Red Hat OpenShift, there are some basic concepts we need to understand first, before we jump into the technical how-to’s below in this blog.

In this blog I will cover the following;

- Understanding the concepts behind controlling Machines in OpenShift
- Editing your MachineSet to control your Virtual Machine Resources
- Editing your MachineSet to scale your cluster manually
- Deleting a node
- Configuring ClusterAutoscaler to automatically scale your environment

Machine API

The Machine API is a combination of primary resources that are based on the upstream Cluster API project and custom OpenShift Container Platform resources.

The Machine API performs all node host provisioning management actions as a post cluster installation method, providing you dynamic provisioning on top of your VMware vSphere platform (and other public/private cloud platforms).

The two primary resources are:

Machines
An object that describes the host for a Node. A machine has a providerSpec, which describes the types of compute nodes that are offered for different cloud platforms. For example, a machine type for a worker node on Amazon Web Services (AWS) might define a specific machine type and required metadata.
MachineSets
Groups of machines. MachineSets are to machines as ReplicaSets are to Pods. If you need more machines or must scale them down, you change the replicas field on the MachineSet to meet your compute need.

These custom resources add capabilities to your OpenShift cluster:

MachineAutoscaler
This resource automatically scales machines in a cloud. You can set the minimum and maximum scaling boundaries for nodes in a specified MachineSet, and the MachineAutoscaler maintains that range of nodes. The MachineAutoscaler object takes effect after a ClusterAutoscaler object exists. Both ClusterAutoscaler and MachineAutoscaler resources are made available by the ClusterAutoscalerOperator.
ClusterAutoscaler
This resource is based on the upstream ClusterAutoscaler project. In the OpenShift Container Platform implementation, this is integrated with the Machine API by extending the MachineSet API. You can set cluster-wide scaling limits for resources such as cores, nodes, memory, GPU, etc. You can configure priorities so that the cluster prioritizes pods so that new nodes are not brought online for less important pods. You can also set the ScalingPolicy, so that for example, you can scale up nodes but not scale down the node count.

MachineHealthCheck

This resource detects when a machine is unhealthy, deletes it, and, on supported platforms, creates a new machine. You can read more here about this technology preview feature in OCP 4.6.

Editing your MachineSet to control your Virtual Machine Resources

To view the current MachineSet objects available run; Continue reading Red Hat OpenShift on VMware vSphere – How to Scale and Edit your cluster deployments →