Tag Archives: openshift

kasten vmware red hat openshift header

How to install and configure Kasten to protect container workloads on Red Hat OpenShift and VMware vSphere

In this blog post I’m going to cover deploying and configuring Kasten, the container based enterprise backup software now owned by Veeam Software.

This deployment will be inside my Red Hat OpenShift Environment which is running on top of VMware vSphere.

I’ll be protecting a cool gaming application that has data persistence using MongoDB.

Installing Kasten on Red Hat OpenShift

In this guide, I am going to use Helm, you can learn how to install it here.

Create a OpenShift project (Kubernetes namespace) called “kasten-io”

oc new-project kasten-io

oc new project kasten-io

Next we are going to use Helm to install the Kasten software into our OpenShift cluster.

helm install k10 kasten/k10 --namespace=kasten-io --set scc.create=true --set route.enabled=true --set route.path="/k10" --set auth.tokenAuth.enabled=true

Breaking down the command arguments;

  • –set scc.create=true
    • This creates the correct Security Contexts against the users created by the install. This is needed in OpenShift as the security context stance is higher OOTB than that of a vanilla Kubernetes install.
  • –set route.enabled=true
    • This creates a route in OpenShift using the default ingress, so that the Kasten dashboard is accessible externally. This will use the default cluster ID domain name.
  • –set route.path=”/k10″
    • This sets the route path for the redirection of the dashboard. Without this, your users will need to go to http://{FQDN}/ and append the path to the end (k10).
  • –set auth.tokenAuth.enabled=true

helm install k10 kasten kasten-io Continue reading How to install and configure Kasten to protect container workloads on Red Hat OpenShift and VMware vSphere

VMware Tanzu Mission Control Red Hat OpenShift header

Enabling Tanzu Mission Control Data Protection on Red Hat OpenShift

Just a quick blog on how to get the Data Protection feature of Tanzu Mission Control on Red Hat OpenShift. By default you will find that once the data protection feature is enabled, the pods for Restic component of Velero error.

  • Enable the Data Protection Feature on your Openshift cluster

TMC Cluster Overview enable data protection

  • You will see the UI change to show it’s enabling the feature.

TMC Enabling Data Protection 2

  • You will see the Velero namespace created in your cluster.

TMC oc get projects velero vmware system tmc

However the “Data Protection is being enabled” message in the TMC UI will continue to show without user intervention. If you show the pods for the Velero namespace you will see they error.

This is because OpenShift has a higher security context out of the box for containers than a vanilla Kubernetes environment.

TMC oc get pods restic error crashloopbackoff

The steps to resolve this are the same for a native install of the Project Velero opensource install to your cluster.

  • First we need to add the velero service account to the privileged SCC.
oc adm policy add-scc-to-user privileged -z velero -n velero

TMC oc adm policy add scc to user privileged velero

  • Secondly we need to patch the DaemonSet to allow the containers for Restic run in a privileged mode.
oc patch ds/restic \
--namespace velero \
--type json \
-p '[{"op":"add","path":"/spec/template/spec/containers/0/securityContext","value": { "privileged": true}}]'

After this, if we run the command to get all pods under the Velero namespace again, we’ll see that they are replaced with the new configuration and running.

TMC oc get pods restic running

Going back to our TMC Console, we’ll see the Data Protection feature is now enabled.

TMC data protection enabled

Regards

VMware vRealize Log Insight Cloud Red Hat OpenShift header

How to configure Red Hat OpenShift to forward logs to VMware vRealize Log Insight Cloud

In this blog post we will cover how to configure Red Hat OpenShift to forward logs from the ClusterLogging instance to an external 3rd party system, in this case, VMware vRealize Log Insight Cloud.

Architecture

The Openshift Cluster Logging will have to be configured for accessing the logs and forwarding to 3rd party logging tools. You can deploy the full suite;

  • Visualization: Kibana
  • Collection: FluentD
  • Log Store: Elasticsearch
  • Curation: Curator

However, to ship the logs to an external system, you will only need to configure the FluentD service.

To forward the logs from the internal trusted services, we will use the new Log Forwarding API, which is GA in OpenShift 4.6 and later (it was a tech preview in earlier releases, and the configuration YAMLs are slightly different, so read the relevant documentation version).

This setup will provide us the architecture below. We will deploy the trusted namespace “OpenShift-Logging” and use the Operator to provide a Log Forwarding API configuration which sends the logs to a 3rd party service.

For vRealize Log Insight Cloud, we will run a standalone FluentD instance inside of the cluster to forward to the cloud service.

Openshift cluster logging to vmware log insight architecture

The log types are one of the following:

  • application. Container logs generated by user applications running in the cluster, except infrastructure container applications.
  • infrastructure. Container logs from pods that run in the openshift*, kube*, or default projects and journal logs sourced from node file system.
  • audit. Logs generated by the node audit system (auditd) and the audit logs from the Kubernetes API server and the OpenShift API server.
Prerequisites
  • VMware vRealize Log Insight Cloud instance setup with Administrator access.
  • Red Hat OpenShift Cluster deployed
    • with outbound connectivity for containers
  • Download this Github Repository for the configuration files
git clone https://github.com/saintdle/openshift_vrealize_loginsight_cloud.git
Deploy the standalone FluentD instance to forward logs to vRealize Log Insight Cloud

As per the above diagram, we’ll create a namespace and deploy a FluentD service inside the cluster, this will handle the logs forwarded from the OpenShift Logging instance and send the to the Log Insight Cloud instance.

Creating a vRealize Log Insight Cloud API Key

First, we will create an API key for sending data to our cloud instance.

  1. Expand Configuration on the left-hand navigation pane
  2. Select “API Keys”
  3. Click the “New API Key” button

vRealize Log Insight Cloud API Key

Give your API key a suitable name and click Create.

vRealize Log Insight Cloud New API Key Continue reading How to configure Red Hat OpenShift to forward logs to VMware vRealize Log Insight Cloud

RH OCP Header

OpenShift – Cluster-Monitoring-Operator Pod Error – cannot verify user is non-root

The issue

After building a brand new OpenShift 4.6.9 cluster, I noticed one of the pods was not running correctly

oc get pods -n openshift-monitoring
.....
NAME READY   STATUS                       RESTARTS   AGE
cluster-monitoring-operator-f85f7bcb5-84jw5 1/2 CreateContainerConfigError 0 112m

Upon inspection of the pod;

oc describe pod cluster-monitoring-operator-XXX -n openshift-
monitoring

I could see the following error message;

Error: container has runAsNonRoot and image has non-numeric user
(nobody), cannot verify user is non-root

The Cause

There is a Red Hat article about this, but it is gated. The reason is cluster-monitoring-operator gets wrongly the non-root SCC assigned.

The Fix

Currently there is no permanent provided fix from Red Hat, but you can track this bug.

However the workaround is to simply delete the pod. This should recreate and load with the correct permissions.

Regards

OpenShift

How to specify your vSphere virtual machine resources when deploying Red Hat OpenShift

When deploying Red Hat OpenShift to VMware vSphere platform, there are two methods:

  • User Provisioned Infrastructure (UPI)
  • Installer Provisioned Infrastructure (IPI)

There are several great blogs covering both options and deployment methods.

In this blog, we are going to use the IPI method but customize the settings of our Virtual Machines that are deployed setting CPU and Memory that is different from the default settings.

Getting Started
Setting up your Jump host Machine

I’ll be using an Ubuntu Machine as my jumphost for the deployment.

Download the OpenShift-Install tool and OC command line tool. (I’ve used version 4.6.4 in my install)

Extract the files and copy to your /usr/bin/local directory

tar -zxvf openshift-client-linux.tar.gz
tar -zxvf openshift-install-linux.tar.gz

Have an available SSH key from your jump box, so that you can connect to your CoreOS VMs one they are deployed for troubleshooting purposes.

You need to download the vCenter trusted root certificates from your instance and import them to your Jump Host.

Curl -O https://{vCenter_FQDN}/certs/download.zip

Then the following to import (ubuntu uses the .crt files, hence importing the win folder);

unzip download.zip
cp certs/win/* /usr/local/share/ca-certificates
update-ca-certificates

You will need an account to connect to vCenter with the correct permissions for the OpenShift-Install to deploy the cluster. If you do not want to use an existing account and permissions, you can use this PowerCLI script to create the roles with the correct privileges based on the Red Hat documentation.

If you are installing into VMware Cloud on AWS, like myself, you will also need to allow connectivity from your segments as follows:

  • Compute gateway
    • OCP Cluster network to the internet
    • OCP Cluster network to your SDDC Management Network
  • Management gateway
    • OCP Cluster network to ESXi – HTTPs traffic

DNS Records – You will need the two following records to be available on your OCP Cluster network in the same IP address space that your nodes will be deployed to.

  • {clusterID}.{domain_name}
    • example: ocp46.veducate.local
  • *.apps.{clusterID}.{domain_name}
    • example: *.apps.ocp46.veducate.local

If your DNS is a Windows server, you can use this script here. Continue reading How to specify your vSphere virtual machine resources when deploying Red Hat OpenShift