Category Archives: Kubernetes

kasten by veeam header

How to update Kasten to the latest version

This is probably one of the simplest blog posts I’ll publish.

To see if there is an available update for your Kasten install.

  • In the dashboard click > Settings
  • Click on Support

See if there is a notification under the Cluster Information heading.

kasten dashboard support cluster information

Clicking the “upgrade to version x.x.x” button will take you to this Kasten Docs page.

Or you can follow the same instructions with real life screenshots below.

To upgrade run the following using helm:

helm repo update && \
    helm get values k10 --output yaml --namespace=kasten-io > k10_val.yaml && \
    helm upgrade k10 kasten/k10 --namespace=kasten-io -f k10_val.yaml

helm upgrade k10 kasten k10

You will see messages similar to the below.

helm upgrade k10 kasten k10 - upgrade in progress - upgrade complete

If I now look at my pods in my namespace “Kasten-IO” I can see they are being recreated as the deployment artifacts will have been updated with the new configuration including container images.

helm upgrade k10 kasten - oc get pods -n kasten-io

And finally looking back at my Kasten Dashboard for the cluster information, I can see I am now at the latest version.

Regards

 

VMware Tanzu Header

VMware Tanzu Mission Control – Using the Data Protection feature for backups and restores

In this blog post we will cover the following topics

- Data Protection Overview
- Create a AWS Data Protection Credential
- Enable Data Protection on a Cluster
- Running a backup manually or via an automatic schedule
- Restoring your data

The follow up blog posts are;

- Tanzu Mission Control
- - Getting Started with TMC
- - - What is Tanzu Mission Control?
- - - Creating a Cluster Group
- - - Attaching a cluster to Tanzu Mission Control
- - - Viewing your Cluster Objects
- - -Where can I demo/test/trial this myself?
- - Cluster Inspections
- - - What Inspections are available 
- - - Performing Inspections 
- - - Viewing Inspections
- - Workspaces and Policies
- - - Creating a workspace 
- - - Creating a managed Namespace 
- - - Policy Driven Cluster Management 
- - - Creating Policies
TMC Data Protection Overview

Tanzu Mission Control implements data protection through the inclusion of the Project Velero,  this tool is not enabled by default. This blog post will take you through the setup.

Data is stored externally to a AWS location, with volume backups remaining as part of the cluster where you’ve connected TMC.

Currently there is no ability to backup and restore data between Kubernetes clusters managed by TMC.

Create a AWS Data Protection Credential

First we need to create a AWS data protection credential, so that TMC can configure Velero within your cluster to save the data externally to AWS.

If you are looking for supported options for protecting data to other locations, I recommend you either look at deploying Project Velero manually outside of TMC (losing access to the data protection features in the UI) or look at another enterprise service such as Kasten.io.

  • On the Administration screen, click Accounts, and Create Account Credential.
  • Select > AWS data protection credential

  • Set your account name for easy identification and click to generate template and save this file to your machine.

The next steps will require configuration in the AWS console to create resources using CloudFormation so that Project Velero can export data to AWS. Here is the official VMware documentation on this configuration.

  • In the AWS Console, go to the CloudFormation service

  • Click to create a new stack
  1. Click “Template is ready” as we will provide our template file from earlier.
  2. Click to upload a template file
  3. Select the file from your machine
  4. Click next

  • Provide a stack name and click next

  • Ignore all the items on this page and click next
  • Review your configuration and click finish.

  • Once you’ve reviewed and clicked create/finish. You will be taken into the Stack itself.
  • You can click the Events tab and the refresh button to see the progress.

Continue reading VMware Tanzu Mission Control – Using the Data Protection feature for backups and restores

Enabling Tanzu Mission Control Data Protection on Red Hat OpenShift

Just a quick blog on how to get the Data Protection feature of Tanzu Mission Control on Red Hat OpenShift. By default you will find that once the data protection feature is enabled, the pods for Restic component of Velero error.

  • Enable the Data Protection Feature on your Openshift cluster

  • You will see the UI change to show it’s enabling the feature.

  • You will see the Velero namespace created in your cluster.

However the “Data Protection is being enabled” message in the TMC UI will continue to show without user intervention. If you show the pods for the Velero namespace you will see they error.

This is because OpenShift has a higher security context out of the box for containers than a vanilla Kubernetes environment.

The steps to resolve this are the same for a native install of the Project Velero opensource install to your cluster.

  • First we need to add the velero service account to the privileged SCC.
oc adm policy add-scc-to-user privileged -z velero -n velero

  • Secondly we need to patch the DaemonSet to allow the containers for Restic run in a privileged mode.
oc patch ds/restic \
--namespace velero \
--type json \
-p '[{"op":"add","path":"/spec/template/spec/containers/0/securityContext","value": { "privileged": true}}]'

After this, if we run the command to get all pods under the Velero namespace again, we’ll see that they are replaced with the new configuration and running.

Going back to our TMC Console, we’ll see the Data Protection feature is now enabled.

Regards

How to configure Red Hat OpenShift to forward logs to VMware vRealize Log Insight Cloud

In this blog post we will cover how to configure Red Hat OpenShift to forward logs from the ClusterLogging instance to an external 3rd party system, in this case, VMware vRealize Log Insight Cloud.

Architecture

The Openshift Cluster Logging will have to be configured for accessing the logs and forwarding to 3rd party logging tools. You can deploy the full suite;

  • Visualization: Kibana
  • Collection: FluentD
  • Log Store: Elasticsearch
  • Curation: Curator

However, to ship the logs to an external system, you will only need to configure the FluentD service.

To forward the logs from the internal trusted services, we will use the new Log Forwarding API, which is GA in OpenShift 4.6 and later (it was a tech preview in earlier releases, and the configuration YAMLs are slightly different, so read the relevant documentation version).

This setup will provide us the architecture below. We will deploy the trusted namespace “OpenShift-Logging” and use the Operator to provide a Log Forwarding API configuration which sends the logs to a 3rd party service.

For vRealize Log Insight Cloud, we will run a standalone FluentD instance inside of the cluster to forward to the cloud service.

Openshift cluster logging to vmware log insight architecture

The log types are one of the following:

  • application. Container logs generated by user applications running in the cluster, except infrastructure container applications.
  • infrastructure. Container logs from pods that run in the openshift*, kube*, or default projects and journal logs sourced from node file system.
  • audit. Logs generated by the node audit system (auditd) and the audit logs from the Kubernetes API server and the OpenShift API server.
Prerequisites
  • VMware vRealize Log Insight Cloud instance setup with Administrator access.
  • Red Hat OpenShift Cluster deployed
    • with outbound connectivity for containers
  • Download this Github Repository for the configuration files
git clone https://github.com/saintdle/openshift_vrealize_loginsight_cloud.git
Deploy the standalone FluentD instance to forward logs to vRealize Log Insight Cloud

As per the above diagram, we’ll create a namespace and deploy a FluentD service inside the cluster, this will handle the logs forwarded from the OpenShift Logging instance and send the to the Log Insight Cloud instance.

Creating a vRealize Log Insight Cloud API Key

First, we will create an API key for sending data to our cloud instance.

  1. Expand Configuration on the left-hand navigation pane
  2. Select “API Keys”
  3. Click the “New API Key” button

Give your API key a suitable name and click Create.

Continue reading How to configure Red Hat OpenShift to forward logs to VMware vRealize Log Insight Cloud

OpenShift – Cluster-Monitoring-Operator Pod Error – cannot verify user is non-root

The issue

After building a brand new OpenShift 4.6.9 cluster, I noticed one of the pods was not running correctly

oc get pods -n openshift-monitoring
.....
NAME READY   STATUS                       RESTARTS   AGE
cluster-monitoring-operator-f85f7bcb5-84jw5 1/2 CreateContainerConfigError 0 112m

Upon inspection of the pod;

oc describe pod cluster-monitoring-operator-XXX -n openshift-
monitoring

I could see the following error message;

Error: container has runAsNonRoot and image has non-numeric user
(nobody), cannot verify user is non-root

The Cause

There is a Red Hat article about this, but it is gated. The reason is cluster-monitoring-operator gets wrongly the non-root SCC assigned.

The Fix

Currently there is no permanent provided fix from Red Hat, but you can track this bug.

However the workaround is to simply delete the pod. This should recreate and load with the correct permissions.

Regards