Category Archives: Kubernetes

Kubernetes Troubleshooting – Kubelet Unable to attach or mount volumes – timed out waiting for the condition

September 30, 2021KubernetesApplication, Deployment, Fix, Kubelet, Kubernetes, Retry, Troubleshooting, VolumeAttachments, VolumeMountsDean

The Issue

When I updated my Kasten application in my Kubernetes cluster, I found that one of the pods was stuck in “init” status.

dean@dean [ ~ ] (⎈ |tkg-wld-01-admin@tkg-wld-01:default) # k get pods -n kasten-io -w
NAME READY STATUS RESTARTS AGE
aggregatedapis-svc-78564d4697-wl9wg 1/1 Running 0 3m9s
auth-svc-7977b9684b-zph27 1/1 Running 0 3m11s
catalog-svc-7ff7779b75-kmvsr 0/2 Init:0/2 0 2m43s

Running a describe on that pod pointed to the fact the volume could not be attached.

Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m58s default-scheduler Successfully assigned kasten-io/catalog-svc-7ff7779b75-kmvsr to tkg-wld-01-md-0-54598b8d99-rpqjf
Warning FailedMount 55s kubelet Unable to attach or mount volumes: unmounted volumes=[catalog-persistent-storage], unattached volumes=[k10-k10-token-lbqpw catalog-persistent-storage]: timed out waiting for the condition

The Cause

Some where along the line I found some stale volumeattachments linked to Kubernetes node that no longer exist in my cluster. This looks to be causing some confusion in the cluster who should be attaching the volume

The image below shows:

Find the Persistent Volume name linked to the associated claim for the failure in the pod events
Map this to the available VolumeAttachments
Reference VolumeAttachments for each node to available nodes in the cluster
- I’ve highlighted the missing node in the red box

The Fix

The fix is to remove the stale VolumeAttachment.

kubectl delete volumeattachment [volumeattachment_name]

After this your pod should eventually pick up and retry, or you could remove the pod and let Kubernetes replace it for you (so long as it’s part of a deployment or other configuration managing your application).

Regards

Follow @Saintdle

Dean Lewis

vSphere with Tanzu – Creating cluster fails with “storage class is not valid”

September 15, 2021VMware, KubernetesStorageClass, vSphere with TanzuDean

The Issue

When you have attached a vSphere Storage Policy to your vSphere Namespace, and tried to create a cluster using the Storage Policy Name, you find it will fail with an error such as:

Error from server (storage class is not valid for control plane VM: StorageClass 'Tanzu Storage Policy' is not assigned for namespace 'deanl', 

storage class is not valid for worker VMs: StorageClass 'Tanzu Storage Policy' is not assigned for namespace 'deanl', 

storage class Tanzu Storage Policy under spec.settings.storage.defaultClass is not valid: StorageClass 'Tanzu Storage Policy' is not assigned for namespace 'deanl'): error when creating "cluster.yaml": 

admission webhook "default.validating.tanzukubernetescluster.run.tanzu.vmware.com" denied the request: storage class is not valid for control plane VM: StorageClass 'Tanzu Storage Policy' is not assigned for namespace 'deanl', 

storage class is not valid for worker VMs: StorageClass 'Tanzu Storage Policy' is not assigned for namespace 'deanl', 

storage class Tanzu Storage Policy under spec.settings.storage.defaultClass is not valid: StorageClass 'Tanzu Storage Policy' is not assigned for namespace 'deanl'

When you look at the vSphere namespace, the Storage Policy is attached.

And example of the erroneous Tanzu Cluster definition YAML:

apiVersion: run.tanzu.vmware.com/v1alpha1
kind: TanzuKubernetesCluster
metadata:
  name: deanl-cluster
  namespace: deanl
spec:
  distribution:
    version: v1.18.5
  topology:
    controlPlane:
      class: best-effort-small
      count: 1
      storageClass: "Tanzu Storage Policy"
    workers:
      class: best-effort-small
      count: 1
      storageClass: "Tanzu Storage Policy"
  settings:
    network:
      cni:
        name: calico
    storage:
      defaultClass: "Tanzu Storage Policy"

The Cause

Continue reading vSphere with Tanzu – Creating cluster fails with “storage class is not valid” →

MongoDB Container data loss issue – A Journey

August 30, 2021KubernetesBitnami, data loss, Kubernetes, MongoDB, Persistent, Volume, WriteDean

Over the past month or so I noticed an issue with my Pac-Man Kubernetes application, which I use for demonstrations as a basic app front-end that writes to a database back end, running in Kubernetes.

When I restored my instances using Kasten, my Pac-Man high scores were missing.
This issue happened when I made some changes to my deployment files to configure authentication to the MongoDB using environment variables in my deployment file.

This blog post is a detail walk-through of the steps I took to troubleshoot the issue, and then rectify it!

Summary if you don’t want to read the post

If you are not looking to read through this blog post, here is the summary:

I changed MongoDB images, I needed to configure a new mount point location to match the MongoDB configuration
New MongoDB image is non-root, so had to use an Init container to configure the permissions on the PV first

Overview of the application

The application is made up of the following components:

Namespace
Deployment
- MongoDB Pod
  - DB Authentication configured
  - Attached to a PVC
- Pac-Man Pod
  - Nodejs web front end that connects back to the MongoDB Pod by looking for the Pod DNS address internally.
RBAC Configuration for Pod Security and Service Account
Secret which holds the data for the MongoDB Usernames and Passwords to be configured
Service
- Type: LoadBalancer
  - Used to balance traffic to the Pac-Man Pods

Confirming the behaviour

The behaviour I was seeing when my application was deployed:

Pac-Man web page – I could save a high score, and it would show in the high scores list
- This showed the connectivity to the database was working, as the app would hang if it could not write to the database.
I would protect my application using Kasten. When I deleted the namespace, and restored everything, my application would be running, but there was no high scores to show.
This was apparent from deploying the branch version v0.5.0 and v0.5.1 from my GitHub.
Deploying the branch v0.2.0 would not product the same behaviour
- This configuration did not have any database authentication setup, meaning MongoDB was open to the world if they could connect without a UN/Password.

Testing the Behaviour

Continue reading MongoDB Container data loss issue – A Journey →

Configuring Kasten Multi-Cluster Manager across Tanzu and OpenShift

August 6, 2021Kubernetes, VMwareConfigure, Kasten, Multi-ClusterDean

In this blog post I’m going to cover setting up the Multi-cluster support for Kasten when you’ve installed the software to multiple Kubernetes clusters.

One K10 cluster you have deployed will become the primary node. You will use this node and dashboard interface to access the cluster UI.

The primary cluster defines policies and other configuration centrally. Centrally defined policies and configuration can then be distributed to designated clusters to be enacted.

Additional clusters are then added in and are called Secondaries.

The secondary clusters receive policies and other configuration from the primary cluster. Once policies are distributed to a secondary, the local K10 installation enacts the policy. This ensures that the policy will continue to be enforced, even if disconnected from the primary.

Pre-Requisites:

Authentication
- Token Authentication must be used
Network
- Secondary K10’s ingress must be accessible by the primary
- Secondary API Server must be accessible by the primary
Run the tool on a bastion host that has connectivity using kubectl to all of the clusters you want to bring together.

Download the K10MultiCluster tool

Download the appropriate version of the tool to match your K10 versions
- https://github.com/kastenhq/external-tools/releases

Set the tool as executable
Move the tool to your /usr/local/bin/ folder

curl -LJO https://github.com/kastenhq/external-tools/releases/download/4.0.9/k10multicluster_4.0.9_linux_amd64


chmod +x k10multicluster_4.0.9_linux_amd64

sudo mv k10multicluster_4.0.9_linux_amd64 /usr/local/bin/k10multicluster

Next let’s list out our available clusters we can connect to from our node

kubectl config get-contexts

Setup Primary Cluster

Now we are ready to setup our primary cluster by running the following command: Continue reading Configuring Kasten Multi-Cluster Manager across Tanzu and OpenShift →

Authenticate Docker to Harbor Image Registry with a Robot Account

July 13, 2021VMware, Kubernetesdocker, Harbor, Image Registry, LoginDean

Just a quick blog post on how to authenticate Docker to a Harbor Image Registry, using a Robot Account, which is good for programmatically access to push/pull images from your registry.

Harbor introduced the capability for administrators to create system robot accounts you can use you run automated actions in your Harbor instances. System robot accounts allow you to use a robot account to perform maintenance or repeated task across all or a subset of projects in your Harbor instance.

In Harbor, to create your robot account, within your project:

Click the Robot Accounts tab
Click New Robot Account
Enter your details as necessary and click save
Select export to file and save the JSON file.

This file will be outputted with a “$” in the name. You can either change the name of the file, or escape the character in the terminal, which I what I’ll be doing below.

Within your terminal run the following commands:

username=$(cat file.json | jq -r .name)
password=$(cat file.json | jq -r .token) <<< See below update
echo "$password" | docker login https://URL --username "$username" --password-stdin

Example
username=$(cat "robot\$veducate.json" | jq -r .name)
password=$(cat robot\$veducate.json | jq -r .token)
echo "$password" | docker login https://harbor-repo.veducate.com --username "$username" --password-stdin

Update October 2021
Since Harbor 2.2 minor release, and I found that within the JSON the key name has changed to secret, so this is the updated example

username=$(cat robot-veducate.json | jq -r .name)
password=$(cat robot-veducate.json | jq -r .secret)
echo "$password" | docker login https://harbor-repo.veducate.com --username "$username" --password-stdin

Now you can pull/push images to your repository using this authentication details.

Regards

Follow @Saintdle

Dean Lewis

vEducate.co.uk

Fixing issues and blogging

Category Archives: Kubernetes

Kubernetes Troubleshooting – Kubelet Unable to attach or mount volumes – timed out waiting for the condition

The Issue

The Cause

The Fix

vSphere with Tanzu – Creating cluster fails with “storage class is not valid”

The Issue

The Cause

MongoDB Container data loss issue – A Journey

Summary if you don’t want to read the post

Overview of the application

Confirming the behaviour

Testing the Behaviour

Configuring Kasten Multi-Cluster Manager across Tanzu and OpenShift

Download the K10MultiCluster tool

Setup Primary Cluster