Category Archives: Kubernetes

Kubernetes

Kubernetes – Kubelet Unable to attach or mount volumes – timed out waiting for the condition

The Issue

When I updated my Kasten application in my Kubernetes cluster, I found that one of the pods was stuck in “init” status.

[email protected] [ ~ ] (⎈ |[email protected]:default) # k get pods -n kasten-io -w
NAME READY STATUS RESTARTS AGE
aggregatedapis-svc-78564d4697-wl9wg 1/1 Running 0 3m9s
auth-svc-7977b9684b-zph27 1/1 Running 0 3m11s
catalog-svc-7ff7779b75-kmvsr 0/2 Init:0/2 0 2m43s

kubectl get pods - status init

Running a describe on that pod pointed to the fact the volume could not be attached.

Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m58s default-scheduler Successfully assigned kasten-io/catalog-svc-7ff7779b75-kmvsr to tkg-wld-01-md-0-54598b8d99-rpqjf
Warning FailedMount 55s kubelet Unable to attach or mount volumes: unmounted volumes=[catalog-persistent-storage], unattached volumes=[k10-k10-token-lbqpw catalog-persistent-storage]: timed out waiting for the condition
kubelet Unable to attach or mount volumes- unmounted volumes=[catalog-persistent-storage], unattached volumes=[k10-k10-token-lbqpw catalog-persistent-storage]- timed out waiting for the condition
The Cause

Some where along the line I found some stale volumeattachments linked to Kubernetes node that no longer exist in my cluster. This looks to be causing some confusion in the cluster who should be attaching the volume

The image below shows:

  • Find the Persistent Volume name linked to the associated claim for the failure in the pod events
  • Map this to the available VolumeAttachments
  • Reference VolumeAttachments for each node to available nodes in the cluster
    • I’ve highlighted the missing node in the red box

kubectl get pv - get volumeattachment - get nodes

The Fix

The fix is to remove the stale VolumeAttachment.

kubectl delete volumeattachment [volumeattachment_name]

kubectl delete volumeattachment

After this your pod should eventually pick up and retry, or you could remove the pod and let Kubernetes replace it for you (so long as it’s part of a deployment or other configuration managing your application).

Regards

Dean Lewis

VMware Tanzu Header

vSphere with Tanzu – Creating cluster fails with “storage class is not valid”

The Issue

When you have attached a vSphere Storage Policy to your vSphere Namespace, and tried to create a cluster using the Storage Policy Name, you find it will fail with an error such as:

Error from server (storage class is not valid for control plane VM: StorageClass 'Tanzu Storage Policy' is not assigned for namespace 'deanl', 

storage class is not valid for worker VMs: StorageClass 'Tanzu Storage Policy' is not assigned for namespace 'deanl', 

storage class Tanzu Storage Policy under spec.settings.storage.defaultClass is not valid: StorageClass 'Tanzu Storage Policy' is not assigned for namespace 'deanl'): error when creating "cluster.yaml": 

admission webhook "default.validating.tanzukubernetescluster.run.tanzu.vmware.com" denied the request: storage class is not valid for control plane VM: StorageClass 'Tanzu Storage Policy' is not assigned for namespace 'deanl', 

storage class is not valid for worker VMs: StorageClass 'Tanzu Storage Policy' is not assigned for namespace 'deanl', 

storage class Tanzu Storage Policy under spec.settings.storage.defaultClass is not valid: StorageClass 'Tanzu Storage Policy' is not assigned for namespace 'deanl'

When you look at the vSphere namespace, the Storage Policy is attached.

vSphere Namespace Storage Policy

And example of the erroneous Tanzu Cluster definition YAML:

apiVersion: run.tanzu.vmware.com/v1alpha1
kind: TanzuKubernetesCluster
metadata:
  name: deanl-cluster
  namespace: deanl
spec:
  distribution:
    version: v1.18.5
  topology:
    controlPlane:
      class: best-effort-small
      count: 1
      storageClass: "Tanzu Storage Policy"
    workers:
      class: best-effort-small
      count: 1
      storageClass: "Tanzu Storage Policy"
  settings:
    network:
      cni:
        name: calico
    storage:
      defaultClass: "Tanzu Storage Policy"
The Cause

Continue reading vSphere with Tanzu – Creating cluster fails with “storage class is not valid”

MongoDB + Kubernetes Header

MongoDB Container data loss issue – A Journey

Over the past month or so I noticed an issue with my Pac-Man Kubernetes application, which I use for demonstrations as a basic app front-end that writes to a database back end, running in Kubernetes.

  • When I restored my instances using Kasten, my Pac-Man high scores were missing.
  • This issue happened when I made some changes to my deployment files to configure authentication to the MongoDB using environment variables in my deployment file.

This blog post is a detail walk-through of the steps I took to troubleshoot the issue, and then rectify it!

Summary if you don’t want to read the post

If you are not looking to read through this blog post, here is the summary:

  • I changed MongoDB images, I needed to configure a new mount point location to match the MongoDB configuration
  • New MongoDB image is non-root, so had to use an Init container to configure the permissions on the PV first
Overview of the application

The application is made up of the following components:

  • Namespace
  • Deployment
    • MongoDB Pod
      • DB Authentication configured
      • Attached to a PVC
    • Pac-Man Pod
      • Nodejs web front end that connects back to the MongoDB Pod by looking for the Pod DNS address internally.
  • RBAC Configuration for Pod Security and Service Account
  • Secret which holds the data for the MongoDB Usernames and Passwords to be configured
  • Service
    • Type: LoadBalancer
      • Used to balance traffic to the Pac-Man Pods

Pac-Man Kubernetes Diagram

Confirming the behaviour

The behaviour I was seeing when my application was deployed:

  • Pac-Man web page – I could save a high score, and it would show in the high scores list
    • This showed the connectivity to the database was working, as the app would hang if it could not write to the database.
  • I would protect my application using Kasten. When I deleted the namespace, and restored everything, my application would be running, but there was no high scores to show.
  • This was apparent from deploying the branch version v0.5.0 and v0.5.1 from my GitHub.
  • Deploying the branch v0.2.0 would not product the same behaviour
    • This configuration did not have any database authentication setup, meaning MongoDB was open to the world if they could connect without a UN/Password.
Testing the Behaviour

Continue reading MongoDB Container data loss issue – A Journey

Kasten K10 Header

Configuring Kasten Multi-Cluster Manager across Tanzu and OpenShift

In this blog post I’m going to cover setting up the Multi-cluster support for Kasten when you’ve installed the software to multiple Kubernetes clusters.

One K10 cluster you have deployed will become the primary node. You will use this node and dashboard interface to access the cluster UI.

  • The primary cluster defines policies and other configuration centrally. Centrally defined policies and configuration can then be distributed to designated clusters to be enacted.

Additional clusters are then added in and are called Secondaries.

  • The secondary clusters receive policies and other configuration from the primary cluster. Once policies are distributed to a secondary, the local K10 installation enacts the policy. This ensures that the policy will continue to be enforced, even if disconnected from the primary.

Pre-Requisites:

  • Authentication
    • Token Authentication must be used
  • Network
    • Secondary K10’s ingress must be accessible by the primary
    • Secondary API Server must be accessible by the primary
  • Run the tool on a bastion host that has connectivity using kubectl to all of the clusters you want to bring together.
Download the K10MultiCluster tool
  • Set the tool as executable
  • Move the tool to your /usr/local/bin/ folder
curl -LJO https://github.com/kastenhq/external-tools/releases/download/4.0.9/k10multicluster_4.0.9_linux_amd64


chmod +x k10multicluster_4.0.9_linux_amd64

sudo mv k10multicluster_4.0.9_linux_amd64 /usr/local/bin/k10multicluster

Download K10Multicluster

Next let’s list out our available clusters we can connect to from our node

kubectl config get-contexts

kubectl config get-contexts

Setup Primary Cluster

Now we are ready to setup our primary cluster by running the following command: Continue reading Configuring Kasten Multi-Cluster Manager across Tanzu and OpenShift