Tag Archives: Fix


Kubernetes Troubleshooting – Kubelet Unable to attach or mount volumes – timed out waiting for the condition

The Issue

When I updated my Kasten application in my Kubernetes cluster, I found that one of the pods was stuck in “init” status.

dean@dean [ ~ ] (⎈ |tkg-wld-01-admin@tkg-wld-01:default) # k get pods -n kasten-io -w
aggregatedapis-svc-78564d4697-wl9wg 1/1 Running 0 3m9s
auth-svc-7977b9684b-zph27 1/1 Running 0 3m11s
catalog-svc-7ff7779b75-kmvsr 0/2 Init:0/2 0 2m43s

kubectl get pods - status init

Running a describe on that pod pointed to the fact the volume could not be attached.

Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m58s default-scheduler Successfully assigned kasten-io/catalog-svc-7ff7779b75-kmvsr to tkg-wld-01-md-0-54598b8d99-rpqjf
Warning FailedMount 55s kubelet Unable to attach or mount volumes: unmounted volumes=[catalog-persistent-storage], unattached volumes=[k10-k10-token-lbqpw catalog-persistent-storage]: timed out waiting for the condition
kubelet Unable to attach or mount volumes- unmounted volumes=[catalog-persistent-storage], unattached volumes=[k10-k10-token-lbqpw catalog-persistent-storage]- timed out waiting for the condition
The Cause

Some where along the line I found some stale volumeattachments linked to Kubernetes node that no longer exist in my cluster. This looks to be causing some confusion in the cluster who should be attaching the volume

The image below shows:

  • Find the Persistent Volume name linked to the associated claim for the failure in the pod events
  • Map this to the available VolumeAttachments
  • Reference VolumeAttachments for each node to available nodes in the cluster
    • I’ve highlighted the missing node in the red box

kubectl get pv - get volumeattachment - get nodes

The Fix

The fix is to remove the stale VolumeAttachment.

kubectl delete volumeattachment [volumeattachment_name]

kubectl delete volumeattachment

After this your pod should eventually pick up and retry, or you could remove the pod and let Kubernetes replace it for you (so long as it’s part of a deployment or other configuration managing your application).


Dean Lewis


How To Fix A PVC Stuck in Terminating Status in Kubernetes: Troubleshooting Guide

Having trouble deleting a persistent volume claim (PVC) stuck interminating status in Kubernetes/Openshift? We‘ve got the fix. Read on to learn how to patch the PVC to allow the final unmount and delete the PVC.

The Issue

Whilst working on a Kubernetes demo for a customer, I was cleaning up my environment and deleting persistent volume claims (PVC) that were no longer need.

I noticed that one PVC was stuck in “terminating” status for quite a while.

Kubernetes pvc terminating

Note: I am using the OC commands in place of kubectl due to this being a Openshift environment

The Cause

I had a quick google and found I needed to verify if the PVC is still attached to a node in the cluster.

kubectl get volumeattachment

I could see it was, and the reason behind this was the configuration for the PVC was not fully updated during the delete process.

Kubernetes pvc terminating kubectl get volumeattachment

The Fix

I found the fix on this github issue log .

You need to patch the PVC to set the “finalizers” setting to null, this allows the final unmount from the node, and the PVC can be deleted.

kubectl patch pvc {PVC_NAME} -p '{"metadata":{"finalizers":null}}'

Kubernetes pvc terminating kubectl patch pvc


Smashing Laptop With Hammer

ESXi 6.0 CBT Issue KB 2136854 – Resolved #vDM30in30

I wrote about the latest CBT issue (November edition) a couple of days ago, and as promised by VMware a patch has been released.

Original issue – KB 2136854

Patch to resolve the issue – ESXi600-201511001 (2137545)

Note: Before applying this patch on HP Proliant Gen 9 servers, see VMware ESXi 6.0 host fails to detect local disk after patching or applying Update 1 on HP Proliant Gen 9 servers (2120539).


Cheers VMware, lets hope this is the last of the CBT issues!!!



HP 2920 Switch – Reboot issue on firmware ver 15.18.0006 #vDM30in30

Updated 25.11.15

Firmware WB.15.18.0007 resolves the issue, see below

A colleague of mine found an issue with the latest HP 2920 switch firmware.
If you create VLANs using the CLI Menu, the switch reboots and the configuration is not saved.

We have reported this to HP, but is currently being treated as a non critical issue as when creating a VLAN via the web interface or native CLI, the issue doesn’t happen.

We have also noticed on this firmware the switch seems to be less responsive. Luckily we had a few units in stock that we could replicate this issue on, and can confirm downgrading to the previous firmware version removes the issue.

A quick cheers to my colleague Marco for finding and researching this issue.

The issue

Switch: HP 2920-48G-POE+

Primary Image    :    12852982 08/12/15 WB.15.18.0006

Software revision  : WB.15.18.0006

  1.        Go to the Main Menu
  2.        Select (2) Switch Configuration…
  3.        Select (8) VLAN Menu…
  4.        Select (3) VLAN Port Assignment
  5.        Select Edit
  6.        Modify the tagging mode for a port
  7.        Select Save
  8.        Switch reboots and doesn’t save configuration

Hopefully HP will release a fix for this firmware soon, as mentioned we have recreated this issue in production and test.

The Fix

The following information was provided by HP Support. Continue reading HP 2920 Switch – Reboot issue on firmware ver 15.18.0006 #vDM30in30


Installing VMware Horizon View Composer fails with Error 1920

The issue

So today I was installing VMware Horizon View Composer on my Server 2012 R2 vCenter machine, where the SQL database is also local to this machine.

And I came across the below error,

"Error 1920, Service VMware Horizon View Composer (svid) failed to start. Verify that you have sufficient privileges to start the system service"

Which I something I’ve dealt with before, but I must admit I forgot the fix, and was looking into the following area’s;

  • Check account is Administrator and has “Log on as a Service” right
  • Check there is no pending restarts
  • Check ODBC is set up correctly and gives a successful message when testing


The fix

Continue reading Installing VMware Horizon View Composer fails with Error 1920