Kubernetes

How To Fix A PVC Stuck in Terminating Status in Kubernetes: Troubleshooting Guide

Having trouble deleting a persistent volume claim (PVC) stuck interminating status in Kubernetes/Openshift? We‘ve got the fix. Read on to learn how to patch the PVC to allow the final unmount and delete the PVC.

The Issue

Whilst working on a Kubernetes demo for a customer, I was cleaning up my environment and deleting persistent volume claims (PVC) that were no longer need.

I noticed that one PVC was stuck in “terminating” status for quite a while.

Kubernetes pvc terminating

Note: I am using the OC commands in place of kubectl due to this being a Openshift environment

The Cause

I had a quick google and found I needed to verify if the PVC is still attached to a node in the cluster.

kubectl get volumeattachment

I could see it was, and the reason behind this was the configuration for the PVC was not fully updated during the delete process.

Kubernetes pvc terminating kubectl get volumeattachment

The Fix

I found the fix on this github issue log .

You need to patch the PVC to set the “finalizers” setting to null, this allows the final unmount from the node, and the PVC can be deleted.

kubectl patch pvc {PVC_NAME} -p '{"metadata":{"finalizers":null}}'

Kubernetes pvc terminating kubectl patch pvc

Regards

VMworld 2020 Header

Dean’s top 10 session picks for VMworld 2020

I thought I’d add to the deluge of posts of recommended VMworld Sessions.

Here are my top ten picks!

Remember VMworld is fully virtual and free this year. So what’s stopping you from registering?

The Sessions Continue reading Dean’s top 10 session picks for VMworld 2020

OpenShift

Using the vSphere CSI Driver with OpenShift 4.x and VSAN File Services

You may have seen my blog post “How to Install and configure vSphere CSI Driver on OpenShift 4.x“.

Here I updated the vSphere CSI driver to work the additional security constraints that are baked into OpenShift 4.x.

Since then, once of the things that has been on my list to test is file volumes backed by vSAN File shares. This feature is available in vSphere 7.0.

Well I’m glad to report it does in fact work, by using my CSI driver (see above blog or my github), you can simply deploy consume VSAN File services, as per the documentation here. 

I’ve updated my examples in my github repository to get this working.

OK just tell me what to do…

First and foremost, you need to add additional configuration to the csi conf file (csi-vsphere-for-ocp.conf).

If you do not, the defaults will be assumed which is full read-write access from any IP to the file shares created.

[Global]

# run the following on your OCP cluster to get the ID 
# oc get clusterversion -o jsonpath='{.items[].spec.clusterID}{"\n"}'
cluster-id = c6d41ba1-3b67-4ae4-ab1e-3cd2e730e1f2

[NetPermissions "A"]
ips = "*"
permissions = "READ_WRITE"
rootsquash = false

[VirtualCenter "10.198.17.253"]
insecure-flag = "true"
user = "[email protected]"
password = "Admin!23"
port = "443"
datacenters = "vSAN-DC"
targetvSANFileShareDatastoreURLs = "ds:///vmfs/volumes/vsan:52c229eaf3afcda6-7c4116754aded2de/"

Next, create a storage class which is configured to consume VSAN File services.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: file-services-sc
annotations:
storageclass.kubernetes.io/is-default-class: "false"
provisioner: csi.vsphere.vmware.com
parameters:
storagepolicyname: "vSAN Default Storage Policy" # Optional Parameter
csi.storage.k8s.io/fstype: "nfs4" # Optional Parameter

Then create a PVC to prove it works. Continue reading Using the vSphere CSI Driver with OpenShift 4.x and VSAN File Services

Veeam vRA Header

How to backup vRealize Automation 8.x using Veeam

In this blog post I am going to dissect backing up vRealize Automation 8.x using Veeam Backup and Replication.

- Understanding the backup methods
- Performing an online backup
- Performing an offline backup

Understanding the Backup Methods

Reading the VMware documentation around this subject can be somewhat confusing at times. And if you pay attention, there are subtle changes between the documents as well. Lets break this down.

  • vRealize Automation 8.0
    • As part of the backup job, you need to run a script to stop the services
    • This is known as an offline backup
    • Depending on your backup software, you can either do this by running a script located on the vRealize Automation appliance or by triggering using the pre-freeze/post-freeze scripts when a snapshot is taken of the VM.
    • The snapshot must not include the virtual machines memory.
    • If you environment is a cluster, you only need to run the script on a single node.
    • All nodes in the cluster must be backed up at the same time.
  • vRealize Automation 8.0.1 and 8.1 (and higher)
    • It is supported to run an online backup
      • No script is needed to shut down the services
    • Snapshot taken as part of the backup must quiesce the virtual machine.
    • The snapshot must not include the virtual machines memory.
    • It is recommended to run the script to stop all services and perform an offline backup.
      • You may also find your backup runs faster, as the virtual machine will become less busy.

Performing an Online Backup

Let’s start with the easier of the two options. Again, this will be supported for vRealize Automation 8.0.1 and higher. Continue reading How to backup vRealize Automation 8.x using Veeam

Veeam new logo featured image

Veeam – Script finished execution with unexpected exit code: 126

This Issue

When using Veeam if you use Pre/Post freeze scripts for application aware processing, these scripts must reside on the VBR server, which uploads to the virtual machine you are protecting.

I hit an issue when running a job that I received the error;

Error: Script finished execution with unexpected exit code: 126

Veeam Script finished execution with unexpected exit code 126

The Cause

I reviewed the task logs on the VBR server;

  • C:\ProgramData\Veeam\Backup\{Job_Name}\task_{VMName}.txt

You are looking for the sections [ScriptInvoker] for troubleshooting. And I found that my script was uploaded successfully, however when it was ran the Guest OS was throwing an error;

[ScriptInvoker] Failed to execute script in lin guest machine over SSH. Script path: C:\Backup\Scripts\stop-vra.sh.

bash: /tmp/d6791b89-e0b8-4cce-acec-45d682ce1f2c_stop-vra.sh: /bin/bash^M: bad interpreter: No such file or directory

I also connected to the linux machine and ran “journalctl -xe -f” whilst running the backup job and seen the same error flash up too.

The script indicates that it must be executed by a shell located at /bin/bash^M. 
There is no such file: it's called /bin/bash. The ^M is a carriage return character. 
Linux uses the line feed character to mark the end of a line, whereas Windows uses the two-character sequence CR LF. Your file has Windows line endings, which is confusing Linux. 
(Source)

You can see the full part of the task logs relating to this below.

[ScriptInvoker] Scripting mode is FailJobOnError.
[ScriptInvoker] Script enabled
[ScriptInvoker] Creating Linux invoker.
[ScriptInvoker] Starting pre-freeze script execution 
[ScriptInvoker] Running Linux script (SSH) 'C:\Backup\Scripts\stop-vra.sh'
[Ssh] SSH connection ae2f5df0-1d28-4dfc-a8a5-7ceb952af2a9 to server 192.168.200.39 created successfully
...
[ScriptInvoker] SSH connection is established (192.168.200.39).
[ScriptInvoker] Exception thrown during script execution (SSH).
[Ssh] Connection ae2f5df0-1d28-4dfc-a8a5-7ceb952af2a9 - [host: '192.168.200.39', port: 22, elevation to root: 'no', autoSudo: no, use su if sudo fails: no, host name: sc-dc1-vra001.simon.local, IPs: [192.168.200.39], AuthenticationData: [UserName: root, AuthTypes: [KeyboardInteractive, Password]]] is disposing.
[ScriptInvoker] Failed to execute script in lin guest machine over SSH. Script path: C:\Backup\Scripts\stop-vra.sh.
bash: /tmp/d6791b89-e0b8-4cce-acec-45d682ce1f2c_stop-vra.sh: /bin/bash^M: bad interpreter: No such file or directory
(System.Exception)
at Veeam.Backup.SSH.CSshCommandResult.GetAnswer(Boolean trimAnswer, String failText, Boolean checkStdErr)
at Veeam.Backup.Core.CSshScriptInvoker.RunScript(CSshScriptFile scriptFile, TimeSpan timeout, Boolean collectLogs, String stdOutFilePath, String stdErrFilePath, Boolean checkStdErr, Int32& exitCode)
at Veeam.Backup.Core.CSshScriptInvoker.ExecScriptInner(String localPath, TimeSpan timeout, Boolean collectLogs, String stdOutFilePath, String stdErrFilePath, Boolean checkStdErr, Int32& exitCode)

[ScriptInvoker] Failed to execute script over SSH, failing over to VIX.
bash: /tmp/d6791b89-e0b8-4cce-acec-45d682ce1f2c_stop-vra.sh: /bin/bash^M: bad interpreter: No such file or directory

...
[ScriptInvoker] Running Linux script (VIX) 'C:\Backup\Scripts\stop-vra.sh'
[ScriptInvoker] Linux script exit code = '126'

The Fix

The fix is quite easy, either change your text editor to show the correct line endings.

Or create the script on a Linux machine and copy it to your VBR Server.

To fix this issue in your text editor

Notepad++ 
Edit --> EOL Conversion --> UNIX/OSX Format

Eclipse  
File > Convert Line Delimiters To > Unix (LF, \n, 0Α, ¶)

Or change the New text file line delimiter to Other: Unix on Window > Preferences > General > Workspace

Sublime Text Editor
View > Line Endings > Unix 

Atom
See this guide. 

(Source)

 

Regards