Tag Archives: delete

Kubernetes

Quick Tip – Kubernetes – Delete all evicted pods across all namespaces

I’m currently troubleshooting an issue with my Kubernetes clusters where pods keep getting evicted, and this is happening across namespaces as well.

The issue now that I am faced with, is being able to keep ontop of the issues. When I run:

kubectl get pods -A | grep Evicted

I’m presented with 100’s of returned results.

kubectl get pods -A grep Evicted

So to quickly clean this up, I can run the following command: Continue reading Quick Tip – Kubernetes – Delete all evicted pods across all namespaces

VMware Tanzu Header

Tanzu Mission Control – Delete a provisioned cluster

In this blog post we are going to cover off how to delete a Tanzu Kubernetes Grid cluster that has been provisioned by Tanzu Mission Control. We will cover the following areas:

Below are the other blog posts in the series.

Tanzu Mission Control 
- Getting Started Tanzu Mission Control 
- Cluster Inspections 
- Workspaces and Policies  
- Data Protection 
- Deploying TKG clusters to AWS 
- Upgrading a provisioned cluster 
- Delete a provisioned cluster 
- TKG Management support and provisioning new clusters
- TMC REST API - Postman Collection

We are going to use the cluster I created in my last blog post.

Below are my EC2 instances that make up my TMC provisioned cluster, here I have filtered my view using the field “tmc.cloud.vmware.com/cluster” + cluster name.

Tanzu Mission Control - AWS Consoles - Instances - Filtered tmc.cloud.vmware.com

Deleting a Provisioned cluster in the TMC UI

In the TMC UI, going to the clusters view, you can click the three dots next to the cluster you want to remove and select delete.

Tanzu Mission Control - Clusters - Delete cluster

Alternatively, within the cluster object view, click actions then delete.

Tanzu Mission Control - Cluster Object - Delete cluster

Both options will bring up the below confirmation dialog box.

You select one of the following options:

  • Delete and remove agent (recommended)
    • Remove from TMC and delete agent extensions
  • Manually delete agent extensions
    • A secondary option whereby a manual removal is needed if a cluster delete fails

Enter the name of the cluster you want to delete, to confirm the cluster deletion.

Tanzu Mission Control - Cluster Object - Delete cluster - Confirm Continue reading Tanzu Mission Control – Delete a provisioned cluster

VMware Cloud Foundation VCF Header

VCF – SDDC Manager – How to delete bundles

Just a quick blog post on deleting unnecessary or unneeded bundles from VCF – SDDC Manager.

There is two parts to this.

  • Getting your Bundle ID you want to delete from the API
  • Deleting the Bundle using a script on the SDDC Manager appliance.

In your SDDC Manager:

  1. Click Development Center
  2. Click API Explorer
  3. Expand “APIs for managing bundles”
  4. Expand the first “GET” command

VCF SDDC Manager - API - Get Bundles

  • Click Execute, no need to fill anything in

VCF SDDC Manager - API - Get Bundles - Execute

  • Download or Copy the response output.

VCF SDDC Manager - API - Get Bundles - Response

  • Find your Bundle ID within your output, you need to look for the top level ID of the JSON block, and ensure that this bundle says successfully downloaded.

VCF SDDC Manager API Get Bundles Response JSON Find Bundle ID

SSH to your SDDC Manager and elevate to root.

# su
{provide password to elevate to root}
# /opt/vmware/vcf/lcm/lcm-app/bin/bundle_cleanup.py {Bundle_id}

Example below
# /opt/vmware/vcf/lcm/lcm-app/bin/bundle_cleanup.py f004390e-26be-4690-9d7b-d447860e3169

VCF SDDC Manager bundle cleanup.py script

You will see the following output when the script has run.

-----------------------------------------------------
LOG FILE : /var/log/vmware/vcf/lcm/bundle_cleanup.log
-----------------------------------------------------
2021-03-08 12:18:31,809 [INFO] root: Performing cleanup for bundle with IDs : [' f004390e-26be-4690-9d7b-d447860e3169']
2021-03-08 12:18:31,809 [INFO] root: Execute cmd: psql --host=localhost -U postg res -d lcm -tc "select count(*) from upgrade where upgrade_status in ('INPROGRES S','CANCELLING');"
2021-03-08 12:18:31,848 [INFO] root: b' 0\n'
2021-03-08 12:18:31,848 [INFO] root: b'\n'
2021-03-08 12:18:31,848 [INFO] root: RC: 0
2021-03-08 12:18:31,849 [INFO] root: Out: 0

2021-03-08 12:18:31,849 [INFO] root: Stopping LCM service.
2021-03-08 12:18:31,849 [INFO] root: Execute cmd: systemctl stop lcm
2021-03-08 12:18:32,290 [INFO] root: RC: 0
2021-03-08 12:18:32,290 [INFO] root: Out:
2021-03-08 12:18:32,291 [INFO] root: Removing LCM NFS mount.
2021-03-08 12:18:32,291 [INFO] root: Execute cmd: rm -rf /nfs/vmware/vcf/nfs-mou nt/bundle/f004390e-26be-4690-9d7b-d447860e3169
2021-03-08 12:18:32,683 [INFO] root: RC: 0
2021-03-08 12:18:32,684 [INFO] root: Out:
2021-03-08 12:18:32,684 [INFO] root: Execute cmd: psql --host=localhost -U postg res -d lcm -tc "select upload_id from bundle_upload where bundle_id = 'f004390e- 26be-4690-9d7b-d447860e3169';"
2021-03-08 12:18:32,704 [INFO] root: b'\n'
2021-03-08 12:18:32,705 [INFO] root: RC: 0
2021-03-08 12:18:32,705 [INFO] root: Out:

2021-03-08 12:18:32,705 [INFO] root: Bundle with ID : f004390e-26be-4690-9d7b-d4 47860e3169 not found in bundle upload table
2021-03-08 12:18:32,706 [INFO] root: Execute cmd: psql --host=localhost -U postg res -d lcm -tc "select download_id from bundledownload_by_id where bundle_id = ' f004390e-26be-4690-9d7b-d447860e3169';"
2021-03-08 12:18:32,724 [INFO] root: b' 0fb2e30e-d991-4b63-8686-42fab98a1c9e\n'
2021-03-08 12:18:32,724 [INFO] root: b'\n'
2021-03-08 12:18:32,725 [INFO] root: RC: 0
2021-03-08 12:18:32,725 [INFO] root: Out: 0fb2e30e-d991-4b63-8686-42fab98a1c9e

2021-03-08 12:18:32,725 [INFO] root: Execute cmd: curl -s -X DELETE localhost/ta sks/registrations/0fb2e30e-d991-4b63-8686-42fab98a1c9e
2021-03-08 12:18:32,830 [INFO] root: RC: 0
2021-03-08 12:18:32,830 [INFO] root: Out:
2021-03-08 12:18:32,830 [INFO] root: Execute cmd: psql --host=localhost -U postg res -d lcm -tc "select upgrade_id from upgrade where bundle_id = 'f004390e-26be- 4690-9d7b-d447860e3169';"
2021-03-08 12:18:32,852 [INFO] root: b'\n'
2021-03-08 12:18:32,853 [INFO] root: RC: 0
2021-03-08 12:18:32,853 [INFO] root: Out:

2021-03-08 12:18:32,853 [INFO] root: Bundle with ID : f004390e-26be-4690-9d7b-d4 47860e3169 not found in upgrade table
2021-03-08 12:18:32,854 [INFO] root: Execute cmd: psql --host=localhost -U postg res -d lcm -tc "select upgrade_id from upgrade where bundle_id = 'f004390e-26be- 4690-9d7b-d447860e3169';"
2021-03-08 12:18:32,873 [INFO] root: b'\n'
2021-03-08 12:18:32,874 [INFO] root: RC: 0
2021-03-08 12:18:32,874 [INFO] root: Out:

2021-03-08 12:18:32,874 [INFO] root: Bundle with ID : f004390e-26be-4690-9d7b-d4 47860e3169 not found in upgrade table
2021-03-08 12:18:32,875 [INFO] root: Execute cmd: psql --host=localhost -U postg res -d lcm -tc "select count(*) from bundle where bundle_id = 'f004390e-26be-469 0-9d7b-d447860e3169';"
2021-03-08 12:18:32,894 [INFO] root: b' 1\n'
2021-03-08 12:18:32,895 [INFO] root: b'\n'
2021-03-08 12:18:32,895 [INFO] root: RC: 0
2021-03-08 12:18:32,895 [INFO] root: Out: 1

2021-03-08 12:18:32,896 [INFO] root: Deleting bundle & upgrade info for bundle I D : f004390e-26be-4690-9d7b-d447860e3169
2021-03-08 12:18:32,896 [INFO] root: Execute cmd: psql --host=localhost -U postg res -d lcm -c "delete from bundle where bundle_id = 'f004390e-26be-4690-9d7b-d44 7860e3169';"
2021-03-08 12:18:32,923 [INFO] root: b'DELETE 1\n'
2021-03-08 12:18:32,924 [INFO] root: RC: 0
2021-03-08 12:18:32,924 [INFO] root: Out: DELETE 1

2021-03-08 12:18:32,924 [INFO] root: Execute cmd: psql --host=localhost -U postg res -d lcm -tc "select count(*) from image where bundle_id = 'f004390e-26be-4690 -9d7b-d447860e3169';"
2021-03-08 12:18:32,943 [INFO] root: b' 1\n'
2021-03-08 12:18:32,943 [INFO] root: b'\n'
2021-03-08 12:18:32,943 [INFO] root: RC: 0
2021-03-08 12:18:32,944 [INFO] root: Out: 1

2021-03-08 12:18:32,944 [INFO] root: Deleting bundle f004390e-26be-4690-9d7b-d44 7860e3169 in image table
2021-03-08 12:18:32,944 [INFO] root: Execute cmd: psql --host=localhost -U postg res -d lcm -c "delete from image where bundle_id = 'f004390e-26be-4690-9d7b-d447 860e3169';"
2021-03-08 12:18:32,967 [INFO] root: b'DELETE 1\n'
2021-03-08 12:18:32,967 [INFO] root: RC: 0
2021-03-08 12:18:32,967 [INFO] root: Out: DELETE 1

2021-03-08 12:18:32,968 [INFO] root: Execute cmd: psql --host=localhost -U postg res -d lcm -tc "select count(*) from partner_bundle_metadata where bundle_id = ' f004390e-26be-4690-9d7b-d447860e3169';"
2021-03-08 12:18:32,990 [INFO] root: b' 0\n'
2021-03-08 12:18:32,990 [INFO] root: b'\n'
2021-03-08 12:18:32,990 [INFO] root: RC: 0
2021-03-08 12:18:32,990 [INFO] root: Out: 0

2021-03-08 12:18:32,990 [INFO] root: Bundle with ID : f004390e-26be-4690-9d7b-d4 47860e3169 not found in partner_bundle_metadata table
2021-03-08 12:18:32,991 [INFO] root: Starting LCM service.
2021-03-08 12:18:32,991 [INFO] root: Execute cmd: systemctl start lcm
2021-03-08 12:18:33,135 [INFO] root: RC: 0
2021-03-08 12:18:33,136 [INFO] root: Out:

Going back into your SDDC Manager UI, and clicking the Bundle Management page, you will see your bundle has now been deleted.

It will take a few minutes for the Bundle services to restart, and you may see the message “Depot still initializing”.

Regards

 

VMware AWS Header

How to delete vCenter Roles in VMC

The Issue

Whilst testing in VMC a PowerCLI script to create some vCenter roles, I noticed in the UI, then I deleted them, they remained, even though I was using the CloudAdmin@vmc.local account.

I also tried to delete them using PowerCLI and received the error message;

Remove-VIRole : 07/11/2020 09:00:42 Remove-VIRole Permission to perform this operation was denied. Required privilege 'VApp.PullFromUrls' on managed object with id 'Folder-group-d1'.
At line:1 char:1
+ Remove-VIRole OpenShift-Install
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Remove-VIRole], NoPermission
+ FullyQualifiedErrorId : Client20_InventoryServiceImpl_RemoveRole_VIError,VMware.VimAutomation.ViCore.Cmdlets.Commands.PermissionManagement.RemoveVIRole

You should not create roles with permissions higher than that of the CloudAdmin Account. You can find further information of these permissions here.

If you do this, the only fix is to log a support call with VMware to resolve.

The Fix

VMware have a KB for this issue and how to delete the vCenter roles.

To resolve you use the vCenter Managed Objects Browser (MOB).

Note: When using the MOB to make changes, users will not be prompted for confirmation before making any changes, including removing roles. A custom role can not have privileges higher than the CloudGlobalAdmin role.

First to view all your existing roles in your browser go to;

  • ¬†https://{VMC_VC_FQDN}/mob/?moid=AuthorizationManager&doPath=roleList

This will list all roles, and note the roleId for the role you want to remove.

vCenter MOB AuthorizationRole

To remove a role: Continue reading How to delete vCenter Roles in VMC