Tag Archives: Fails

AWS EKS Header

Deleting AWS EKS Cluster fails – Cannot evict pod as it would violate the pod’s disruption budget

The Issue

I had to remove a demo EKS Cluster where I had screwed up an install of a Service Mesh. Unfortunately, it was left in a rather terrible state to clean up, hence the need to just delete it.

When I tried the usual eksctl delete command, including with the force argument, I was hitting errors such as:

2021-12-21 23:52:22 [!] pod eviction error ("error evicting pod: istio-system/istiod-76f699dc48-tgc6m: Cannot evict pod as it would violate the pod's disruption budget.") on node ip-192-168-27-182.us-east-2.compute.internal

With a final error output of:

Error: Unauthorized

eksctl delete cluster - Cannot evict pod as it would violate the pod's disruption budget - Error Unauthorized

The Cause

Well, the error message does call out the cause, moving the existing pods to other nodes is failing due to the configured settings. Essentially EKS will try and drain all the nodes and shut everything down nicely when it deletes the cluster. It doesn’t just shut everything down and wipe it. This is because inside of Kubernetes there are several finalizers that will call out actions to interact with AWS components (thanks to the integrations) and nicely clean things up (in theory).

To get around this, I first tried the following command, thinking if delete the nodegroup without waiting for a drain, this would bypass the issue:

 eksctl delete nodegroup standard --cluster veducate-eks --drain=false --disable-eviction

This didn’t allow me to delete the cluster however, I still got the same error messages.

The Fix

So back to the error message, and then I realised it was staring me in the face!

Cannot evict pod as it would violate the pod's disruption budget

What is a Pod Disruption Budget? It’s essentially a way to ensure availability of your pods from someone killing them accidentality.

A PDB limits the number of Pods of a replicated application that are down simultaneously from voluntary disruptions. For example, a quorum-based application would like to ensure that the number of replicas running is never brought below the number needed for a quorum. A web front end might want to ensure that the number of replicas serving load never falls below a certain percentage of the total.

To find all configured Pod Disruption Budgets:

kubectl get poddisruptionbudget -A

Then delete as necessary:

kubectl delete poddisruptionbudget {name} -n {namespace}

eks - kubectl get poddisruptionbudgets -A - kubectl delete poddisruptionbudgets

Finally, you should be able to delete your cluster.

eksctl delete cluster - successful

 

Regards

Dean Lewis

Ubuntu – apt-get update fails “The repository no longer has a Release file”

The Issue

Running sudo apt-get update or sudo apt update fails with the following or similar errors:

Hit:1 http://apt.postgresql.org/pub/repos/apt eoan-pgdg InRelease
Ign:2 http://archive.ubuntu.com/ubuntu eoan InRelease
Ign:3 http://archive.ubuntu.com/ubuntu eoan-updates InRelease
Ign:4 http://archive.ubuntu.com/ubuntu eoan-backports InRelease
Ign:5 http://archive.ubuntu.com/ubuntu eoan-security InRelease
Err:6 http://archive.ubuntu.com/ubuntu eoan Release
404 Not Found [IP: 91.189.88.142 80]
Err:7 http://archive.ubuntu.com/ubuntu eoan-updates Release
404 Not Found [IP: 91.189.88.142 80]
Err:8 http://archive.ubuntu.com/ubuntu eoan-backports Release
404 Not Found [IP: 91.189.88.142 80]
Err:9 http://archive.ubuntu.com/ubuntu eoan-security Release
404 Not Found [IP: 91.189.88.142 80]
Reading package lists... Done
E: The repository 'http://archive.ubuntu.com/ubuntu eoan Release' no longer has a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository 'http://archive.ubuntu.com/ubuntu eoan-updates Release' no longer has a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository 'http://archive.ubuntu.com/ubuntu eoan-backports Release' no longer has a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository 'http://archive.ubuntu.com/ubuntu eoan-security Release' no longer has a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
The Cause

Older releases of Ubuntu are moved to an archive server, so you need to update your repo lists.

The Fix
# backup your sources file
cp /etc/apt/sources.list /etc/apt/sources.list.bak 

# replace the links with the archive address
sudo sed -i -re 's/([a-z]{2}.)?archive.ubuntu.com|security.ubuntu.com/old-releases.ubuntu.com/g' /etc/apt/sources.list

# run update again
sudo apt-get update && sudo apt-get dist-upgrade

Regards

2015 03 12 23 16 34

Deploying VCSA 6.0 fails with supplied system name is invalid

2015-04-14_16-28-02_01

 

A nice and simple one, the supplied system name (which is the FQDN of the VCSA)  is invalid.

This is because when deploying a new VCSA from scratch, your DNS records should already be created, as the new ISO based installer for the appliance runs a number of scripts to setup vCenter services and the Platform Services Controller.

So you get this error either because your DNS Settings are wrong, or you have not created your DNS records for your vCenter and PSC if it is an external server.

 

Regards

Dean