Red Hat OpenShift Header

Openshift-install CLI Tool – Crash – Unable to decode instructions – Apple MacBook M1

Kubernetes, ,
The Issue

When running the OpenShift-Install CLI tool on my Apple MacBook M1 to create an OpenShift Cluster I kept hitting the same error:

assertion failed [inst.has.value()]: failed to decode instruction: 0x0

Openshift-install CLI Tool - Crash - Unable to decode instructions - Apple MacBook M1

The Cause

This is believed to be an issue created with the use of Rosetta 2 and Golang, and is somewhat documented on this GitHub issue by Apple Engineering.

The OpenShift-Install CLI Tool uses Terraform which relies on GoLang.

The Fix

In the above GitHub issue, it is found that running the below command either locally, or keeping it in your ~/.zshrc file will resolve the issue as a workaround.

export GODEBUG=asyncpreemptoff=1

Thank you to Andrew Sullivan from Red Hat, who pointed me to this blog post to help me find the answer!

Regards

Dean Lewis

AWS EKS Header

EKS – Kubectl – Unable to connect to the server – Exec plugin is configured to use API version client.authentication.k8s.io/v1alpha1

Kubernetes, , ,
The Issue

After moving my life over to a new Macbook and installing the latest AWS CLI tools including “aws-iam-authenticator” tool, I couldn’t run commands against my EKS Clusters. I kept hitting the following issue;

> kubectl get pods

Unable to connect to the server: getting credentials: exec plugin is configured to use API version client.authentication.k8s.io/v1alpha1, plugin returned version client.authentication.k8s.io/v1beta1
eks - aws-iam-authenticator - unable to connect to cluster
The Cause

AWS updated the aws-iam-authenicator component in version 0.5.4 to require v1beta1 your kubeconfig file for the cluster context. You will be using v1alpha1 more than likely, which generates this error.

The Fix

Update your kubeconfig file as necessary, replacing “v1alpha1” for “v1beta1” for any contexts for EKS clusters.

vi ~/.kube/config

# Alernatively you could run something like the below to automate the changes. This will also create a "config.bak" file of the orignal file before the changes

sed -i .bak -e 's/v1alpha1/v1beta1/' ~/.kube/config

eks - aws-iam-authenticator - v1alpha1 - v1beta1 - kubeconfig file

Below you can see I used the “sed” command, checked my file using “vi” then run the kubectl command successfully.

eks - aws-iam-authenticator - sed -i .bak -e 's:v1alpha1:v1beta1:' ~:.kube:config

Official GitHub Page

 

Regards

Dean Lewis

Tanzu Blog Logo Header

Tanzu Service Mesh – Monitor Service Level Objectives and Configure Service Autoscaling

VMware, Kubernetes, , , ,

Continuing from the First Look blog post, where we created a distributed application between different public cloud Kubernetes deployments and connected them via Tanzu Service Mesh. We will move onto some of the more advanced capabilities of Tanzu Service Mesh.

In this blog post, we’ll look at how we can setup monitoring of our application components and performance against a Service Level Objective, and then how Tanzu Mission Control and action against violations of the SLO using auto-scaling capabilities.

What is a Service Level Objective and how do we monitor our app?

Service level objectives (SLO/s) provide a structured way to describe, measure, and monitor the performance, quality, and reliability of micro-service apps.

A SLO is used to describe the high-level objective for acceptable operation and health of one or more services over a length of time (for example, a week or a month).

  • For example, Service X should be healthy 99.1% of the time.

In the provided example, Service X can be “unhealthy” 1% of the time, which is considered an “Error Budget”. This allows for downtime for errors that are acceptable (keeping an app up 100% of the time is hard and expensive to achieve), or for the likes of planned routine maintenance.

The key is the specification of which metrics or characteristics, and associated thresholds are used to define the health of the micro-service/application.

  • For example:
    • Error rate is less than 2%
    • CPU Average is Less than 80%

This specification makes up the Service Level Indicator (SLI/s), of which one or multiple can be used to define an overall SLO.

Tanzu Service Mesh SLOs options

Before we configure, let’s quickly discuss what is available to be configured.

Tanzu Service Mesh (TSM) offers two SLO configurations:

  • Monitored SLOs
    • These provide alerting/indicators on performance of your services and if they meet your target SLO conditions based on the configured SLIs for each specified service.
    • This kind of SLO can be configured for Services that are part of a Global Namespace (GNS-scoped SLOs) or services that are part of a direct cluster (org-scoped SLOs).
  • Actionable SLOs
    • These extend the capabilities of Monitored SLOs by providing capabilities such as auto-scaling for services based on the SLIs.
    • This kind of SLO can only be configured for services inside a Global Namespace (GNS-scoped SLO).

    • Each actionable SLO can have only have one service, and a service can only have one actionable SLO.

The official documentation also takes you through some use-cases for SLOs. Alternatively, you can continue to follow this blog post for an example.

Quick overview of the demo environment
  • Tanzu Service Mesh (of course)
    • Global Namespace configured for default namespace in clusters with domain “app.sample.com”
  • Three Kubernetes Clusters with a scaled-out application deployed
    • AWS EKS Cluster
      • Running web front end (shopping) and cart instances
    • Azure AKS Cluster
      • Running Catalog Service that holds all the images for the Web front end
    • GCP GKE
      • Running full copy of the application

In this environment, I’m going to configure a SLO which is focused on the Front-End Service – Shopping, and will scale up the number of pods when the SLIs are breached.

Configure a SLO Policy and Autoscaler
  • Under the Policies header, expand
  • Select “SLOs”
  • Select either New Policy options

Continue reading Tanzu Service Mesh – Monitor Service Level Objectives and Configure Service Autoscaling →

gke header

GKE – User cannot create resource – requires one of [“container.roles.create”] permission(s)

Kubernetes, , , , ,
The Issue

I stood up my first ever GKE cluster! Woo, go me!

However when I was trying to setup Tanzu Service Mesh, I hit issues such as:

Error from server (Forbidden): error when creating "operator-deployment.yaml": roles.rbac.authorization.k8s.io is forbidden: User "[email protected]" cannot create resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "vmware-system-tsm": requires one of ["container.roles.create"] permission(s).
The Cause

This is because your initial Kubernetes login has no cluster level permissions, due to the RBAC setup.

The Fix

You need to create some new Cluster Level roles and bind to them with your account, or use the existing ones.

As this is a demo environment. I just bound my account to the out-of-the-box cluster-admin ClusterLevelRole (that is a mouthful!).

kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole=cluster-admin \
--user=[gcp user email]

# Example
kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole=cluster-admin \
[email protected]

If you need to double check with google account you are using, you can run:

gcloud info | grep Account

Regards

Dean Lewis

google cloud header

Google Cloud – Invitation email not received – Project IAM role pending

General, , ,
The Issue

For me, it started off with having some odd issues in a GKE cluster, where I didn’t have permissions to do things at a cluster level. After some digging it pointed to the wrong IAM roles on the Google Cloud Project.

When I investigated this, I found I wasn’t yet confirmed as the owner of the project. It said an email was sent, but I had received nothing!

google cloud - IAM - Invitation sent pending acceptance

The Cause

Maybe something wrong with Googles SMTP? Or spam filters on the receivers side. But it doesn’t help you cannot resent the email!

The Fix

You can accept the invitation by going to the below link.

https://console.cloud.google.com/invitation?project=[your-project-id]&account=[the-account-email-invited]&memberEmail=[the-account-email-invited]

Example
https://console.cloud.google.com/invitation?project=veducate-demo&[email protected]&[email protected]

Regards

Dean Lewis