Kubernetes

vSphere CSI Driver Images unable from gcr.io – quick fix

The Issue

Someone has deleted the Cloud-Provider-vSphere project in the gcr.io registry for container images. The default pull policy for the vSphere CSI when using VMware’s manifests is set to always, meaning that if you reboot your cluster, it will not come back online.

vSphere-CSI Driver image unable - project deleted

This is what my cluster looked like when I booted it up today;

❯ kubectl get pods -n vmware-system-csi
NAME READY STATUS RESTARTS AGE
vsphere-csi-controller-776fb75cd8-ptw4s 5/7 ErrImagePull 0 84m
vsphere-csi-controller-776fb75cd8-qt7kv 5/7 ImagePullBackOff 0 84m
vsphere-csi-controller-776fb75cd8-s7btf 5/7 ImagePullBackOff 0 84m
vsphere-csi-node-5qjjw 1/3 CrashLoopBackOff 80 (111s ago) 142d
vsphere-csi-node-fmdkz 2/3 ImagePullBackOff 84 (3m5s ago) 143d
vsphere-csi-node-gbt9w 1/3 CrashLoopBackOff 6 (26s ago) 5m56s
vsphere-csi-node-jkj98 1/3 CrashLoopBackOff 86 (24s ago) 143d
vsphere-csi-node-r69bl 1/3 CrashLoopBackOff 85 (102s ago) 143d
vsphere-csi-node-ww2zx 2/3 ImagePullBackOff 89 (3m5s ago) 143d

And when describing the pod;

❯ kubectl describe pod -n vmware-system-csi vsphere-csi-controller-776fb75cd8-ptw4s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 85m default-scheduler 0/6 nodes are available: 2 node(s) didn't match Pod's node affinity/selector, 4 node(s) were unschedulable. preemption: 0/6 nodes are available: 6 Preemption is not helpful for scheduling.
Warning FailedScheduling 84m default-scheduler 0/6 nodes are available: 6 node(s) were unschedulable. preemption: 0/6 nodes are available: 6 Preemption is not helpful for scheduling.
Warning FailedScheduling 6m54s default-scheduler 0/6 nodes are available: 6 node(s) had untolerated taint {node.kubernetes.io/unschedulable: }. preemption: 0/6 nodes are available: 6 Preemption is not helpful for scheduling.
Normal Scheduled 6m27s default-scheduler Successfully assigned vmware-system-csi/vsphere-csi-controller-776fb75cd8-ptw4s to talos-2tp-6ld
Normal Created 6m26s kubelet Created container liveness-probe
Warning Failed 6m26s kubelet Failed to pull image "gcr.io/cloud-provider-vsphere/csi/release/syncer:v3.0.0": failed to pull and unpack image "gcr.io/cloud-provider-vsphere/csi/release/syncer:v3.0.0": failed to resolve reference "gcr.io/cloud-provider-vsphere/csi/release/syncer:v3.0.0": failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://gcr.io/v2/token?scope=repository%3Acloud-provider-vsphere%2Fcsi%2Frelease%2Fsyncer%3Apull&service=gcr.io: 401 Unauthorized
Normal Started 6m26s kubelet Started container csi-attacher
Normal Pulled 6m26s kubelet Container image "k8s.gcr.io/sig-storage/csi-resizer:v1.7.0" already present on machine
Normal Created 6m26s kubelet Created container csi-resizer
Normal Started 6m26s kubelet Started container csi-resizer
Normal Pulling 6m26s kubelet Pulling image "gcr.io/cloud-provider-vsphere/csi/release/driver:v3.0.0"
Warning Failed 6m26s kubelet Failed to pull image "gcr.io/cloud-provider-vsphere/csi/release/driver:v3.0.0": failed to pull and unpack image "gcr.io/cloud-provider-vsphere/csi/release/driver:v3.0.0": failed to resolve reference "gcr.io/cloud-provider-vsphere/csi/release/driver:v3.0.0": failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://gcr.io/v2/token?scope=repository%3Acloud-provider-vsphere%2Fcsi%2Frelease%2Fdriver%3Apull&service=gcr.io: 401 Unauthorized
Warning Failed 6m26s kubelet Error: ErrImagePull
Normal Pulled 6m26s kubelet Container image "k8s.gcr.io/sig-storage/livenessprobe:v2.9.0" already present on machine
Normal Pulled 6m26s kubelet Container image "k8s.gcr.io/sig-storage/csi-attacher:v4.2.0" already present on machine
Normal Started 6m26s kubelet Started container liveness-probe
Normal Pulling 6m26s kubelet Pulling image "gcr.io/cloud-provider-vsphere/csi/release/syncer:v3.0.0"
Normal Created 6m26s kubelet Created container csi-attacher
Warning Failed 6m26s kubelet Error: ErrImagePull
Normal Pulled 6m26s kubelet Container image "k8s.gcr.io/sig-storage/csi-provisioner:v3.4.0" already present on machine
Normal Created 6m26s kubelet Created container csi-provisioner
Normal Started 6m25s kubelet Started container csi-provisioner
Normal Pulled 6m25s kubelet Container image "k8s.gcr.io/sig-storage/csi-snapshotter:v6.2.1" already present on machine
Normal Created 6m25s kubelet Created container csi-snapshotter
Normal Started 6m25s kubelet Started container csi-snapshotter
Warning Failed 6m24s kubelet Error: ImagePullBackOff
Normal BackOff 6m24s kubelet Back-off pulling image "gcr.io/cloud-provider-vsphere/csi/release/syncer:v3.0.0"
Warning Failed 6m24s kubelet Error: ImagePullBackOff
Normal BackOff 83s (x21 over 6m24s) kubelet Back-off pulling image "gcr.io/cloud-provider-vsphere/csi/release/driver:v3.0.0"
The Cause

Who knows? Maybe it cost Broadcom too much to host the images in Google Cloud. Or maybe they are moving to a model where you can only access the files when you pay for VCF.

The Workaround

Luckily the images are mirrored by Rancher, so I just updated the vSphere CSI manifest from:

– https://raw.githubusercontent.com/kubernetes-sigs/vsphere-csi-driver/v3.3.1/manifests/vanilla/vsphere-csi-driver.yaml

And updated the image locations, you can get the updated file from my GitHub Gist belo. Continue reading vSphere CSI Driver Images unable from gcr.io – quick fix

So you want to work in Technical Marketing or other associated roles - cover image

So you want to work in Technical Marketing or other associated roles?

Does working in Tech Marketing appeal to you? Or another associated role, maybe DevRel, or a similar title?

Are you looking for the next career ladder step, or do you already have your heart set on this goal?

Yeah, I’ve been there, too, and as time passes, I’ve considered writing this post for a while.

Why? I think it’s because I like the sound of my keyboard clacking away as I type words. No, in all seriousness, I’ve been in this role for a while now, and somewhere on this timeline, I believed I had something worth sharing.

There’s also a slight irony highlighted to me in writing a blog post on this subject of being a blog post writer; maybe I should record a video, too!

Figuring out why you want this role!

If there is one piece of advice I can give anyone with their heart and mind set on a specific next role, it’s to figure out why you want that particular role. And be completely honest with yourself, whether it’s money, getting away from a shitty boss or colleagues, furthering your experiences, or even just because it’s an interim step to something else.

Most people I know who work in the technology field at some point want or have wanted to work in the role generally known as “Technical Marketing”, myself included (and currently I do).

Most of the time, it stems from meeting someone in the role, either at a conference or watching a presentation delivered by someone in this role. Next, you learn about the supposed glitz and glamour of the role, the travel to various conference rooms across the area, maybe even the globe, working with your latest favourite technologies, and seemingly endless knowledge about a particular area.

You may pick up a slight sense of sarcasm in that last paragraph, and that’s because I remind myself of the rose-tinted glasses I wore looking on at those who worked in vendor-land as I yearned to make a move across. Unfortunately, I have to dispel a myth for you now. It’s not all first-class flights, full-expense trips, and parties across the globe; sometimes, you must create PowerPoint presentations!

I already do the tech marketing role anyway, kind of!

Continue reading So you want to work in Technical Marketing or other associated roles?

Cilium Event Types Header

Understanding cilium_event_type when using Cilium & Hubble

The Issue

In a platform that’s deployed with Cilium, when using Hubble either to view the full JSON output or to configure which events are captured using the allowlist or denylist you may have seen a field called event_type which uses an integer.

Below is an example allow list using “event_type”, to define which flows to be captured. When I first saw this, I was confused; where do these numbers come from? How do I map this back to a friendly name that I understand?;

allowlist:
- '{"source_pod":["kube-system/"],"event_type":[{"type":1}]}'
- '{"destination_pod":["kube-system/"],"event_type":[{"type":1}]}'

Example Hubble Dynamic Exporter configuration;

hubble:
  export:
    dynamic:
      enabled: true
      config:
        enabled: true
        content:
        - name: "test001"
          filePath: "/var/run/cilium/hubble/test001.log"
          fieldMask: []
          includeFilters: []
          excludeFilters: []
          end: "2023-10-09T23:59:59-07:00"
        - name: "test002"
          filePath: "/var/run/cilium/hubble/test002.log"
          fieldMask: ["source.namespace", "source.pod_name", "destination.namespace", "destination.pod_name", "verdict"]
          includeFilters:
          - source_pod: ["default/"]
            event_type:
            - type: 1
          - destination_pod: ["frontend/webserver-975996d4c-7hhgt"]

and finally, a Hubble flow in full JSON output, with the event_type showing towards the end of the output;

{
  "flow": {
    "time": "2024-07-08T10:09:24.173232166Z",
    "uuid": "755b0203-d456-452d-b399-4fa136cdb4fd",
    "verdict": "FORWARDED",
    "ethernet": {
      "source": "06:29:73:4e:0a:c5",
      "destination": "26:50:d8:4a:94:d2"
    },
    "IP": {
      "source": "10.0.2.163",
      "destination": "130.211.198.204",
      "ipVersion": "IPv4"
    },
    "l4": {
      "TCP": {
        "source_port": 37736,
        "destination_port": 443,
        "flags": {
          "PSH": true,
          "ACK": true
        }
      }
    },
    "source": {
      "ID": 2045,
      "identity": 14398,
      "namespace": "endor",
      "labels": [
        "k8s:app.kubernetes.io/name=tiefighter"
      ],
      "pod_name": "tiefighter-6b56bdc869-2t6wn",
      "workloads": [
        {
          "name": "tiefighter",
          "kind": "Deployment"
        }
      ]
    },
    "destination": {
      "identity": 16777217,
      "labels": [
        "cidr:130.211.198.204/32",
        "reserved:world"
      ]
    },
    "Type": "L3_L4",
    "node_name": "kind-worker",
    "destination_names": [
      "disney.com"
    ],
    "event_type": {
      "type": 4,
      "sub_type": 3
    },
    "traffic_direction": "EGRESS",
    "trace_observation_point": "TO_STACK",
    "is_reply": false,
    "Summary": "TCP Flags: ACK, PSH"
  },
  "node_name": "kind-worker",
  "time": "2024-07-08T10:09:24.173232166Z"
}
The Explanation

Cilium Event types are defined in this Go package. The first line iota == 0 then increments by one for each type, so drop =1, debug =2, etc.

const (
	// 0-128 are reserved for BPF datapath events
	MessageTypeUnspec = iota

	// MessageTypeDrop is a BPF datapath notification carrying a DropNotify
	// which corresponds to drop_notify defined in bpf/lib/drop.h
	MessageTypeDrop

	// MessageTypeDebug is a BPF datapath notification carrying a DebugMsg
	// which corresponds to debug_msg defined in bpf/lib/dbg.h
	MessageTypeDebug

	// MessageTypeCapture is a BPF datapath notification carrying a DebugCapture
	// which corresponds to debug_capture_msg defined in bpf/lib/dbg.h
	MessageTypeCapture

	// MessageTypeTrace is a BPF datapath notification carrying a TraceNotify
	// which corresponds to trace_notify defined in bpf/lib/trace.h
	MessageTypeTrace

	// MessageTypePolicyVerdict is a BPF datapath notification carrying a PolicyVerdictNotify
	// which corresponds to policy_verdict_notify defined in bpf/lib/policy_log.h
	MessageTypePolicyVerdict

	// MessageTypeRecCapture is a BPF datapath notification carrying a RecorderCapture
	// which corresponds to capture_msg defined in bpf/lib/pcap.h
	MessageTypeRecCapture

	// MessageTypeTraceSock is a BPF datapath notification carrying a TraceNotifySock
	// which corresponds to trace_sock_notify defined in bpf/lib/trace_sock.h
	MessageTypeTraceSock

	// 129-255 are reserved for agent level events

	// MessageTypeAccessLog contains a pkg/proxy/accesslog.LogRecord
	MessageTypeAccessLog = 129

	// MessageTypeAgent is an agent notification carrying a AgentNotify
	MessageTypeAgent = 130
)

const (
	MessageTypeNameDrop          = "drop"
	MessageTypeNameDebug         = "debug"
	MessageTypeNameCapture       = "capture"
	MessageTypeNameTrace         = "trace"
	MessageTypeNameL7            = "l7"
	MessageTypeNameAgent         = "agent"
	MessageTypeNamePolicyVerdict = "policy-verdict"
	MessageTypeNameRecCapture    = "recorder"
	MessageTypeNameTraceSock     = "trace-sock"
)

Therefore, in the above JSON output (last example), event type 4 is defined as trace, this particular event type also has a sub_typeas you can see here in the Hubble CLI, help output. You can see the definitions in the Go package here.

  -t, --type filter                         Filter by event types TYPE[:SUBTYPE]. Available types and subtypes:
                                            TYPE             SUBTYPE
                                            capture          n/a
                                            drop             n/a
                                            l7               n/a
                                            policy-verdict   n/a
                                            trace            from-endpoint
                                                             from-host
                                                             from-network
                                                             from-overlay
                                                             from-proxy
                                                             from-stack
                                                             to-endpoint
                                                             to-host
                                                             to-network
                                                             to-overlay
                                                             to-proxy
                                                             to-stack
                                            trace-sock       n/a

I hope this helps!

Regards

Dean Lewis

Kubernetes

KubeVirt – Creating Data Volume fails – DataVolume.storage spec is missing accessMode and volumeMode

The Issue

I had a freshly deployed Red Hat OpenShift cluster on which I had set up OpenShift Virtualization (KubeVirt). When I tried to create my first data volume so I could start creating VMs, nothing happened. The ISO file upload would fail.

Running oc describe dv {name} -n {namespace} showed the below event/error.

DataVolume.storage spec is missing accessMode and volumeMode, cannot get access mode from StorageProfile managed-nfs-storage
The Cause

This is caused by default created StorageProfile used by KubeVirt, which will be created from your default StorageClass, not setting the correct values, as per this git issue comment.

Previously virtctl (CLI for KubeVirt) used ReadWriteOnce if accessMode was not specified. But now, as we want to use possibilities provided by StorageProfiles, it does not set accessMode.
The Fix

Run the following command to update your StorageProfile with the correct settings, pay attention to whether you needReadWriteOnce or ReadWriteMany.

kubectl patch StorageProfile {StorageProfileName} \
  --type merge -p \
  '{"spec": {"claimPropertySets": [{"accessModes": ["ReadWriteOnce"]}]}}' 

Regards

Dean Lewis

Cilium Hubble CLI - Header Image

Cilium Hubble CLI – Configure Auto Completion

One of the little nits I have is when I use the terminal, and as I start typing a command if I press the Tab key, autocomplete doesn’t work. It feels like it should be the default out of the box.

You can configure the Hubble CLI for Cilium, but it’s not documented in the docs.cilium.io pages yet, so I thought I’d throw up a quick post adding it here!

This command pushes the auto-complete config into my zsh config on macOS.

hubble completion zsh > $(brew --prefix)/share/zsh/site-functions/_hubble

For other platforms, you can see the examples provided for Cilium Agent, and apply the logic to your own environment.

Cilium Hubble CLI - Autocompletion

Regards

Dean Lewis