Tag Archives: Cilium

Kubernetes Header Image

Highlight Kubernetes Labels in your Terminal with AWK

A quick tip and bit of code: if you’re outputting a lot of Kubernetes metadata using the --show-labels command, it can feel like looking for a needle in a haystack. The snippet below colorizes key label outputs to make them stand out.

The Code Snippet

When working with Kubernetes, it can be helpful to visually scan for certain node labels—such as service.cilium.io/node=... or custom readiness flags like ingress-ready=true. Using a simple awk script, we can colorize these labels directly in our terminal output. This script uses ANSI escape codes to wrap matched text in color and awk’s gsub() function to apply substitutions line by line. It’s a lightweight and effective way to highlight key data points in otherwise dense CLI output.

kubectl get ciliumnodes --show-labels | awk '
BEGIN {
  color_start = "\033[1;36m"; # cyan
  color_end = "\033[0m";
}
{
  gsub(/service\.cilium\.io\/node=[^, ]+/, color_start "&" color_end);
  gsub(/ingress-ready=true/, color_start "&" color_end);
  print
}'

Screenshot Example


Screenshot showing the use of an awk command to color-highlight the ingress-ready=true label in red within kubectl get ciliumnodes --show-labels output in a Kubernetes terminal session.

Breakdown of the Code

We pipe the output of the kubectl command to awk. The BEGIN block sets up the ANSI color codes used for matching patterns.

  • \033[1;36m is an ANSI escape code that starts cyan-colored text.
  • \033[0m resets the text color back to normal.

gsub(...)

These two lines apply substitutions to each input line:

  • gsub() is a global substitution function that replaces all matches in the line.
    • service\.cilium\.io\/node=[^, ]+ matches a full key-value pair like service.cilium.io/node=mynode
    • [^, ]+ grabs the node value until the next comma or space
    • ingress-ready=true matches the exact label string
    • & refers to the entire matched string, which we wrap in color codes

print

This prints the modified line after substitutions are applied.

Customize the Highlight Color

You can change \033[1;36m to another color code:

  • Red: \033[1;31m
  • Green: \033[1;32m
  • Yellow: \033[1;33m
  • Blue: \033[1;34m
  • Magenta: \033[1;35m

A Final Note on sub() vs gsub()

  • sub() replaces only the first occurrence of the regex in the line
  • gsub() replaces all occurrences of the regex in the line

Regards


Bluesky Icon
Follow me on Bluesky

Dean Lewis

Cilium Event Types Header

Understanding cilium_event_type when using Cilium & Hubble

The Issue

In a platform that’s deployed with Cilium, when using Hubble either to view the full JSON output or to configure which events are captured using the allowlist or denylist you may have seen a field called event_type which uses an integer.

Below is an example allow list using “event_type”, to define which flows to be captured. When I first saw this, I was confused; where do these numbers come from? How do I map this back to a friendly name that I understand?;

allowlist:
- '{"source_pod":["kube-system/"],"event_type":[{"type":1}]}'
- '{"destination_pod":["kube-system/"],"event_type":[{"type":1}]}'

Example Hubble Dynamic Exporter configuration;

hubble:
  export:
    dynamic:
      enabled: true
      config:
        enabled: true
        content:
        - name: "test001"
          filePath: "/var/run/cilium/hubble/test001.log"
          fieldMask: []
          includeFilters: []
          excludeFilters: []
          end: "2023-10-09T23:59:59-07:00"
        - name: "test002"
          filePath: "/var/run/cilium/hubble/test002.log"
          fieldMask: ["source.namespace", "source.pod_name", "destination.namespace", "destination.pod_name", "verdict"]
          includeFilters:
          - source_pod: ["default/"]
            event_type:
            - type: 1
          - destination_pod: ["frontend/webserver-975996d4c-7hhgt"]

and finally, a Hubble flow in full JSON output, with the event_type showing towards the end of the output;

{
  "flow": {
    "time": "2024-07-08T10:09:24.173232166Z",
    "uuid": "755b0203-d456-452d-b399-4fa136cdb4fd",
    "verdict": "FORWARDED",
    "ethernet": {
      "source": "06:29:73:4e:0a:c5",
      "destination": "26:50:d8:4a:94:d2"
    },
    "IP": {
      "source": "10.0.2.163",
      "destination": "130.211.198.204",
      "ipVersion": "IPv4"
    },
    "l4": {
      "TCP": {
        "source_port": 37736,
        "destination_port": 443,
        "flags": {
          "PSH": true,
          "ACK": true
        }
      }
    },
    "source": {
      "ID": 2045,
      "identity": 14398,
      "namespace": "endor",
      "labels": [
        "k8s:app.kubernetes.io/name=tiefighter"
      ],
      "pod_name": "tiefighter-6b56bdc869-2t6wn",
      "workloads": [
        {
          "name": "tiefighter",
          "kind": "Deployment"
        }
      ]
    },
    "destination": {
      "identity": 16777217,
      "labels": [
        "cidr:130.211.198.204/32",
        "reserved:world"
      ]
    },
    "Type": "L3_L4",
    "node_name": "kind-worker",
    "destination_names": [
      "disney.com"
    ],
    "event_type": {
      "type": 4,
      "sub_type": 3
    },
    "traffic_direction": "EGRESS",
    "trace_observation_point": "TO_STACK",
    "is_reply": false,
    "Summary": "TCP Flags: ACK, PSH"
  },
  "node_name": "kind-worker",
  "time": "2024-07-08T10:09:24.173232166Z"
}
The Explanation

Cilium Event types are defined in this Go package. The first line iota == 0 then increments by one for each type, so drop =1, debug =2, etc.

const (
	// 0-128 are reserved for BPF datapath events
	MessageTypeUnspec = iota

	// MessageTypeDrop is a BPF datapath notification carrying a DropNotify
	// which corresponds to drop_notify defined in bpf/lib/drop.h
	MessageTypeDrop

	// MessageTypeDebug is a BPF datapath notification carrying a DebugMsg
	// which corresponds to debug_msg defined in bpf/lib/dbg.h
	MessageTypeDebug

	// MessageTypeCapture is a BPF datapath notification carrying a DebugCapture
	// which corresponds to debug_capture_msg defined in bpf/lib/dbg.h
	MessageTypeCapture

	// MessageTypeTrace is a BPF datapath notification carrying a TraceNotify
	// which corresponds to trace_notify defined in bpf/lib/trace.h
	MessageTypeTrace

	// MessageTypePolicyVerdict is a BPF datapath notification carrying a PolicyVerdictNotify
	// which corresponds to policy_verdict_notify defined in bpf/lib/policy_log.h
	MessageTypePolicyVerdict

	// MessageTypeRecCapture is a BPF datapath notification carrying a RecorderCapture
	// which corresponds to capture_msg defined in bpf/lib/pcap.h
	MessageTypeRecCapture

	// MessageTypeTraceSock is a BPF datapath notification carrying a TraceNotifySock
	// which corresponds to trace_sock_notify defined in bpf/lib/trace_sock.h
	MessageTypeTraceSock

	// 129-255 are reserved for agent level events

	// MessageTypeAccessLog contains a pkg/proxy/accesslog.LogRecord
	MessageTypeAccessLog = 129

	// MessageTypeAgent is an agent notification carrying a AgentNotify
	MessageTypeAgent = 130
)

const (
	MessageTypeNameDrop          = "drop"
	MessageTypeNameDebug         = "debug"
	MessageTypeNameCapture       = "capture"
	MessageTypeNameTrace         = "trace"
	MessageTypeNameL7            = "l7"
	MessageTypeNameAgent         = "agent"
	MessageTypeNamePolicyVerdict = "policy-verdict"
	MessageTypeNameRecCapture    = "recorder"
	MessageTypeNameTraceSock     = "trace-sock"
)

Therefore, in the above JSON output (last example), event type 4 is defined as trace, this particular event type also has a sub_typeas you can see here in the Hubble CLI, help output. You can see the definitions in the Go package here.

  -t, --type filter                         Filter by event types TYPE[:SUBTYPE]. Available types and subtypes:
                                            TYPE             SUBTYPE
                                            capture          n/a
                                            drop             n/a
                                            l7               n/a
                                            policy-verdict   n/a
                                            trace            from-endpoint
                                                             from-host
                                                             from-network
                                                             from-overlay
                                                             from-proxy
                                                             from-stack
                                                             to-endpoint
                                                             to-host
                                                             to-network
                                                             to-overlay
                                                             to-proxy
                                                             to-stack
                                            trace-sock       n/a

I hope this helps!

Regards

Dean Lewis

Cilium Hubble CLI - Header Image

Cilium Hubble CLI – Configure Auto Completion

One of the little nits I have is when I use the terminal, and as I start typing a command if I press the Tab key, autocomplete doesn’t work. It feels like it should be the default out of the box.

You can configure the Hubble CLI for Cilium, but it’s not documented in the docs.cilium.io pages yet, so I thought I’d throw up a quick post adding it here!

This command pushes the auto-complete config into my zsh config on macOS.

hubble completion zsh > $(brew --prefix)/share/zsh/site-functions/_hubble

For other platforms, you can see the examples provided for Cilium Agent, and apply the logic to your own environment.

Cilium Hubble CLI - Autocompletion

Regards

Dean Lewis

Cilium Hubble CLI - Header Image

Cilium Hubble CLI – Using a local configuration file

Did you know that the Cilium Hubble CLI supports using a configuration file?

Below is an example command where Isovalent Enterprise for Cilium is deployed and Hubble RBAC is configured. Therefore, I must provide additional details such as the server location and certificates to authenticate using the CLI. The steps in this blog post also work with Cilium OSS, which is especially handy when setting allow and deny lists to prune the information returned.

This can become cumbersome for every command you want to run.

❯ hubble observe \
--server tls://localhost:4245 \
--tls-ca-cert-files ca-cert.pem \
--tls-server-name 'cli.hubble-relay.cilium.io' \
--namespace kube-system
Mar 20 13:09:38.061: tenant-jobs/resumes-58c6678bc8-5nkcg:36459 (ID:88195) -> kube-system/coredns-77fcb74c4c-wfw4f:53 (ID:102385) policy-verdict:L3-L4 EGRESS ALLOWED (UDP)
Mar 20 13:09:38.061: tenant-jobs/resumes-58c6678bc8-5nkcg:36459 (ID:88195) -> kube-system/coredns-77fcb74c4c-wfw4f:53 (ID:102385) to-proxy FORWARDED (UDP)
Mar 20 13:09:38.062: tenant-jobs/resumes-58c6678bc8-5nkcg (ID:88195) <> kube-system/coredns-77fcb74c4c-wfw4f:53 (ID:102385) post-xlate-fwd TRANSLATED (UDP)
Mar 20 13:09:38.062: tenant-jobs/resumes-58c6678bc8-5nkcg:36459 (ID:88195) -> kube-system/coredns-77fcb74c4c-wfw4f:53 (ID:102385) dns-request proxy FORWARDED (DNS Query coreapi.tenant-jobs.svc.cluster.local. A)

Below you can see the various configuration options that the Hubble CLI supports. The above example is using flags as part of the command.

hubble config -h
Config allows to modify or view the hubble configuration. Global hubble options
can be set via flags, environment variables or a configuration file. The
following precedence order is used:

1. Flag
2. Environment variable
3. Configuration file
4. Default value

The "config view" subcommand provides a merged view of the configuration. The
"config set" and "config reset" subcommand modify values in the configuration
file.

Environment variable names start with HUBBLE_ followed by the flag name
capitalized where eventual dashes ('-') are replaced by underscores ('_').
For example, the environment variable that corresponds to the "--server" flag
is HUBBLE_SERVER. The environment variable for "--tls-allow-insecure" is
HUBBLE_TLS_ALLOW_INSECURE and so on.

Usage:
  hubble config [flags]
  hubble config [command]

Available Commands:
  get         Get an individual value in the hubble config file
  reset       Reset all or an individual value in the hubble config file
  set         Set an individual value in the hubble config file
  view        Display merged configuration settings

Using the below commands, I can set the flags as values in the configuration file, for any CLI flag, the set value will be prepended with HUBBLE_+ the flag name.

❯ hubble config set HUBBLE_SERVER tls://localhost:4245
unknown key: HUBBLE_SERVER
❯ hubble config set server tls://localhost:4245
❯ hubble config set tls-ca-cert-files ca-cert.pem

❯ hubble config set tls-server-name 'cli.hubble-relay.cilium.io'

Now we can use the Hubble CLI without the additional flags.

❯ hubble observe -n tenant-jobs
Mar 20 13:13:30.004: tenant-jobs/coreapi-6748664db6-rmr2j:42935 (ID:111121) <- kube-system/coredns-77fcb74c4c-wfw4f:53 (ID:102385) to-endpoint FORWARDED (UDP)
Mar 20 13:13:30.496: tenant-jobs/crawler-6dbf4f8b5d-vr7gr:47804 (ID:71705) -> tenant-jobs/loader-68544b8b87-zrxwt:50051 (ID:115137) http-request FORWARDED (HTTP/2 POST http://loader:50051/loader.Loader/LoadCv)
Mar 20 13:13:30.505: tenant-jobs/crawler-6dbf4f8b5d-vr7gr:47804 (ID:71705) <- tenant-jobs/loader-68544b8b87-zrxwt:50051 (ID:115137) http-response FORWARDED (HTTP/2 200 9ms (POST http://loader:50051/loader.Loader/LoadCv))

We can validate the configuration in use by running the below command, which also confirms the location of the config file itself, which you can edit directly.

❯ hubble config view
allowlist: []
client-id: ""
client-secret: ""
config: /Users/veducate/Library/Application Support/hubble/config.yaml
debug: false
denylist: []
grant-type: auto
issuer: ""
issuer-ca: ""
refresh: false
scopes: []
server: tls://localhost:4245
timeout: 5s
tls: false
tls-allow-insecure: false
tls-ca-cert-files:
- ca-cert.pem
tls-client-cert-file: ""
tls-client-key-file: ""
tls-server-name: cli.hubble-relay.cilium.io
token-file

Regards

Dean Lewis

Red Hat OpenShift - Cilium CNI Migration - Header

How to migrate from Red Hat OpenShiftSDN/OVN-Kubernetes to Cilium

Recently, I’m seeing more and more queries about migrating to Cilium within an existing Red Hat OpenShift cluster, due to Cilium’s advanced networking capabilities, robust security features, and enhanced observability out-of-the-box. This increase of interest is also boosted by the fact that Cilium became the first Kubernetes CNI to graduate in the CNCF Landscape.

In this blog post, we’ll cover the step-by-step process of migrating from the traditional OpenShiftSDN (default CNI pre-4.12) or OVN-Kubernetes (default CNI from 4.12) to Cilium, exploring the advantages and considerations along the way.

If you need to understand more about the default CNI options in Red Hat OpenShift first, then I highly recommend this blog post, as pre-reading before going through this walkthrough.

Cilium Overview

For those of you who have not heard of Cilium, or maybe just the name and know there’s a buzz about it. In short Cilium, is a cloud native networking solution to provide security, networking and observability at a software level.

The reason why the buzz is so huge is due to being implemented using eBPF, a new way of interacting and programming with the kernel layer of the OS. This implementation opens a whole new world of options.

I’ll leave you with these two short videos from Thomas Graf, co-founder of Isovalent, the creators of Cilium.

Does Red Hat support this migration?

Cilium has achieved the Red Hat OpenShift Container Network Interface (CNI) certification by completing the operator certification and passing end-to-end testing. Red Hat will support Cilium installed and running in a Red Hat OpenShift cluster, and collaborate as needed with the ecosystem partner to troubleshoot any issues, as per their third-party software support statements. This would be a great reason to look at Isovalent Enterprise for Cilium, rather than using Cilium OSS, to get support from both vendors.

However, when it comes to performing a CNI migration for an active existing OpenShift cluster, Red Hat provides no guidance, unless it’s migrating from OpenShiftSDN to OVN-Kubernetes.

This means CNI migration to a third party CNI in an existing running Red Hat OpenShift Cluster is a grey area.

I’d recommend speaking to your Red Hat account team before performing any migration like this in your production environments. I have known large customers to take on this work/burden/supportability themselves and be successful.

Follow along with this video!

If you prefer watching a video or seeing things live and following along, like I do at times, then I’ve got you covered with the below video that covers the content from this blog post.

Pre-requisites and OpenShift Cluster configuration
As per the above, understand this process in detail, and if you follow it, you do so at your own risk.

For this walkthrough, I’ve deployed a OpenShift 4.13 cluster with OVN-Kubernetes, with a sample application (see below). You can see these posts I’ve written for deployments of OpenShift, or follow the official documentation.

Here is a copy of my install-config.yaml file. It was generated using the openShift-install create install-config wizard. Then I ran the openshift-install create cluster command. Continue reading How to migrate from Red Hat OpenShiftSDN/OVN-Kubernetes to Cilium