How to configure Red Hat OpenShift to forward logs to VMware vRealize Log Insight Cloud

In this blog post we will cover how to configure Red Hat OpenShift to forward logs from the ClusterLogging instance to an external 3rd party system, in this case, VMware vRealize Log Insight Cloud.

Architecture

The Openshift Cluster Logging will have to be configured for accessing the logs and forwarding to 3rd party logging tools. You can deploy the full suite;

  • Visualization: Kibana
  • Collection: FluentD
  • Log Store: Elasticsearch
  • Curation: Curator

However, to ship the logs to an external system, you will only need to configure the FluentD service.

To forward the logs from the internal trusted services, we will use the new Log Forwarding API, which is GA in OpenShift 4.6 and later (it was a tech preview in earlier releases, and the configuration YAMLs are slightly different, so read the relevant documentation version).

This setup will provide us the architecture below. We will deploy the trusted namespace “OpenShift-Logging” and use the Operator to provide a Log Forwarding API configuration which sends the logs to a 3rd party service.

For vRealize Log Insight Cloud, we will run a standalone FluentD instance inside of the cluster to forward to the cloud service.

Openshift cluster logging to vmware log insight architecture

The log types are one of the following:

  • application. Container logs generated by user applications running in the cluster, except infrastructure container applications.
  • infrastructure. Container logs from pods that run in the openshift*, kube*, or default projects and journal logs sourced from node file system.
  • audit. Logs generated by the node audit system (auditd) and the audit logs from the Kubernetes API server and the OpenShift API server.
Prerequisites
  • VMware vRealize Log Insight Cloud instance setup with Administrator access.
  • Red Hat OpenShift Cluster deployed
    • with outbound connectivity for containers
  • Download this Github Repository for the configuration files
git clone https://github.com/saintdle/openshift_vrealize_loginsight_cloud.git
Deploy the standalone FluentD instance to forward logs to vRealize Log Insight Cloud

As per the above diagram, we’ll create a namespace and deploy a FluentD service inside the cluster, this will handle the logs forwarded from the OpenShift Logging instance and send the to the Log Insight Cloud instance.

Creating a vRealize Log Insight Cloud API Key

First, we will create an API key for sending data to our cloud instance.

  1. Expand Configuration on the left-hand navigation pane
  2. Select “API Keys”
  3. Click the “New API Key” button

Give your API key a suitable name and click Create.

Continue reading How to configure Red Hat OpenShift to forward logs to VMware vRealize Log Insight Cloud

OpenShift – Cluster-Monitoring-Operator Pod Error – cannot verify user is non-root

The issue

After building a brand new OpenShift 4.6.9 cluster, I noticed one of the pods was not running correctly

oc get pods -n openshift-monitoring
.....
NAME READY   STATUS                       RESTARTS   AGE
cluster-monitoring-operator-f85f7bcb5-84jw5 1/2 CreateContainerConfigError 0 112m

Upon inspection of the pod;

oc describe pod cluster-monitoring-operator-XXX -n openshift-
monitoring

I could see the following error message;

Error: container has runAsNonRoot and image has non-numeric user
(nobody), cannot verify user is non-root

The Cause

There is a Red Hat article about this, but it is gated. The reason is cluster-monitoring-operator gets wrongly the non-root SCC assigned.

The Fix

Currently there is no permanent provided fix from Red Hat, but you can track this bug.

However the workaround is to simply delete the pod. This should recreate and load with the correct permissions.

Regards

TAM Lab 079 – Using vRA Cloud to operate a Multi-Cloud Environment

Myself and Katherine Skilling (LinkedIn, Twitter) recorded a session for TAM Lab and VMUG Events.

In the below session, we cover how to use vRealize Automation Cloud (or vRA 8.x for on-prem) to operate your Multi-Cloud environment.

So what does this actually mean?

We cover how to use vRealize Automation to deploy and consume your public cloud provider of choice. This is a demo heavy recording and we cover the following;

  • vRealize Automation Core Components
  • Image Mapping
  • Flavour Mapping
  • Machine Flavours
  • Using the Cloud Template canvas in design and code view (Blueprints)
  • Deploying your first virtual machine
  • Deploying your virtual machine to different public cloud providers
  • Creating inputs for configuration
  • Advanced configuration with CloudConfig
  • Basic Troubleshooting

Regards

Folding@Home Header

Deploying the VMware Appliance for [email protected] using Terraform

To simplify the deployment of [email protected] appliances to vSphere environments, I have wrote a set of Terraform configuration files (script).

You will need two packages downloaded to your jump host.

And either download locally the VMware [email protected] Appliance, or host it at remote location.

Use the git software to download my Terraform Git folder which contains the folder called Deploy-FAH.

git clone https://github.com/saintdle/Terraform.git

Move into the “Deploy-FAH” folder, and edit the terraform.tfvars file as needed;

cd Deploy-FAH
vi terraform.tfvars

Below is an example;

// Name of the vSphere server. E.g "vcsa.vmware.local"
vsphere_server = "vcenter.veducate.local"

// User on the vSphere server. E.g "administrator@vsphere.local"
vsphere_user = "administrator@vsphere.local"

// Password of the user on the vSphere server. E.g "password"
vsphere_password = "Password1234!"

// Name of the vSphere data center. E.g "datacenter"
vsphere_datacenter = "Datacenter"

// Name of the vSphere cluster. E.g "Cluster"
vsphere_cluster = "Cluster"

// Name or IP of the vSphere host in the cluster to deploy your VM to. E.g "esxi-01" or "192.168.1.20"
vsphere_host = "10.10.2.4"

// Name of the vSphere data store to use for the VMs. E.g "VSAN"
vsphere_datastore = "Datastore"

// Network to connect virtual machine
vm_network = "Freale_NW1"

// Number of instances to deploy
instance_count = 2

// VM Machine Name (an index will be appended i.e FAH-1, FAH-2,)
vm_name = "dean-test"

// Number of CPUs to set on deployed Virtual Machines
num_cpu = 2

// Memory to set on deployed Virtual Machines (in MB)
memory = 4096

// Name of vSphere Resouce Pool to be created. E.g "FAH-VMs"
vsphere_resource_pool = "dean-test"

// Name of VM folder to be created. E.g "FAH-VMs"
vsphere_vm_folder = "dean-test"

// Location of OVA file if using a local location - if using remote location, leave this as null
local_ovf_path = "/home/dean/Deploy-FAH-3/VMware-Appliance-FaH_1.0.4.ova"

// Location of OVA file if using a remote location - if using local location, leave this as null
remote_ovf_path =

// Enable SSH in FAH Appliance (True or False)
ssh_enable = "True"

// FAH appliance root password
root_password = "VMware1!"

// FAH Username you wish to be associated with in the statistics tables
fah_user = ""

// FAH Team you wish to be associated with in the statistics tables
fah_team = "52737"

// FAH Passkey to verify your user in the statistical tables (this is optional from FAH project)
fah_passkey = "unique_id"

That’s it, no more changes needed, it’s as simple as running the following to deploy your appliances;

#This will download the terraform providers as needed

terraform init

#This will show you the planned changes and make sure they are possible

terraform plan

#This will run the configuration to run the deployment

terraform apply

You can use the latest version of Terraform, version 0.13.5 as of the publishing of this post.

Quick notes

This terraform configuration uses some advance configuration in the folder “FAH-Appliance”, under the main.tf file. Here it reads the “remote_ovf_path” variable, and acts based on if it is null or not. If there is a variable set, then it runs the command to deploy from a remote location. If variable is null, then it looks to the “local_ovf_path”, and processes this to deploy an OVF/OVA from the local location.

  dynamic "ovf_deploy" {
  for_each = "${var.local_ovf_path}" != "" || "${var.remote_ovf_path}" != "" ? [0] : []
  content {
  // Path to local or remote ovf/ova file
  local_ovf_path = "${var.local_ovf_path}" != "" ? "${var.local_ovf_path}" : null
  remote_ovf_url = "${var.remote_ovf_path}" != "" ? "${var.remote_ovf_path}" : null
   disk_provisioning    = "thin"
   ovf_network_map = {
        "VM Network" = data.vsphere_network.network.id
    }
   }
  }

Thanks to Grant Orchard from HashiCorp helping me with this part of the config.

Interesting in where you can take this further, check out this post from Robert Jenson, using VMware CodeStream for an Infrastructure as Code deployment using GitHub as a source repository, and terraform for the deployment.

Regards

VMware AWS Header

How to delete vCenter Roles in VMC

The Issue

Whilst testing in VMC a PowerCLI script to create some vCenter roles, I noticed in the UI, then I deleted them, they remained, even though I was using the CloudAdmin@vmc.local account.

I also tried to delete them using PowerCLI and received the error message;

Remove-VIRole : 07/11/2020 09:00:42 Remove-VIRole Permission to perform this operation was denied. Required privilege 'VApp.PullFromUrls' on managed object with id 'Folder-group-d1'.
At line:1 char:1
+ Remove-VIRole OpenShift-Install
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Remove-VIRole], NoPermission
+ FullyQualifiedErrorId : Client20_InventoryServiceImpl_RemoveRole_VIError,VMware.VimAutomation.ViCore.Cmdlets.Commands.PermissionManagement.RemoveVIRole

You should not create roles with permissions higher than that of the CloudAdmin Account. You can find further information of these permissions here.

If you do this, the only fix is to log a support call with VMware to resolve.

The Fix

VMware have a KB for this issue and how to delete the vCenter roles.

To resolve you use the vCenter Managed Objects Browser (MOB).

Note: When using the MOB to make changes, users will not be prompted for confirmation before making any changes, including removing roles. A custom role can not have privileges higher than the CloudGlobalAdmin role.

First to view all your existing roles in your browser go to;

  • ¬†https://{VMC_VC_FQDN}/mob/?moid=AuthorizationManager&doPath=roleList

This will list all roles, and note the roleId for the role you want to remove.

To remove a role: Continue reading How to delete vCenter Roles in VMC