vRealize Log Insight 8.6 brings the ability to build a hybrid log management platform, utilizing the functionality of an on-premises deployment of vRLI and vRLI Cloud.
From the release notes, in this blog post we’ll be looking at how to configure the following:
- Simplify Log Archival with Non-Indexed Partitions: Use vRealize Log Insight Cloud to archive logs to meet your long-term retention requirements. vRealize Log Insight Cloud provides a no-limit logging solution at a low cost and eliminates any storage management overheads of the past. This enables easy accessibility to archived logs through on-demand queries.
For this, you will need access to a vRealize Log Insight Cloud Instance, with a cloud proxy deployed to your environment that can be accessed by the on-premises vRealize Log Insight platform.
The expectation is that you would forward you vRealize Log Insight on-premises logs to the vRealize Log Insight Cloud instance storing them only in a Non-Indexed Partition (discussed below). As your on-premises deployment act as your easy to analyse near time (within 30 days) copy of your logs. In this blog post I also explore the configuration and use of Index Partitions which essentially offers that near time usability and analysing of logs as well.
The high-level steps for the configuration discussed in this blog post are:
- Send infrastructure or application logs to your on-premises vRealize Log Insight deployment
- Setup the cloud proxy (if not already done)
- Setup log forwarding from the on-premises Log Insight instance
- In vRealize Log Insight Cloud, configure Non-Index Partition to receive the forwarded logs
What are Log Partitions?
Log Partitions are a feature that allows you to ingest logs based on user-defined filters. This feature is available as a paid subscription (or Trial).
There are two types of log Log Partitions:
- Indexed Partitions
- Stores logs for up to 30 days
- Billed only for volume of logs ingested into the partition
- Search and analyse logs in this partition without additional costs
- Non-Indexed Partitions
- Stores logs for up to 7 years
- Billed for the volume of logs ingested into the partition, and for searching the logs.
- If you need to query logs frequently, you can move logs to a recall partition for 30 days.
- No additional cost for searching and analysing logs in the recall partition
Logs that do not match a query criteria in any of the configured partitions, will be stored in the Default Indexed Partition. This is read only and stores logs for 30 days.
Note: - Alerts and dashboard widgets are not operational in non-indexed partitions. - Log partitions store logs ingested in the last 24 hours only. - You can create a maximum of 10 log partitions in an organization.
In my Log Insight environment, I have setup the FluentD configuration to forward the Tanzu Kubernetes Grid logs from two clusters to vRealize Log Insight (on-premises deployment).
You can find the configuration settings for this within vRealize Log Insight, under the Sources Tab > Containers > Tanzu Kubernetes Grid.