vRLI Header

Using vRealize Log Insight Cloud to archive on-premise Log Insight Data

vRealize Log Insight 8.6 brings the ability to build a hybrid log management platform, utilizing the functionality of an on-premises deployment of vRLI and vRLI Cloud.

From the release notes, in this blog post we’ll be looking at how to configure the following:

  • Simplify Log Archival with Non-Indexed Partitions: Use vRealize Log Insight Cloud to archive logs to meet your long-term retention requirements. vRealize Log Insight Cloud provides a no-limit logging solution at a low cost and eliminates any storage management overheads of the past. This enables easy accessibility to archived logs through on-demand queries.

For this, you will need access to a vRealize Log Insight Cloud Instance, with a cloud proxy deployed to your environment that can be accessed by the on-premises vRealize Log Insight platform.

The expectation is that you would forward you vRealize Log Insight on-premises logs to the vRealize Log Insight Cloud instance storing them only in a Non-Indexed Partition (discussed below). As your on-premises deployment act as your easy to analyse near time (within 30 days) copy of your logs. 

In this blog post I also explore the configuration and use of Index Partitions which essentially offers that near time usability and analysing of logs as well.

The high-level steps for the configuration discussed in this blog post are:

  • Send infrastructure or application logs to your on-premises vRealize Log Insight deployment
  • Setup the cloud proxy (if not already done)
  • Setup log forwarding from the on-premises Log Insight instance
  • In vRealize Log Insight Cloud, configure Non-Index Partition to receive the forwarded logs
What are Log Partitions?

Log Partitions are a feature that allows you to ingest logs based on user-defined filters. This feature is available as a paid subscription (or Trial).

There are two types of log Log Partitions:

  • Indexed Partitions
    • Stores logs for up to 30 days
    • Billed only for volume of logs ingested into the partition
    • Search and analyse logs in this partition without additional costs
  • Non-Indexed Partitions
    • Stores logs for up to 7 years
    • Billed for the volume of logs ingested into the partition, and for searching the logs.
    • If you need to query logs frequently, you can move logs to a recall partition for 30 days.
      • No additional cost for searching and analysing logs in the recall partition

Logs that do not match a query criteria in any of the configured partitions, will be stored in the Default Indexed Partition. This is read only and stores logs for 30 days.

Note:  

- Alerts and dashboard widgets are not operational in non-indexed partitions.
- Log partitions store logs ingested in the last 24 hours only.
- You can create a maximum of 10 log partitions in an organization.
Video Walk-through

Example Logs

In my Log Insight environment, I have setup the FluentD configuration to forward the Tanzu Kubernetes Grid logs from two clusters to vRealize Log Insight (on-premises deployment).

You can find the configuration settings for this within vRealize Log Insight, under the Sources Tab > Containers > Tanzu Kubernetes Grid.

vRLI Log Archive - Configure Fluentd for Tanzu Kubernetes Grid

vRLI Log Archive - Tanzu Kubernetes Grid Logs

Setup the Cloud Proxy

  • Step 1 – Download the Cloud Proxy ova file
    • https://ci-data-collector.s3.amazonaws.com/VMware-Cloud-Services-Data-Collector.ova
  • Step2 – Import the .ova file to the vCenter Server and start the installation
  • Step 3 – When prompted for a key, Login into vRLI cloud and navigate to cloud proxy page, click on New to generate the key
  • Step 4 – It takes a few minutes to detect your cloud proxy on Log Insight Cloud after it is deployed and powered up in the vCenter
Setup log forwarding from the On-premises vRealize Log Insight instance

The vRLI on-premises instance has built in walkthrough guides, which you can visit following the below screenshot.

This blog post will continue taking you through the full configuration.

vRLI Log Archive - LI Cloud - Non-Index Partitions Setup Instructions

  • Step 5 – Open Log Insight on-premises deployment
  • Step 6 – Navigate to Log Forwarding and click on Create New Destination
  • Step 7 – For Host address input the IP address used during Cloud Proxy Deployment
  • Step 8 – Input queries to Filter Logs to be forwarded
  • Step 9 – Save Log Forwarding Configuration

You can read more about the Log Forwarding settings here.

vRLI Log Archive - Configure Log Forwarding

Below is an example of the two log forwarding rules I have created setup for each Tanzu Kubernetes Grid Environment logs, I could have multi-selected the environment names in same single rule as well.

vRLI Log Archive - Configure Log Forwarding examples

Logging into vRealize Log Insight Cloud, I can see the forwarded logs in my environment, I’ve used the search query to match the tags I set in the log forwarding rules.

vRLI Log Archive - Log Insight Cloud - view forwarded logs

Creating a Log Partition

If you are using a trial version of vRealize Log Insight, you will need to enable the Log Partitions feature as per the below screenshot.

If you have a premium subscription, log partitions are available by default, and you do not have to activate the feature.

vRLI Log Archive Enable Log Partitions

Once the feature is enabled, you can now create a new partition.

vRLI Log Archive - Log Partitions - New Partition

On the create new partition page provide the following details:

  • Name
  • Description
  • Type – Select whether you want to create an indexed or a non-indexed partition
    • An indexed partition stores logs for up to 30 days. Use indexed partitions to store logs that you plan to query regularly.
    • A non-indexed partition stores logs for up to seven years. Use non-indexed partitions to store logs that you do not plan to query regularly.
  • Retention
  • Routing Filter – definition of which logs should be ingested into this partition.
    • Add one or more routing filters to ingest logs corresponding to the filters into your partition. You can also use a favourite query.
    • Optionally, click Show Logs to preview the filtered log results and Show Chart to view a graphical representation of the log results based on your provided filter.

Click Create to finalise the configuration.

vRLI Log Archive - Create Log Partition - Indexed

For Non-Indexed Partitions, there are some additional configurations available:

  • Data Groups
    • Select the Group Data By check box and select the field by which you want to group the data.
    • Grouping log data by the relevant field helps store the logs effectively in sub-folders and displays quicker results when you query logs from your partition in the Explore Logs page.
  • Data Forwarding to Indexed Partitions
    • This configures the storing of the logs in both your partition and in indexed partitions.
    • Choose from the following options:
      • Forward all the logs in your partition
      • Add one or more filters to forward specific logs in your partition
    • If some or all the forwarded logs match the filters defined in certain indexed partitions, these logs are stored in the relevant partitions, based on the ingestion order. The forwarded logs that are not stored in any indexed partition go to the default indexed partition.

An example use of this feature to provide data forwarding to Indexed Partitions, is moving all Kubernetes based logs to your Non-Indexed Partition, however any logs which contain the tag “production” are copied into an Indexed Partition, to ensure you have the ability to query these quickly over the next 30 days.

vRLI Log Archive - Create Log Partition - Non Indexed

You will get a confirmation dialog box when you have created a partition.

vRLI Log Archive - Create Log Partition - Non Indexed - Successfully created

Changing the Log Ingestion Order

On the main Log Partitions screen, you can alter the ingestion order of your logs. As previously mentioned, all logs are forwarded to Non-Index partitions first, and then based on the rules in those Non-Index partitions, forwarded to Indexed Partitions.

vRLI Log Archive - Log Partitions - Modify Ingestion Order

After you have completed your configuration, on the main Log Partitions page, you will see your configured partitions, but also the ingestion order, and volume of logs stored as well as the last ingestion time.

vRLI Log Archive - Log Partitions

Viewing Logs inside a Log Partition

Clicking into one of the Partitions you can see a chart of the log ingestion based on the configured filter, and you can expand this to see the logs in the partition itself. You can edit the configuration for the partition under the “Actions” button, as well as exploring the logs.

vRLI Log Archive - Log Partitions - view logs in the partition

If you click the “Explore Logs” Button, this will take you to the Explore Logs UI page, with your particular partition set as the search partition.

vRLI Log Archive - Log Partitions - view logs in the partition - explore logs

If you try to run a query against a Non-Index Partition in the Explore Logs view you will be presented with this message.

Running a query on Non-Indexed Partition

Recalling logs from a Non-Indexed Partition

Querying logs that sit in a Non-Indexed partition, can be slow and incur additional costs. If there are logs in this type of partition you are expecting to need to query on a regular basis for a known amount of time (under 30 days), you can move all or specified logs to a common recall partition.

On the Log Partitions page, click the “Recall Logs” button

vRLI Log Archive - Log Partitions - Create Recall logs

Provide the following details:

  • Name
  • Description
  • Select the Non-Index Partition you want to recover the logs from
  • Select the time range you want to recover from
  • Set the filter criteria
  • Optional – Set who to notify once the recall logs are available.

Click the “Recall” button.

vRLI Log Archive - Log Partitions - Create Recall logs

You will be shown the Recalled Logs page confirming the creation is successful and in progress.

vRLI Log Archive - Log Partitions - Recall logs - Pending

Here is an example for the email notification.

vRLI Log Archive - Log Partitions - Recall Logs Completed - Email

Back in the vRLI Cloud Log Partitions page, I can now see there is an entry for “Recalled Logs”, and clicking this entry, will show the recall logs configuration runs.

We can then select our Recalled Logs and search them as per the other screens I’ve shown previously.

vRLI Log Archive - Log Partitions - Recall Logs Completed

Clicking into your Recalled Logs, you will then see the information regarding how long is left. Who the recall was requested by. The timeframe for the logs, and which partition they were recalled from.

You also have the ability to see a chart information of the logs pulled and the raw logs themselves.

By clicking Explore logs, this will take you to the Explore logs screen.

vRLI Log Archive - Log Partitions - Recall Logs Completed View

On the Explore Logs screen, you will a filter is set to identify the logs from their Recall Id. You can then add more filters as needed.

vRLI Log Archive - Log Partitions - Recall Logs in Explore Logs

Decommission and removing a Log Partition

You can delete a log partition you have created, called Decommissioning, this stops the ingestion of logs into this partition.

The partition will remain active and accessible for its retention period starting from the last ingestion data & time. You can view logs held in this partition but will no longer be able to edit it.

After the retention period, the partition is removed permanently, and you will no longer have access to the logs held within it.

  • Select the three dots next to your Log Partition name, and select to Decommission Partition

vRLI Log Archive - Log Partitions - Decommission Partition

  • Accept the confirmation

vRLI Log Archive - Log Partitions - Decommission Partition Confirmation

You will then see in the Log Partition UI page and on the Explore Logs page that this partition is now actively marked Decommissioned.

vRLI Log Archive - Log Partitions - Decommission Partition - Decomissioned

Summary

I ended up diving fully into the Log Partitions feature here, after initially just looking at howe we can create a hybrid logging platform, offloading logs from on-premises to the cloud instance. By using vRealize Log Insight Cloud, it means we retain that functionality to search and analyse the logs and control them in the same platform for retention as well.

Other possibly solutions are to export logs from your system, send them to another separate service, and then use that service to manage retention. But as it’s separate products, this means bringing those logs back can be a pain and cause a disconnect.

Regards

Dean Lewis

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.