AIOps Blog

Archive logs to optimize storage & gain full visibility

4 minute read
Anuj Gupta

To gain full visibility into modern cloud environments, businesses must collect an ever-growing avalanche of log data from a range of overly complex data sources. Retaining logs is key for real-time monitoring and troubleshooting, but it can quickly become expensive at high volumes, meaning that organizations must often choose which logs to index and which to archive.

With new business requirements to log everything all the time, it can be a challenge to store and analyze all this data effectively and cost-efficiently. The proliferating number of applications doesn’t help, either. Another consideration is that the value of log data can transition from high to historical in a matter of weeks or days, which presents its own challenges when the storage cost of the data outweighs its potential value as a source of business insights.

With BMC Helix Log Analytics, you can archive and retain logs for a longer period of time at a cheaper cost. Before we dive into the details of the archival feature, let us look at some of the use cases that show us why archiving is important.

Why archiving log data is required

  • To meet regulatory standards—For compliance reasons, archiving your logs ensures that you are fully protected. Data retention policies can vary from several months to several years, depending on the type of service you provide and the standard or regulation with which you need to comply. For instance, section 802 of the Sarbanes-Oxley Act (SOX) requires organizations to archive their data for at least seven years.
  • To identify patterns and trends—Logs are essential for identifying and troubleshooting short-term problems but are less effective at identifying long-term trends. Older entries may get overwritten, deleted, or lost. Archives make it easier to identify patterns over a longer period than rolling log files.
  • To optimize log data storage—Archiving log data by employing compression techniques and storing archived logs in a location that does not need to be optimized for quick access are effective ways to save storage space and reduce costs. Furthermore, since the data can be decompressed and loaded into active databases any time without any data loss, it can still easily be used for on-demand troubleshooting or any other operation.

Perform historical analysis and investigations

Having the ability to store and analyze enormous amounts of historical log data is vital for situations that do not necessarily need immediate query responses. These include things like running security investigations across large environments, conducting audits to adhere to strict compliance frameworks, and performing long-term analytics on high cardinality datasets.

For example, when you experience a security breach or receive a report of an insider threat, your security team will need to comb through weeks, if not months, of log events to identify malicious activity. An investigation of all the activity from a suspicious IP address may require scanning petabytes of data, assessing the timeline of activity from that IP, and generating reports for other teams (e.g., legal and executive).

Similarly, businesses operating in regulated industries—such as financial services, insurance, healthcare, and aviation—have stringent requirements around servicing audit requests from among vast amounts of historical log data.

E-commerce providers, digital content makers, sports and entertainment companies, and businesses using Internet of Things (IoT) devices frequently need to perform long-term analytics on high cardinality datasets, such as users, IP addresses, device IDs, or items purchased, among others.

The log archival solution provided by BMC Helix Log Analytics addresses these use cases by retaining logs for longer duration, and restoring them back for on-demand analysis so teams do not spend valuable time spinning up new solutions, finding data loss, or worrying about query capacity and associated costs.

Log archival solution from BMC

BMC Helix Log Analytics provides an easy-to-use solution to archive logs in multi-cloud, software-as-a-service (SaaS), and on-premises platforms to help you perform historical analysis and investigations.

The following diagram illustrates conceptual flow of log archival and restore. Logs are first saved in a hot storage for a predetermined retention period. After that, they are moved to cold storage for longer-duration retention. Logs stored in cold storage archives are not available for search. To analyze these logs, you need to restore and bring them back into hot storage. Post-analysis, the logs are then auto-archived. After the archive duration, logs are purged and unavailable for search.

 Conceptual overview of log archival and restore

Figure 1. Conceptual overview of log archival and restore

Effortless configuration and data exploration

It is easy to configure this feature in your BMC Helix Log Analytics deployment by providing the application logs to be archived and the duration for which they need to be archived. Once enabled, all the logs’ data residing in different indexes will move to cold storage following the specified hot-retention period. You can select the logs index to be used for troubleshooting and restore that data.

Configuration to archive and restore logs

Figure 2. Configuration to archive and restore logs

The restored data is then available for analysis in the log explorer view, where you can query, search, and perform further action. The following diagram shows archived logs are restored and further analyzed in the log explorer.

Figure 3. Analyzing restored logs in log explorer

Figure 3. Analyzing restored logs in log explorer

Once your analysis is done, you can archive the data back, or have the restored data auto-archived once the restore period is over. This capability is accessible by administrator users who manage all the archival and restore operations.

The log-archival capability of BMC Helix Log Analytics is a cost-effective, cold-storage-based solution that helps organizations retain data for historical investigation and analysis and better meet compliance and regulatory standards, while continuing to use hot storage for real-time log streaming and alerting.

To find out more about log archival and restore, check out our BMC Helix Log Analytics product documentation and watch our overview video.

Related content

Make Your Data Smarter with BMC Helix Log Analytics and reduce MTTR

Join us to learn about how BMC Helix Log Analytics makes it easier to collect, analyze and visualize logs and solve problems faster before they impact the users and business. You will also learn how you can use logs data to enhance observability and support AIOps.


These postings are my own and do not necessarily represent BMC's position, strategies, or opinion.

See an error or have a suggestion? Please let us know by emailing blogs@bmc.com.

Business, Faster than Humanly Possible

BMC works with 86% of the Forbes Global 50 and customers and partners around the world to create their future. With our history of innovation, industry-leading automation, operations, and service management solutions, combined with unmatched flexibility, we help organizations free up time and space to become an Autonomous Digital Enterprise that conquers the opportunities ahead.
Learn more about BMC ›

About the author

Anuj Gupta

Anuj Gupta works in Product Management team in Digital Service & Operations Management for BMC Software. In this role, Anuj is building product capabilities for Operations Management using BMC Helix AIOps and BMC Helix Log Analytics and helping customers improve their IT Operations and DevOps processes.