AIOps Blog

AWS Cloud Observability with Log Analytics

5 minute read
Anuj Gupta

There are many types of logs in Amazon Web Services (AWS), and the more applications and services you run in AWS, the more complex your logging needs are bound to be. Logs originate from two primary sources—applications running on AWS services, and the AWS services themselves. An AWS centralized logging solution, therefore, becomes essential to manage this complexity. To achieve this kind of end-to-end visibility requires a conscious effort to centralize all the disparate logging data irrespective of their source of origin. AWS provides CloudWatch to centralize this data. CloudWatch is the primary collector that collects logs from different AWS services such as Amazon VPC Flow Logs, Route 53 Logs, Lambda Logs, CloudTrail Logs, and so on, in addition to log data from applications like Nginx or Apache system that you may be using in your AWS deployment.

Once collected in CloudWatch, you can use the BMC Helix Log Analytics solution to monitor and analyze logs and set up alerts. Doing this, you can get the native benefit of AWS log collection and the analytical power of the machine learning (ML)-powered observability platform provided by BMC. The ability to collect logs from CloudWatch allows you to aggregate all your log data combined with data from other sources across hybrid and multi-cloud environments. BMC Helix Log Analytics provides advanced monitoring and alerting capabilities to derive meaningful insights from logs, such as filtering by log metadata; enriching logs to add meaningful context; customizing alert messaging; automated alerting with ML-assisted anomaly detection; and service monitoring with BMC Helix Operations Management with artificial intelligence for IT operations (AIOps).

Centralized AWS logging with BMC Helix Log Analytics

The following diagram shows logs being collected from different applications and services across different AWS regions. CloudWatch collects logs by defining log streams and log groups for each region. A log stream is a sequence of log events coming from the application instance or resource being monitored. For example, a log stream may be associated with an Apache access log on a specific host. Log groups define groups of log streams that share the same retention, monitoring, and access control settings. One or more log streams belong to a log group. If you have a separate log stream for the Apache access logs from each host, you could group those log streams into a single log group called MyWebsite.com/Apache/access log.

Logs are aggregated in CloudWatch, and then collected and stored in BMC Helix Log Analytics for further analysis.

Collecting logs from AWS deployment

Figure 1. Collecting logs from AWS deployment

Collecting logs

BMC Helix Log Analytics uses BMC Helix Developer Tools to manage all the log collection integrations and provide log integration to collect logs from AWS CloudWatch.

AWS log integration

Figure 2. AWS log integration

To configure an AWS CloudWatch log integration you must:

  1. Provide credentials to access the AWS account that contains the logs to be collected.
  2. Download and install the BMC-provided connector from where to collect logs
  3. Configure the refresh time, define the tags to use, provide details of region, log groups and log streams from where to collect logs.

You can also specify the application format to parse logs and provide log filters to include or exclude specific data. Log parsing helps to convert unstructured raw logs into meaningful key-value pairs and make them ready for analysis. It also enables you to get statistics on log message parameter values, conduct faceted searches, and filter logs by specific fields and values.

Configuration to collect AWS logs

Figure 3. Configuration to collect AWS logs

Once the configuration is done, you can see the status of your integration, which tells whether your AWS log integration is in the connected state or not. It also provides the number of logs ingested and the heartbeat status of the underlying connector. This information helps the administrator and a site reliability engineer (SRE) resolve any issues with log configuration early in the cycle.

Configured AWS log integration

Figure 4: Configured AWS log integration

Monitoring logs

Once the logs are collected and stored, you can further analyze them in the log explorer to search, discover, or query any log record. Insights from logging can provide a great deal of context around the behavior of your applications and services and help you troubleshoot when you are dealing with an outage.

Discover and search logs in log explorer

Figure 5. Discover and search logs in log explorer

You can slice and dice a given log record to see detailed attributes and values, which helps you understand and further troubleshoot your issue.

Anatomy of AWS log record

Figure 6. Anatomy of AWS log record

Creating alerts

When things go wrong, alerts are essential for reducing response and recovery times. BMC Helix Log Analytics offers the ability to set alarms or alerts on any given condition occurring in the logs. Log events generated from these alerts can be operated in BMC Helix Operations Management, which takes proactive actions and sends notifications to you.

Log alert policies

Figure 7. Log alert policies

 

Analyzing log events in the BMC Helix Operations Management console

Figure 8. Analyzing log events in the BMC Helix Operations Management console

Visualizing logs

Dashboards help you track the most important metrics so you are always aware of the state of the system. You can create dashboards to monitor metrics derived from your logs, and visualize the data in the form of a line chart, a stacked chart, or a numerical metric. Taking things further, you can add alarms to widgets for quick and simple monitoring. BMC Helix Log Analytics uses BMC Helix Dashboards to provide out-of-the-box log dashboards that allow you to create self-service dashboards as needed. Here is a self-service dashboard to monitor and visualize logs from an AWS deployment.

Self-service AWS log monitoring dashboard

Figure 9. Self-service AWS log monitoring dashboard

BMC Helix Log Analytics is a great way to manage AWS observability with log monitoring. It acts as an advanced log monitoring solution to collect logs from AWS CloudWatch, perform monitoring and alerting, and deliver meaningful insights to improve and optimize your application. When used with monitoring metrics data from AWS and BMC Helix Operations Management with AIOps, it provides fully contextualized data about the state of your AWS services and the applications running in it.

The BMC Helix platform is an integral part of the BMC observability solution, giving SREs, DevOps engineers, and developers a seamless and streamlined workflow for IT monitoring, troubleshooting, and investigation to easily move from problem detection to resolution in minutes.

To find out more about log collection from AWS, check out our BMC Helix Log Analytics product documentation and video.

To learn more about BMC Helix Log Analytics capabilities, watch our overview video or refer to our product documentation.

Try BMC’s AIOps Capabilities - Free!

Start your free, easy-to-use, self-service trial to see firsthand how BMC Helix Operations Management with AIOps can revolutionize the way your enterprise does IT.


These postings are my own and do not necessarily represent BMC's position, strategies, or opinion.

See an error or have a suggestion? Please let us know by emailing blogs@bmc.com.

BMC Brings the A-Game

BMC works with 86% of the Forbes Global 50 and customers and partners around the world to create their future. With our history of innovation, industry-leading automation, operations, and service management solutions, combined with unmatched flexibility, we help organizations free up time and space to become an Autonomous Digital Enterprise that conquers the opportunities ahead.
Learn more about BMC ›

About the author

Anuj Gupta

Anuj Gupta works in Product Management team in Digital Service & Operations Management for BMC Software. In this role, Anuj is building product capabilities for Operations Management using BMC Helix AIOps and BMC Helix Log Analytics and helping customers improve their IT Operations and DevOps processes.