AIOps Blog

AWS Cloud Observability with Log Analytics

5 minute read
Anuj Gupta

There are many types of logs in Amazon Web Services (AWS), and the more applications and services you run in AWS, the more complex your logging needs are bound to be. Logs originate from two primary sources—applications running on AWS services, and the AWS services themselves. An AWS centralized logging solution, therefore, becomes essential to manage this complexity. To achieve this kind of end-to-end visibility requires a conscious effort to centralize all the disparate logging data irrespective of their source of origin. AWS provides CloudWatch to centralize this data. CloudWatch is the primary collector that collects logs from different AWS services such as Amazon VPC Flow Logs, Route 53 Logs, Lambda Logs, CloudTrail Logs, and so on, in addition to log data from applications like Nginx or Apache system that you may be using in your AWS deployment.

Once collected in CloudWatch, you can use the BMC Helix Log Analytics solution to monitor and analyze logs and set up alerts. Doing this, you can get the native benefit of AWS log collection and the analytical power of the machine learning (ML)-powered observability platform provided by BMC. The ability to collect logs from CloudWatch allows you to aggregate all your log data combined with data from other sources across hybrid and multi-cloud environments. BMC Helix Log Analytics provides advanced monitoring and alerting capabilities to derive meaningful insights from logs, such as filtering by log metadata; enriching logs to add meaningful context; customizing alert messaging; automated alerting with ML-assisted anomaly detection; and service monitoring with BMC Helix Operations Management with artificial intelligence for IT operations (AIOps).

Centralized AWS logging with BMC Helix Log Analytics

The following diagram shows logs being collected from different applications and services across different AWS regions. CloudWatch collects logs by defining log streams and log groups for each region. A log stream is a sequence of log events coming from the application instance or resource being monitored. For example, a log stream may be associated with an Apache access log on a specific host. Log groups define groups of log streams that share the same retention, monitoring, and access control settings. One or more log streams belong to a log group. If you have a separate log stream for the Apache access logs from each host, you could group those log streams into a single log group called MyWebsite.com/Apache/access log.

Logs are aggregated in CloudWatch, and then collected and stored in BMC Helix Log Analytics for further analysis.

Collecting-logs-from-AWS-deployment

Figure 1. Collecting logs from AWS deployment

Collecting logs

BMC Helix Log Analytics uses a log collection policy to collect logs from AWS CloudWatch. To configure the collection policy, you must provide credentials to access the AWS account that contains the logs to be collected. Next, you would need to download and install the connector on the AWS platform to collect application or services logs for monitoring.

Log-policy-for-collecting-AWS-logs

Figure 2: Log policy for collecting AWS logs

You may configure the refresh time and provide details of the region, log groups, and log streams for the logs to be collected.

You can also specify the application format to parse logs and provide log filters to include or exclude specific data. Log parsing helps to convert unstructured raw logs into meaningful key-value pairs and make them ready for analysis. It also enables you to get statistics on log message parameter values, conduct faceted searches, and filter logs by specific fields and values.

Configure-AWS-logging

Figure 3: Configure AWS logging

Once the configuration is done, you can see the health status of the log connector and policy configured to ensure there is no error with log collection. Then you can start to analyze logs in Explorer and create alerts and other policies as appropriate.

Monitoring logs

Once the logs are collected and stored, you can further analyze them in the log explorer to search, discover, or query any log record. Insights from logging can provide a great deal of context around the behavior of your applications and services and help you troubleshoot when you are dealing with an outage.

Discover and search logs in log explorer

Figure 4. Discover and search logs in log explorer

You can slice and dice a given log record to see detailed attributes and values, which helps you understand and further troubleshoot your issue.

Anatomy of AWS log record

Figure 5. Anatomy of AWS log record

Creating alerts

When things go wrong, alerts are essential for reducing response and recovery times. BMC Helix Log Analytics offers the ability to set alarms or alerts on any given condition occurring in the logs. Log events generated from these alerts can be operated in BMC Helix Operations Management, which takes proactive actions and sends notifications to you.

Log alert policies

Figure 6. Log alert policies

 

Analyzing log events in the BMC Helix Operations Management console

Figure 7. Analyzing log events in the BMC Helix Operations Management console

Visualizing logs

Dashboards help you track the most important metrics so you are always aware of the state of the system. You can create dashboards to monitor metrics derived from your logs, and visualize the data in the form of a line chart, a stacked chart, or a numerical metric. Taking things further, you can add alarms to widgets for quick and simple monitoring. BMC Helix Log Analytics uses BMC Helix Dashboards to provide out-of-the-box log dashboards that allow you to create self-service dashboards as needed. Here is a self-service dashboard to monitor and visualize logs from an AWS deployment.

Self-service AWS log monitoring dashboard

Figure 8. Self-service AWS log monitoring dashboard

BMC Helix Log Analytics is a great way to manage AWS observability with log monitoring. It acts as an advanced log monitoring solution to collect logs from AWS CloudWatch, perform monitoring and alerting, and deliver meaningful insights to improve and optimize your application. When used with monitoring metrics data from AWS and BMC Helix Operations Management with AIOps, it provides fully contextualized data about the state of your AWS services and the applications running in it.

The BMC Helix platform is an integral part of the BMC observability solution, giving SREs, DevOps engineers, and developers a seamless and streamlined workflow for IT monitoring, troubleshooting, and investigation to easily move from problem detection to resolution in minutes.

To find out more about log collection from AWS, check out our BMC Helix Log Analytics product documentation and video.

To learn more about BMC Helix Log Analytics capabilities, watch our overview video or refer to our product documentation.

Related content

Make Your Data Smarter with BMC Helix Log Analytics and reduce MTTR

Join us to learn about how BMC Helix Log Analytics makes it easier to collect, analyze and visualize logs and solve problems faster before they impact the users and business. You will also learn how you can use logs data to enhance observability and support AIOps.


These postings are my own and do not necessarily represent BMC's position, strategies, or opinion.

See an error or have a suggestion? Please let us know by emailing [email protected].

Business, Faster than Humanly Possible

BMC empowers 86% of the Forbes Global 50 to accelerate business value faster than humanly possible. Our industry-leading portfolio unlocks human and machine potential to drive business growth, innovation, and sustainable success. BMC does this in a simple and optimized way by connecting people, systems, and data that power the world’s largest organizations so they can seize a competitive advantage.
Learn more about BMC ›

About the author

Anuj Gupta

Anuj Gupta works in Product Management team in Digital Service & Operations Management for BMC Software. In this role, Anuj is building product capabilities for Operations Management using BMC Helix AIOps and BMC Helix Log Analytics and helping customers improve their IT Operations and DevOps processes.