INTRO

The public cloud is the incubator of change. Use of cloud IaaS (Infrastructure as a Service) and PaaS (Platform) services is growing 24% YoY, reaching $90 billion in 2020. Cloud sprawl, change velocity, scrum teams, CI/CD pipelines – your cloud footprint is constantly changing, which complicates cloud security. To underscore the point, 93% of IT execs are moderately to extremely concerned about their cloud security. This blog offers some guideposts on the journey to securing your public cloud, including items like automated configuration, cloud server patching, asset discovery, and more.

AUTOMATE CONFIGURATION SECURITY & COMPLIANCE

Misconfigurations are the #1 cause of cloud security failures, so we begin here. AWS, Azure, and Google Cloud offer hundreds of IaaS and PaaS services in their catalogs which developers devour as they deploy cloud-native apps. Every instance of those IaaS and PaaS services must be appropriately configured to be secure, a responsibility which falls solely on the enterprise, not the CSP.

Thankfully, the CIS (Center for Internet Security) publishes benchmarks detailing how to securely configure IaaS and PaaS services. Even so, enterprises continue to struggle. The list of high-profile exposures includes such misconfigured resources as:

• S3 buckets (here, Lion Air, Ford, TD, Netflix, plus a non-exhaustive list),
• Elasticsearch clusters (Honda Exposes Vehicle Owner Records), and
• web application firewalls (Capital One 2019).

As the saying goes, an attacker only has to be right once. While automation has made it all too easy for bad actors to probe our cloud defenses, so too can automation become our best defense.

Automating policy-based configuration security checks against the CIS recommendations is a good start.  These checks should not only happen on a periodic cadence (for trend analysis, etc.) but also the moment a new resource is deployed, or an existing resource modified.  This is known as event-driven security.  Yet simply pointing at a misconfig is not enough, which brings us to our next recommendation.

AUTOMATE REMEDIATION

Automated remediation brings many benefits, not the least of which is continuous, real-time security, plugging the gap immediately, in the moment of change, to prevent data exposure in the white spaces between scheduled sweeps. Just as the configuration policies are under centralized governance of your Security and Compliance Team, so too should remediation action, to drive consistent configurations across the dynamic enterprise. Exceptions can be managed/approved as needed. Automation also eliminates human error, diminishing the prospect of disruptive rework or costly service outages. Now, before you object about the dangers of automation run wild, be assured that we will soon discuss the virtues of change management.

AUTOMATE CLOUD SERVER MANAGEMENT

Patching and OS compliance of virtual cloud servers are also the responsibility of enterprises under the Shared Responsibilities Model.  Given that the speed and scale of change challenge cloud security posture management (CSPM), one might rightly infer that manual efforts to locate new cloud server instances and add to the backlog of servers to be patched are likely to fall short.  Here too, automation is the solution.

Automatically find and enroll every newly deployed cloud server instance (e.g., AWS EC2) into an automated server management solution.  Then, you can schedule automated patch jobs to keep your cloud servers’ guest OS patched and updated.  Having a single context from which to manage servers, whether on-prem, virtual, or cloud is helpful and, where possible, highly recommended.

CLOSE THE LOOP ON INCIDENT & CHANGE MANAGEMENT

Given the pace of change and the importance of quickly resolving resource misconfigurations, automating incident ticketing and enrichment and orchestrating change management are key to managing change smoothly and maintaining an audit trail. Perhaps incident management is overkill for developers in their DEV accounts but consider production operations. The moment a misconfigured resource is identified in PROD, a security incident ticket is opened and supporting security details are automatically shared with the ITSM solution. The developer responsible for the app will be automatically alerted. When the developer goes to remediate the configuration, a change request is created, and the change management workflow initiated. Once approved, the change goes live, the CMDB updated, and the ticket closed. The vulnerability window is minimized, and a fully documented audit trail recorded.

KNOW WHAT’S IN YOUR CLOUD

Developers can be an eager and capable force multiplier for an overburdened Security and Compliance Team.  Developers view their professional world through the lens of their application or microservice, so let’s show them their cloud security posture through that same lens.  Intuitively, this implies a need to know what resources are deployed and how they interconnect within their cloud footprint.  With asset discovery automatically locating cloud resources and mapping their dependencies, developers can exert more directed force on their app’s security and compliance.  There is another key component to achieving this goal:  simplify security.

DEMOCRATIZE CLOUD SECURITY

If we simplify cloud security posture management for the developers, we increase their cooperation in the common cause of securing our cloud footprint.  An automated CSPM solution should ingest asset relationships and logical groupings to present the security posture of their IaaS, PaaS, and containerized resources in an app-centric view.  This implies multi-tenancy – you have more than 1 development team, and they only need to worry about their app’s posture.  It also implies role-based access control – you would not want a developer changing the configurations of resources which are not in their app.  Ideally, the CSPM solution should be multi-cloud, because asking a developer to drive a handful of security solutions in AWS, another set in Azure, and yet another in Google Cloud is neither simple nor agile.

ARCHITECTURAL DESIGN SECURITY

A beautiful aspect of the public cloud is you can build security into the design of your business services.  You can begin by understanding what architectural options exist with your CSP, and your organization’s Cloud Architect should be able to provide valuable guidance to the app development teams.  Consider the following simple example in Figure 1 built in AWS with a combination of IaaS and PaaS services.

Let’s say you have some business service which you have migrated to the cloud and which now runs on an EC2 instance.  This EC2 will be fed data records from your on-prem data center.  For expediency and simplicity, data encryption/decryption is not shown but assumed.

Cloud network segmentation.  Firstly, you can segment that workload off from any other workload in your cloud by giving it its own VPC (virtual private cloud).  Some organizations will also dedicate a specific cloud account to each workload, shown here as well.  Network segmentation is a means of mitigating blast radius, should a compromise occur.

FIGURE 1: SIMPLE BUSINESS SERVICE ARCHITECTURAL DESIGN EXAMPLE

Web application firewall.  A WAF is a highly effective means of protecting your cloud apps from common exploits, through security rules which you define to control and/or block certain types of traffic.  For example, you can create a security rule to only accept traffic from specific IP addresses, such as from your on-prem data center.

Data IO and Service Endpoint.  Once admitted through the firewall, the data are simplify dropped into an S3 bucket. (Every IaaS and PaaS resource in this example will be securely configured by your automated, policy-driven CSPM solution.)  Data in the drop bucket will trigger a serverless AWS Lambda function to give the data to the business service running on the EC2.  Results are returned to the Lambda function for transfer into the results bucket.  Note that the Lambda function was the sole endpoint:  only it could trigger the business service.  IAM roles in this architecture are discussed in the next section.

IAM

Appropriate IAM roles and policies provide only the least privileges necessary to get the job done.  In the example of Figure 1, you would create an IAM role for the Lambda function to access the S3 drop bucket, a separate IAM role for the Lambda function to trigger the EC2, and yet another to write to the results bucket.  With appropriate IAM roles, only the Lambda function could write to the drop bucket in the example above.

Multi-factor authentication (MFA) is critical due to the broad access and to offset the risk of a cloud account takeover attack.

NOTE: MFA is mandatory for adequately protecting highly privileged accounts, as compromised admin credentials represent the greatest threat to an enterprise cloud.

The CIS benchmarks for IAM include several, high-value recommendations, including but not limited to the following:

1. DO NOT create IAM policies that allow full admin privilege
2. DO require MFA for all IAM users with a console password
3. DISABLE credentials which have been unused for more than 90 days
4. ROTATE access keys every 90 days or less

There are several additional rules within the framework which speak to the importance of least privileges.

OTHER RECOMMENDATIONS

In addition to IAM, CSPM, discovery, and architectural security, here are some other key points to consider in securing your public cloud footprint.

• Key management. Absolutely essential.  Rotate data encryption keys regularly and store them separately from the decryption engine.  Look for a managed key service from your CSP which greatly simplifies key management.
• Encrypt data at rest and in transit. Your CSP should have data encryption options when storing data.  When transferring data between various cloud network segments, use encryption.  These use cases also tie back to key management and IAM roles.
• Event logging. Storage is ridiculously inexpensive.  There is no excuse not to enable event logging such as AWS CloudTrail, incredibly valuable for forensic analysis.  After a certain time period, you can automate data transfer to long-term “cold” storage for more savings.
• Automation. To err is human, so automate what you can to minimize errors, rework, and disruption.  Cloud configurations, discovery, and ticketing should be automated as much as possible to track with the speed and scale of change.

CULTURAL ASPECTS

Executives know the important role culture plays in helping organizations achieve their objectives.  With the speed the cloud enables and the implications to your business, consider the following aspects to bolster your cloud security.

Foster cross-functional collaboration.  Devise incentives that create collaborative muscle memory among app developers, IT operations, and security experts.  Offer career development opportunities such as cross-functional training and project assignments to spark new ideas.  Invite Operations, Security, and Compliance to sprint retrospectives.

Encourage risk-taking.  Do not punish teams for making small bets, iterating, and learning.  Be open to new ideas and create an environment that makes it safe to fail.

Gamify security.  Have a monitor on display – near the elevator, the break room, wherever – showing KPIs across various scrum teams’ security posture.  Healthy competition is good but keep it light:  you want the developers sharing best practices.

SUMMARY

While devising an effective multi-cloud security strategy requires many interworking pieces and considerations, an easy win is to begin by addressing the #1 cause of cloud security failures, misconfigured IaaS and PaaS resources.  A multi-cloud CSPM solution automates policy-driven configuration management and, ideally, remediation; policies and remediation actions would remain under centralized governance to drive consistency across the enterprise’s rapidly changing cloud footprint.  Moreover, one must also take care to automate cloud server patching and OS compliance, given how popular cloud IaaS services such as AWS EC2 or Azure VM have become.

Developers represent an eager and capable force-multiplier for an overburdened Security and Compliance department, so simplifying security for those developers is instrumental to success.  Present an app-centric cloud security perspective to the various scrum teams, achieved by integrating your CSPM solution to automated asset discovery.  Orchestration of incident and change management accelerates responsiveness and business agility.  Architectural considerations can augment your security efforts, and the principle of least privileges should be central to cloud security.

CLOUD SECURITY SUCCESS CHECKLIST

• automate policy-based IaaS and PaaS resource configuration checks and remediation
• automate cloud server (AWS EC2, Azure VM) patching and OS compliance
• automate asset discovery and application dependency mapping
• orchestrate security incident and change management
• architect your cloud applications for security
• turn on event logging
• apply the principle of least privileges to IAM
• consider a managed key service
• encrypt data at rest and in transit

BMC SOLUTIONS

BMC has a wealth of solutions which work together to automate and secure the hybrid cloud enterprise.

• Adding BMC Helix Vulnerability Management to your arsenal super-charges security backlog grooming and automated remediation for hybrid cloud servers and networks.
• BMC Helix ITSM provides industry-leading incident management, while BMC Helix Discovery enables security posture management through an app-centric lens.
• And TrueSight Orchestration keeps all the multi-faceted process runbooks running smoothly together.

To learn more about BMC Software’s portfolio for securing the hybrid cloud enterprise, please visit bmc.com/remediate, or contact us for a conversation about your specific security challenges.

When more than 100 million personal data records are exposed, it’s bound to spark questions and concerns – rightfully so. In the recent cloud security breach involving Capital One, U.S. Senator Rod Wyden of Oregon is asking, does Amazon Web Services share responsibility for the data exposure? And, if so, is there more that can be done to protect other AWS customers from the same issues? His letter is here.

One may think this issue is not worth Senate attention, but the Senator does point out that if it’s one company involved, then it’s likely on that organization. However, in this case a well-known “vulnerability” in the way certain AWS services are configured led to the data exposure. The Wall Street Journal says it found an additional 800 AWS customers potentially at risk from the same configuration error.

The Shared Responsibility Model

When it comes to security in the cloud, Amazon tries to limit the gray area via its Shared Responsibility Model. In short, Amazon says it is responsible for security OF the cloud – the infrastructure and software that run it. The customer is responsible for securing what’s IN the cloud.

It states: “This shared model can help relieve the customer’s operational burden as AWS operates, manages and controls the components from the host operating system and virtualization layer down to the physical security of the facilities in which the service operates. The customer assumes responsibility and management of the guest operating system (including updates and security patches), other associated application software as well as the configuration of the AWS provided security group firewall.”

Why Does This Keep Happening?

Simply stated, the cause of the cloud security failure is, once again, a misconfigured cloud service. If it has a configuration switch, the enterprise is responsible for configuring that service so that it is secure.

This is somewhat easy and manageable in a simple environment, but how many of today’s enterprise applications are simple? This gets more complicated by velocity of change and scale of change. Dozens (or more) of scrum teams continuously update their microservices, and with each update comes the risk of a simple misconfiguration. Security and compliance checks must be run with each release, but manual and/or ad-hoc checks create bottlenecks and lack of consistent governance can introduce risk.

This type of error can happen to ANYONE; the scale and speed of change which the cloud enables leads to just such a risk. Publicly disclosed exposures due to misconfigured cloud resources are up 20 percent year-over-year¹, and misconfigured cloud resources remain the #1 cause of cloud security failures². Either through misunderstanding – 53% of organizations mistakenly believe their CSP is wholly or majority responsible for securing their data³ – or negligence, these misconfigurations prove time and again that getting cloud security right is a monumental undertaking.

Why Has This Specific Breach Captured Our Attention?

Consumers are highly protective – rightly so – of their personal and financial data. Whether it’s because the brand is well-known, or that this is yet another financial breach, or that instead of a nameless, faceless actor of a hostile foreign power we have the name and face of a suspect who documented her exploits on social media, the Capital One breach, more so than any since Equifax (which was an on-prem breach), has captured the attention of the media.

Far too much has been made of the fact that the hacker was a former AWS employee. True, she was employed there some number of years ago, but her employment history with AWS is entirely irrelevant. More precisely, it is her familiarity and expertise in using AWS cloud services which is key. One does not have to ever have been an employee of Amazon to develop these skills. There is no public evidence available thus far to warrant an insider threat.

So, what did happen? Simply stated, the suspect allegedly found and exploited a misconfigured WAF (web application firewall, a type of cloud service) to access Capital One’s VPC (virtual private cloud). Once inside, she launched an EC2 instance to trick the AWS Metadata Service into trusting it, thereby receiving credentials to access and decrypt data stored in an S3 bucket. An excellent detail of the hack can be found at Krebs on Security. See Server-Side Request Forgery for an explanation of how an SSRF works. In lay terms, a misconfigured cloud resource (WAF) caused the cloud security failure.

How Do We Prevent This from Happening Over and Over?

Thanks to the suspect having left extensive breadcrumbs on her social media, we have a very clear view of how she exploited the cloud security of Capital One. Beyond the “how” it happened, what can we – the collective “we,” the enterprise “we” – do to prevent recurrence? Fortunately, the answer is plenty.

A solution for effectively managing the enterprise’s cloud security posture must include:

• Automation. Automation is the key, because manual checks are a fool’s errand: the cloud footprint changes too fast. Multiple, self-organizing scrum teams push updates to PROD asynchronously and multiple times daily. With each update comes the risk of a single cloud resource being misconfigured. As such, you cannot solve the problem of cloud security posture management (CSPM) by throwing bodies at it. The speed and scale of change outstrips human ability to keep pace. Automation eliminates human error and delays caused by manpower bottlenecks, making cloud security and governance as agile as the developers who use the cloud. It also enables your high-cost security professionals to rise above the noise of repetitive ditch-digging so that they can focus on higher value work. In sum, you get better staff productivity, higher quality, better security, and higher customer satisfaction.
  • Recommendation: Use automated security checks against policy-based best practices such as those published by the Center for Internet Security.
  • Recommendation: Use automated remediation, working programmatically to reconfigure those cloud resources which violate security and compliance policy. Automation removes the element of human error and dramatically diminishes the window of vulnerability.
  • The difference between automatic and automated remediation sometimes causes confusion. Automated action runs a remediation script triggered by some human intervention, such as clicking a button. Automatic remediation takes the same action, but without any human intervention, so that the moment a vulnerability is found, it is automatically fixed – consistently, securely, and with a documented audit trail according to the organization’s change management workflow. Both automated and automatic remediation offer profound advantages above manual security and compliance checks.
• Policy-based security, compliance, and governance. The enterprise’s Infosec Team codifies security and compliance mandates into a library of security policies, which are applied uniformly throughout the enterprise. Because security checks are run comparing all cloud resources automatically against these policies, real-time security is achieved because there is no manpower bottleneck at the security checkpoint in the process. These security checks are automated, as the fixes should be as well. Exceptions to any security violations are managed according to whatever process the organization puts in place.
  • Recommendation: use the security frameworks published by the Center for Internet Security as the foundation of your security policy library, to speed your time-to-value and achieve consistent configurations across your cloud footprint. Extend those policies as you see fit.
• Multi-cloud. Although the Capital One breach was on AWS, AWS is not the only cloud service provider (CSP). AWS, Azure, and Google Cloud dominate the CSP market, and each have hundreds of similar, but not identical, cloud services, every instance of which must be correctly configured. AWS have a laundry list of security tools available for purchase. The trouble is, (1) naturally they focus on AWS services, (2) do not provide for remediation (which is the customer’s responsibility), and (3) numerous tools complicate security and make it difficult for developers to secure their microservices. Most enterprises are multi-cloud, and, as already discussed, it is imperative that the organization consistently apply security and compliance policies across its scrum teams, accounts, and cloud platforms. The flexibility which the cloud provides is powerful, though it is a razor-sharp, double-edged sword.
  • Recommendation: enable business agility by making security easy on the developers. Choose a cloud security solution which supports the “Big 3” CSPs IaaS and PaaS services, so that you get more cooperation from your internal customers (*cough* developers *cough*).
• Integrate to incident and change management workflows. As previously mentioned, the speed of change in the cloud from numerous, self-organizing scrum teams challenges the organization’s change management. As security professionals, our goal should be to enable agility, and not hamstring it. Once a violation is identified, we should automatically open an incident and change request at the service desk, and kickoff our change management workflow. Once approved, the configuration change is automatically made, thereby closing the vulnerability and the incident closed. The CMDB is updated and an audit trail documented.

Conclusion

Capital One is a leader and expert in the use of cloud technology. While this type of cloud security failure could happen to anyone, there is plenty we can do to prevent these types of failures. The cloud itself is not inherently insecure, but we (the collective “we,” users of the public cloud) continue to struggle with securing our ever-changing footprint in the public cloud. Just like when AWS changed the default configuration of its S3 (Simplified Storage Service) bucket so that it is private, not public, it is worthwhile to have a rational, non-politicized discussion of what actions AWS and all CSPs can take.

At the same time, all organizations are encouraged to take stock of their current methods, skills, and technology for securing their public cloud estate. Waiting for a regulation to fix the problem might invite more trouble. The better course is to take action on your own to make sure the data you store and the services you offer are secure, regardless of where they reside.

¹ 2019 IBM X-Force Threat Intelligence Index Report ↩
² i.b.i.d. ↩
³ EMA Security Megatrends 2019 ↩

I’ve been spending a fair amount of time recently speaking with enterprises about their cloud strategy. As I sit in airport terminals, mindlessly people-watching and purposefully avoiding my inbox (what is this “inbox zero” sorcery of which you speak?), my thoughts gravitate to common themes from those meetings. While practically every enterprise already uses the public cloud, they recognize they can be doing better; I mean, it’s why they took the meeting in the first place. Struggles with technical and cultural transformations are even more prevalent than I expected going in. Cost is a challenge, security is a raging concern, and automation is needed to remove manpower bottlenecks. Let’s throw some thoughts on digital paper on what a successful cloud strategy looks like. I will make every effort to avoid the obvious or cliché, so if I do so here, I beg your indulgence, that the perspective is not only mine but also comes directly from customer interviews. Just looking to share common themes and practical guidance on moving your cloud journey forward. From Gate A24, here goes…

Enable Agility, Don’t Hinder It

“Enable agility, don’t hinder it.” I’ve been hammering this mantra for quite some time now, and for the first time, in one of my meetings, a security leader led the conversation to this mantra before I could even show a slide with those very words on it. It was like some sort of peculiar magic trick, and we all had a good laugh. (Aside: most of our meetings were conversational, with very few PowerPoint slides, and in most cases, none.) If the governance team – or security, or security and compliance, or a rose by any other name – makes cloud usage difficult, then the scrum teams will simply go around them.

And you already know you do not want that. The answer is not more, but better, restrictions.

Pro tip: do not over-rotate on governance. Cloud usage outside of centralized governance is bad. Very bad. Like an existential threat to your business competitive advantage bad (e.g., exposed IP, fines, brand damage, loss of talent/turnover). If you over-rotate, the scrum teams will just go around you and then you won’t know what is out in the cloud. And by the time customer data or IP is inadvertently exposed in an unsecured Elasticsearch service, it really will not matter whose “fault” it is. It was reassuring to hear how receptive everyone was to this theme.

If “enable agility, do not hinder it” is the guiding principle, then the Cloud Center of Excellence (CCOE) will go a long way in effectively supporting the business strategy.

Enable the Scrum Teams to Self-Serve

A few months back, a developer bristled a bit when I wrote that they were responsible for the configuration of their cloud services. He pointed out that the tools forced upon him by the CCOE were numerous and not easy to use. I’m quick to admit when I am wrong. In this case, we were both RIGHT. And I was wrong to not give the weight that ease of use deserves in this cultural transformation. To lead a successful cultural and technical transformation which the public cloud requires, the CCOE needs to deliver fewer complicated, point-specific tools requiring a Ph.D. to use; and the customers – the dev teams – need to be accountable for their own cloud spend and cloud security posture management.

The practical advice I can offer here is to have executive sponsorship from the application services leadership, the CCOE, and the CIO. Executive leadership is helpful in leading cultural change. Plus, if you’ve chosen easy-to-use solutions that help devs do their job, then agreeing to use those solutions should not be a burden.

Cloud services must be appropriately configured if they are to be secure. The speed and scale of change in the enterprise’s cloud footprint is not anything that a centralized security and compliance could, or even should, be responsible for managing. That responsibility rests upon the user themselves. Cloud Operations should not be a traffic cop. Instead, autonomous and self-organizing scrum teams should be able to spin up cloud resources on-demand, provided they have the tools to manage their cloud resources.

Automate Public Cloud Security

And while the developers are spinning up those cloud resources, the Cloud Governance team can, and should, have deployed policies which automatically test that the cloud resources are securely configured. If they are not so configured, then the organization can decide the next step. Should they block deployment to PROD? Probably, unless an exception is agreed upon. Who fixes the misconfiguration? Well, that would be the developers. How do they do it? Doing so should be as simple as clicking a button; that is, remediation should also be automated.

DO use best practice recommendations from the Center for Internet Security as the standard for configuring your cloud resources. DO automate security checks and remediation – doing so will not only ensure consistent cloud resource configurations across your large, growing, and rapidly changing cloud footprint, but also radically diminish risk. DO let the scrum teams own their security posture. DO provide security and compliance policies for the appropriate configuration of cloud resources.

Misconfiguration of cloud resources remains the #1 cause of cloud security failures.

Orchestrate Change Management

While the developers are busy deploying new resources, and/or changing existing ones, it is important to manage this process so that change management does not itself become a manual bottleneck. Automate change request tickets. Have an agile change management workflow which keeps the devs innovating and accelerating along sensible guardrails. DO keep that CMDB updated.

If you can get cloud security and change management working together synchronously and automatically, you will be doing this better than 99% of your competition. Agility and security are not mutually exclusive. It’s just hard to imagine this future state, when cloud security failures from open S3 buckets still routinely make the news. Automate cloud security. Orchestrate change management.

Asset Discovery and Tagging Taxonomy

I’ve been trying to weave this important topic in, but it didn’t seem a good time until now. You cannot manage what you cannot see. And from what customers and prospects are saying, visibility remains a big challenge. Of course, a lot of this, but not all, is the unintended consequence (shadow cloud usage) of draconian governance as previously discussed. A successful cloud strategy begins with knowing what resources you have and how they interconnect. Application mapping is crucial, as is enforcing an asset tagging taxonomy. Taxonomy enforcement – gosh, that sounds draconian itself, sorry – is easily codified with compliance policies. For example, IF a resource is deployed without a tag to an approved department, THEN take down the resource AND notify the user. In this case, the developer will quickly add the department tag and redeploy: no harm, no foul. I’d also have a tag for the application, and the reader can easily decide which tags make sense for their organization.

Cloud Expense Management

You are overprovisioned on-premises, so why would you expect to be any different in the cloud? My apologies if you changed your process, recognizing that cloud OPEX charged back to your department for resources not being used is particularly painful. Setting money on fire is generally frowned upon by financial controllers.

A successful cloud strategy will empower application owners themselves to proactively manage their own cloud budget. Machine learning will reveal usage patterns, improve forecasting, and predict when budget excursions could materialize, so that the scrum team can intervene before the cost overage happens. Were I in the CIO’s shoes, I would also want machine learning to proactively identify cost savings opportunities – show me WHERE resources are overprovisioned, and how much money we could save. Show me how right-sizing these resources would affect application performance, so my team can factor that into its decision-making. Also, release cloud resources that are not being used so you are not burning OPEX. In short, show cloud cost by application, proactively manage budget, improve forecasting, and proactively recommend cost saving opportunities.

Gamification

Friendly competition among teammates raises everyone’s performance level. This is the theory behind gamification of cloud operations management. Since we’ve established above that cloud ops should enable agility, and that scrum teams should actively manage their own cloud budget and security posture, let’s challenge everyone to raise their game. Why not have a dashboard (or, “information radiator” in the parlance of scrum) which shows how the different app teams are performing relative to each other? You’ll identify and share best practices from the highest performing teams, as well as get help to where it will have the most impact. Over time, the leaderboard will read like a bunch of Olympic swimmers: even the one at the bottom is still a top-notch world athlete.

Summary

In short, we need AI to reveal insights buried under a mountain of data and automation to put those insights into action, removing bottlenecks and accelerating agility. Enabling scrum teams to self-serve within sensible policy guardrails and to proactively manage their cloud cost and security will drive the organization to new heights. Friendly competition and sharing best practices will raise everyone’s game. Hopefully these themes from recent meetings are helpful to you in managing your own cloud journey. I’d like to hear your thoughts. What’s missing? What cloud challenges are your organization facing? What’s working well, and what could be better? Tweet me @rickbosworth8 (#cloudops), or even share this post with your thoughts. When we expand the conversation, everyone benefits.

CloudOps is shorthand for cloud operations, the managerial practice and activities relating to the delivery and optimization of IT services and workloads run in the public cloud. CloudOps includes the definition and ongoing refinement of business processes and methods to optimize the availability, flexibility, and efficiency of cloud services so that the business can successfully execute its mission. CloudOps is the incubator for business agility.

The rise of Cloud Centers of Excellence. You may wonder, “Why would there be a separate team dedicated to IT operations in the cloud? I mean, why wouldn’t they just be in IT operations?” Sometimes, CloudOps is within IT Operations. More often, however, organizations have specialized cloud centers of excellence, or Cloud COE, consisting of cloud experts from cross-functional disciplines such as cloud architecture, cloud services, IT operations, security, and compliance. The Cloud COE exists as a tacit acknowledgement that the people, processes, and technologies used to deliver on-premises IT operations simply do not translate to the unique needs of the cloud. To punctuate this point, according to the Flexera 2019 State of the Cloud Report 66% of enterprises already have a Cloud COE, with another 21% planning one.

Cloud COE must define solution requirements, optimize processes, and manage daily cloud operations which enable this agility, not hamstring it. And therein lay the challenge.

The challenges of CloudOps. The Cloud COE face a monumental task in making sure their entire enterprise is “cloud smart” instead of simply “cloud first.” For many, accelerating agility without addressing the underlying challenges means, unfortunately, increasing the capacity to break glass.

• Wasted cloud spend. Overprovisioned or idle assets frivolously set precious budget on fire. Estimates of wasted spend range from 25% all the way up to 90%. While the extremes are the exception, 35-40% is a more likely the norm.
• Budget overruns. By the time you get that unwelcome surprise bill from your CSP, it’s too late. The liability is on the books. By some estimates, 80% of enterprises are overspent. Half of enterprises spend at least $1.2M annually on public cloud, and 13% spend $12M or more².
  Despite the financial risks, public cloud IaaS and PaaS segments are growing north of 26% annually³. Slowing down is not an option, but neither is blasting through expense forecasts.
• Limited governance. Distributed autonomy to purchase and use cloud has impaired visibility and governance. Cloud consumptions in the shadows has quite likely contributed to the cost problems, and it most certainly has contributed to security and compliance issues.
• Security risk. The public cloud is not inherently insecure, though each cloud resource must be appropriately configured if it is to be secure. In the 2019 Cloud Security Report, 93% of enterprises are moderately to extremely concerned about cloud security, so much so that 55% expect to deploy a new cloud security solution within the next 12 months⁴.

Change is a must. Managing our cloud operations by looking in the rear view is no way to run a business while you accelerate its reinvention. By the time a security leak reaches the news, or a 10x monthly cloud bill arrives, the financial damage is done. CloudOps must be predictive and proactive if it is to be successful. We need fewer tools, so that it is easier for application owners to manage their budget and security, while providing CloudOps the governance necessary to accelerate business agility on sensible guardrails.

There is a solution.Cloud Operations Management, from BMC, is a unified cloud management suite using policy-based governance to optimize cost and enhance security for cloud applications and workloads. App-level views, together with machine learning and built-in automation, enable service owners to proactively manage budget and security. No more budget crises. Radically diminished risk. Greater cloud ROI. With better cost performance and security posture, service owners have real incentives to work with the Cloud COE. Turns out, governance and agility are not mutually exclusive after all.

¹ SiriusDecisions Summit 2019, https://www.siriusdecisions.com/blog/myths-of-agile-new-aligned-process ↩
² Flexera 2019 State of the Cloud Report ↩
³ Gartner,https://www.gartner.com/en/newsroom/press-releases/2019-04-02-gartner-forecasts-worldwide-public-cloud-revenue-to-g ↩
⁴ Cybersecurity-Insiders, 2019 Cloud Security Report ↩

If you haven’t already taken Self-Driving Remediation (a new feature of TrueSight Cloud Security) for a test drive, then you probably are wondering what it means for you and your cloud security posture management.  Let me clarify before discussing why one should use it.

You know the difference between a car and a self-driving car.  In the former, you drive the car, making numerous, seemingly instinctive decisions as you manoeuvre through traffic.  In contrast, in self-driving mode – admittedly something most of us do not have a lot of personal experience with – probably all you need to do is tell the car the destination and then just sit back and relax, perhaps using your commute time for higher value activity than just moving from Point A to B.  With TrueSight Cloud Security, On-Demand Remediation and Self-Driving Remediation operate in similar fashion.

Difference between On-Demand Remediation and Self-Driving Remediation

As a cloud operations member or an application developer, one of your primary tasks is to ensure that the configurations of all cloud resources are consistent with security standards.  Depending upon how frequently you make changes to your cloud resources, you would login to TrueSight Cloud Security (TSCS) and look for any violations reported for the resources which you own. Then you go through each violation, look at the resources which have violated a specific rule, and then click the “Remediate Violations” button. This is On-Demand Remediation, an automated means of fixing cloud resource misconfigurations with a simple click of the mouse.

As our Cloud SecOps processes mature, we may become increasingly comfortable with remediating certain violations, under certain conditions, without any human intervention.  For non-production environments like DEV, TEST, or STAGE, we may even have a zero-tolerance policy for any violation and would like to remediate all violations immediately as soon as they are reported.  This fully automatic remediation gives a powerful mechanism to ensure that configuration security is assured and that any exceptions are documented, added to “Exceptions” in TSCS, and approved for production.  To enable Self-Driving Remediation in TSCS, simply set the remediation trigger “Auto” for specific policies.  In this case the remediation is automatically triggered when the violation is identified, thereby eliminating the human bottleneck and rapidly closing vulnerabilities from cloud resource misconfigurations.

Why Use Self-Driving Remediation?

First, it saves lot of time going through each known violation and clicking the Remediate Violation button.  Although automated, such repetitive tasks, while comforting in that they offer a sense of control, are begging to be handed over to the solution for completely automatic handling.  For example, if you know a specific violation must be remediated every time it is identified in certain cloud accounts, just enable the “Auto” remediation trigger from the violation page.  The time you save can be spent elsewhere.

Secondly, Self-Driving Remediation helps enforce certain security practices without fail.  You don’t have to wait for someone to log into TSCS to identify the violations and initiate remediation, or wait for someone to act on a compliance summary or new violation notification email.  As an enterprise, if you don’t want any S3 bucket to be publicly accessible, then simply go to the Manage Policy page and set the remediation trigger to “Auto” for that rule.  Anytime a violation is reported for that rule, remediation will be initiated then and there.  Doesn’t that bring you more peace of mind?

Next, if you worry that you may not want to perform same remedial action for all accounts, and for some you want a separate action to be invoked, you can do that too.  You can configure one remedial action for a certain group of accounts and another remedial action for another group of accounts.  You can even create your custom actions.

Finally, if you fear that you may not be aware of all the remediations that are happening, or you want an approval process to kick in for certain resources before they are acted upon, then let me assure you that you can enable change management for remediation.  In so doing, you will have a formal change approval process for your most critical applications.

In summary, you are still in control.  You decide the conditions in which Self-Driving Remediation takes the wheel.  So, take the plunge now and see for yourself if it saves your time and helps you concentrate on more challenging things and make your time count!

Those who operate in the enterprise tech space have a responsibility to ensure that new advances in technology comply with overarching standards and regulations like ISO, GDPR, HIPAA, and, of course, PCI. Indeed, any company that collects, holds or processes personal credit card information must adhere to PCI Payment Card Industry Data Security Standards, or PCI DSS. The PCI DSS helps thwart security breaches and identity theft across e-commerce platforms.

However, as it pertains to the use of containers with applications involving credit card data, little guidance had been provided until earlier this year when the PCI Security Standard Council created an updated version of their Cloud Computing Guidelines. The first update in five years, it’s an important one given the depth and breadth of utility that containers offer enterprise businesses and its fast-paced adoption.

Indeed, containers provide businesses with an easier, more standardized solution to digital service deployment, enabling them to leverage multiple cloud service platforms while streamlining their deployment process. These operational efficiencies become even more pronounced as enterprises scale their cloud operations.

Keep reading to learn why compliance is important to cloud computing, and how the update to these Cloud Computing Guidelines could impact your enterprise organization.

A Brief Look at the History of Containers and Cloud Computing

Containerization seems to have emerged rapidly over the last three or four years, but it’s a concept that had been in the works well before that. It was sometime around the turn of the Millennium that Linux developers started playing with the idea of creating a contained virtual server that could run in an isolated environment with both security and independence.

This project by Jacques Gélinas was called the VServer Project.

As a first-time effort, the project was groundbreaking but had some drawbacks. Ultimately, VServer technology lacked some important features of containerization with too much administrative overhead resulting from the requirement that VServer needed a patched kernel to run.

In 2007, Paul Menage of Google made containerization possible on an existing Linux kernel with minimal overhead. It was this development that served as a catalyst for container innovation that we see today.

For developers and cloud operations personnel alike, today’s container platforms solve an important issue: how to ensure software runs the same in different environments.

There are a number of scenarios where software in development may need to run in different environments. For instance:

• Throughout the software development lifecycle, microservices might transfer from a dev’s laptop to a test environment, or from Staging to Production.
• A developer may be testing on Debian whereas production runs on Red Hat
• A piece of deployed software may transition from a data center to a virtual server.

You get the idea. There are a number of environments in which software can be developed and used. And depending on the environment glitches and other unanticipated problems can occur.

Containers offer an all-in-one runtime solution where everything needed to run the application properly is bundled into one unit that can pass through different environments.

Types of Containerization

Containerization is a process by which an entire run-time environment can be isolated into a single package that doesn’t require its own OS kernel. They are lighter weight, startup more quickly than VMs, and use resources more effectively.

Capitalizing on existing technology, several container platforms exist for users today. These include:

• Docker
• Google Kubernetes
• Amazon AWS
• Cloud Foundry; and
• CoreOS’rkt

Using containers offers a number of benefits that were discussed at length here, and will be listed below. The benefits of developing in a container environment include the following:

• Faster startup
• Light weight, greater density
• Multi-cloud supported
• Offers testing and continuous deployment
• Version control
• Isolation and security
• Portability
• Easy to reproduce

While the implementation of containers in a DevOps environment is often straightforward, remaining compliant is not always as clear-cut.

Why Compliance is Important in Every Environment

When it comes to cloud computing, CIOs face numerous compliance concerns. Generally, these include:

Being Aware of Challenges to IT Workload

It isn’t enough for enterprise business leaders to look for typical compliance challenges when selecting a cloud vendor. Once the baseline compliance needs are met, IT departments must look at unique compliance needs and determine how the vendor will be able to add value while remaining compliant. Mapping these requirements to vendor controls can cause challenges in the IT workload.

Making Security Priority One

As an early adopter of new technology, the enterprise must make cloud security a top priority in order to remain compliant. While the cloud itself is not insecure, the way the enterprise uses it often leads to security and compliance violations. In fact, an overwhelming majority of business leaders cite security as their #1 concern with the public cloud. On the plus side, moving to the cloud may help enterprise businesses formalize the process of risk assessment audits for their organization. This trend can help enterprises become more compliant.

Tracking Changes in Standards

In addition to formalizing a risk audit process, organizations must be ready to track changes in standards as they occur such as the PCI updates which will be discussed below.

The Evolution of the PCI Standards

Every entity that accepts credit or debit cards, regardless of their size, must comply with the 250 regulations outlined by the PCI DSS. It’s important for companies to understand the significance of abiding by the PCI DSS rules and the consequences of non-compliance.

Initially, the rules were only applied to the credit card companies (i.e.. Visa, MasterCard, Discover, etc.). However, this quickly led to disparities in understanding how the standards should be adhered to. As a result, in 2004, the major credit companies came together to create a unified security standard or the first version of PCI DSS. Shortly after, the PCI Security Standards Council was born, an autonomous body responsible for continuing to build and manage the standards and educate entities (merchants) as well as banks as to their application.

Consequences of PCI Non-Compliance

In general, penalties have ranged from verbal warnings to large fines — in some cases up to $500,000. Such fines usually go hand-in-hand with increased transaction fees. And in the case of repeat offenders, PCI might revoke their privilege to accept payment cards altogether. As you might imagine, for many businesses, this would be a death sentence.

Compliance for Containers

As mentioned above, PCI has been slow to address how to use containers with applications involving credit card data. As a result, many organizations relied on container management companies who were PCI-compliant. However, as you are well aware, the onus rests on you as the business leader to ensure compliance.

This meant that organizations were left to internally audit their own container systems for compliance without the standardization of a compliance organization behind them.

But just because containers are new doesn’t mean they are entirely different than the enterprise systems that have powered development for many years. It stood to reason that the self-contained run-time environment can be audited just like any other cloud development environment that did have established guidelines. Existing compliance standards around security, confidentiality, availability, and integrity were used as a baseline for container compliance guidance.

To clear up any confusion, PCI finally updated their guidelines to include compliance for containers.

PCI Guidelines 3.0 Update

While the updated guidelines cover several issues that have arisen since the 2013 update, section E7 deals specifically with container compliance. Here are the PCI container guidelines your organization needs to be aware of and integrated into your overall risk management approach.

• Proper access points to orchestration framework and containers with special care being taken to protect sensitive information
• Process isolation with best practices listed
• Significant access restrictions to containers and container files
• Implementing a container-specific system-call firewall
• Separation between network and administrative containers with different workloads that meet certain protocols
• Kernel separation where possible
• A defined audit system where access logs can be generated, and other components can be reviewed
• Ensure strong CI/CD protocols with review of pipeline and change controls
• Secure management of repository storage for containers
• Containers that use open-source images should be patched and undergo vulnerability testing according to guidelines
• Whenever supported, read-only containers should be used

The above list provides a condensed look at the most important guidelines. However, we recommend that you download the full text click here. The document outlines various scenarios and examples of appropriate responses to ensure compliance.

Final Thoughts

Cloud and container security is challenging, if only because of the supercharged velocity of change. Qualified security talent is scarce, and so automation is all the more important to keep enterprise IT propelling the business forward is a secure and compliant manner. For enterprises adopting new technology, the importance of ensuring compliance cannot be overlooked. For most organizations, that means making sure security measures are met, being aware of potential challenges, and of course, staying on top of industry trends in compliance.

Recently, PCI made this last part a little easier by updating their cloud computing standards to take some of the guesswork out for organizations using compliance guidelines that pre-date containers themselves. This should be a gamechanger for organizations who want to pursue a DevOps approach.

Yes, we eat our own cooking.

Have you ever wondered how BMC Software keeps its cloud environments safe and secure? One of the proudest and most thrilling moments for our Cloud Engineering team was using our TrueSight Cloud Security, BMC’s very own automated cloud security and compliance solution, to achieve 100% compliance of our multiple cloud environments. Seeing that dashboard transform from red to green in such a short time was quite an achievement. In this blog, we describe how we run Dev environment security with TrueSight Cloud Security (TSCS). BMC being our first customer provides us direct feedback so that we continuously deliver an ever-improving solution.

The Challenge – You cannot manage what you don’t measure

We have thousands of cloud resources changing every day in our development cloud accounts as developers continuously push new functionality to prod. S3 buckets, firewall security groups, IAM roles, EC2 instances, and more are created or updated with daily pushes through DevOps pipelines. After each release, the big question on everybody’s mind was, “Are we secure? Did we mistakenly open a port to the internet?” Yes, just like your dev teams, we are constantly striving to increase our agile velocity; yet, we do not want to compromise on security. The old business axiom, “You cannot manage what you do not measure” is especially true for DevSecOps and cloud security. So, our first step was to benchmark our security posture and look for ways to fix high risk vulnerabilities, burning down our security backlog to secure our cloud resources… and keep them that way!

The First Security Scan

It took us less than an hour to assess our security posture, much to the delight of the team and exactly as we had envisioned when we set out to build our cloud security service. We pointed TSCS cloud service to our own AWS cloud accounts and started scanning them for CIS compliance with over 50 controls for AWS. Within a few minutes, we finished the setup, and the initial security posture data started lighting up on the dashboard. RED! To our surprise, about 15% of our resources were noncompliant to the CIS security benchmarks.

Figure 1: Example dashboard in TrueSight Cloud Security showing security at a glance

Thankfully, no S3 buckets were open to public, as so many highly publicized data breaches were caused by publicly accessible S3 buckets. With automation built-in by design, TrueSight Cloud Security can easily find and fix such open public buckets in minutes. We were, however, alerted that 50 S3 buckets did NOT have encryption turned on. Ouch. Encryption at rest is a best practice for data security that we needed to resolve. We also found 70 IAM policy rule violations, such as lack of MFA, lack of key rotation, and more. Key compromise is one of the ways that can lead to data or account breaches, and we knew this one also would need a quick resolution. Once we knew our complete security posture in our development accounts, the team started triaging, risk assessment, and putting together a plan for remediation. PRO TIP: Assess posture, triage results, and build a remediation plan, much like you would groom a Scrum backlog.

Instant Remediations

First things first: 70 critical IAM security violations were easy to fix, and the team remediated most of these through a single click from TSCS. Cloud Operations were confident about these not impacting Application and quickly remediated these. Nice going. As the engineering team kept deepening remediation action content, we soon realized the power of instant remediation with a single click from our UI. On a few others security issues, such as S3 buckets and EBS web tier encryption, the team created Jira tickets to track findings from TSCS and assigned them to developers. As our infrastructure is immutable, these issues were resolved through the DevOps pipeline updates to infrastructure. Within hours and days, we dramatically improved our security posture, with only 5% noncompliant rules remaining. Measuring security posture and reporting it in our dashboards and PDF reports motivated the team to continuously improve. PRO TIP: Gamification of security posture across multiple accounts and teams clearly leads to higher security as nobody wants to be at the bottom of the security grade.

Exceptions

The process of managing security issues can at first seem daunting but TSCS helps manage the volume of these issues very effectively. Powerful search capability available in the UI details security posture by application, services, owner, or account tags, context which prioritizes the most critical app and infrastructure issues for resolution first. Many issues can be remediated quickly through actions from UI or through incident integration process. There are always exceptions. After discussing with Security, a few findings were identified as acceptable low risks and were moved to the bottom of the backlog. Security is a risk management process where highest risk issues need to be fixed first, while low risk issues can be deferred and put on an exception list. This is where our last of remaining 5% issues ended up. We used TSCS to mark the last of our findings as exceptions, to reduce noise and “alert fatigue” while the Dev team added this to their technical debt. PRO TIP: Some security violations should not be resolved by Cloud Ops, but by Development. Use RBAC to provide remediation privileges to those who own the code. Use the UI to filter security by app, tag, accounts, etc.

Multiple Accounts

As we completed securing our first batch of accounts, we quickly realized that multi-account management is a challenge. Leveraging the multi-account connector within TSCS, we created trust relationships to child accounts, streamlining multi-account security management. We now began collecting security data for multiple accounts. Teams are now filtering security findings by account and environment, as well as visualizing the aggregate security posture across accounts. Many enterprises, including BMC, have 100’s, even 1,000’s of public cloud accounts, where this account consolidation capability enables security at scale and simplifies security operations. PRO TIP: Begin with the end in mind. Start small, securing a handful of accounts, and be prepared to use account consolidation to simplify your cloud security methods.

We are the Watchers on the Wall

Getting the first scans and securing our accounts is only the first step. Are we done? Of course not. Our next step was to ensure that the high level of security is continuously maintained.

• Daily reports automatically notify us of new security violations.
• We are also working on a self-driving remediation feature, to automatically fix certain vital security issues (which we define), thereby reducing the mean time to resolve.

“We are the watchers on the wall… Night gathers, and so my watch begins… For this night, and all the nights to come.” — #GameOfThrones, from Oath of the Night’s Watch

Take the Test Drive

Would you like to see what TrueSight Cloud Security can do for you? Take the free 14-day trial. Connect the service to your account. Kick the tires. See how it drives. Then, if you’re interested contact Sales.