Researchers have discovered a new way that hackers can potentially exploit systems to expose passwords, keys, and other sensitive data – and this time it involves the physical hardware. (Source: NYTimes) These new vulnerabilities, called Spectre and Meltdown, were discovered in Intel processors, and more specifically the way that the processors anticipate requests and cache data to speed up processing. Intel believes that it can solve the Meltdown, and one of the Spectre vulnerabilities with a software update, but because the vulnerability is in the CPU chip design, it will need to be addressed in the next generation of CPUs. This is not an easy fix and as a result, the underlying vulnerability will likely remain in organizations for years to come.
Two things make this vulnerability extra challenging:
- The pervasiveness of the vulnerability across servers, devices and operating systems is nearly unprecedented.
- Since this vulnerability impacts a feature that improves performance, there is a potential significant performance hit from applying the software patches.
Intel has said that the performance degradation is minimal for most computer users and that updates to be provided will lessen the impact over time. However, according to The Register, [for high-performance systems], “Machines hammering disk storage, slamming the network, or otherwise making lots of system calls, may experience up to 30 per cent reduction in performance.” As a result, organizations might have to make a critical trade-off to patch a potential security risk and incur an expected performance hit.
As of January 8th, there are no known Spectre / Meltdown exploits of the vulnerability in the wild. In addition, Intel has released some fixes that they claim will make their processors “immune” to this vulnerability, with more due later this week to cover ~90% of all CPUs made in the last 5 years. Microsoft, Google, Linux and other software vendors are also quickly releasing patches to their operating systems and applications to mitigate the issue, essentially making it much harder to exploit the underlying vulnerability.
While this security vulnerability challenge may be unique in that it originates at the CPU, the fixes are essentially the same as for other vulnerabilities that can be solved with software updates / patches. The real challenge for most organizations is to effectively apply their patching process across a multitude of tools and teams to correct the systems before hackers start to exploit the now-public vulnerabilities.
As with any software patch, organizations should follow their defined patching process which typically involves the following phases. Here are some questions for each phase to help you better plan for Spectre and Meltdown remediation.
- Assessment – Does the vulnerability pose a significant risk to the organization in terms of data breach, potential downtime or loss of business reputation? With Spectre and Meltdown, that answer is yes, though with no known exploit published yet, it is uncertain how successful hackers might be. What is the cost to your organization from applying the patch in terms of time and effort and maintenance downtime? Some of the Spectre/Meltdown patches can impose a performance hit of between 5 and 30%, depending on the kind of workload. (Source: The Register) If true, do you have the ability to scale up resources to offset the loss in performance? Some systems carry higher risk of being exploited. Do you have good visibility into which systems are internet-facing (in a DMZ), running critical business systems, or multi-tenant applications? Typically DMZ-hosted applications are patched first, while critical business systems are often prioritized ahead of utility or less-critical systems.
- Packaging – While some vulnerabilities can be closed with relatively simple patches, there are often many different patches needed, each applying to a specific level or architecture of the operating system. It’s important that the systems receive the correct patch for their specific operating system variant. With Spectre / Meltdown, there are patches currently available, or patches coming, from each OS vendor; from Intel; from VMware; and even from application vendors. Deployment orchestration should also include any pre- or post- processes, and tie into reboot or cluster management processes that ensure end-to-end success. These can vary by target system.
- Testing – Software changes, including patches and the pre- and post-processing, should be tested before releasing into production environments to ensure stability and success. Testing of these patches is critical given that: a) these patches may directly interact with the CPU firmware, and b) there have been some reported instances of “bricking” of systems and other unintended effects.
- Targeting – Do you know which critical business systems are most vulnerable and should therefore receive the patch first? Second? Do you know the maintenance windows for each system group so that you schedule appropriately? Can you use that information so that a complicated environment doesn’t “cost extra” to patch?
- Change approval – A change ticket should have been opened by now with all of the details about the change. Are the proposed changes and plans for execution approved by the right IT and application owners? If the patch hasn’t been applied in say, 30 days, can you require senior-level IT approval for the exception and if not, force the update? Can you quickly report on which business units are in compliance vs. lagging behind?
- Deployment – Did the changes execute as planned? What systems didn’t update properly? Can they be resolved automatically or do they need manual attention? Can you quickly assess overall success and “how things are going” without individual inspection?
- Change tracking – Close the change ticket and report on the success of the effort including compliance with regulations such as PCI-DSS, SOX or HIPAA. If only the high-risk systems were patched, what is the plan to deal with the remaining systems? Can you validate that changes were only made in the window, document the effort and results back into the change, and identify any out-of-window activity?
Fortunately, BladeLogic Server Automation (BSA) and SecOps Response Service (SRS) are built to help solve issues just like these.
- With cross-platform support, you can easily patch Linux and Windows systems that may be running on vulnerable CPUs, whether on-premises or in public cloud (IaaS) servers.
- SRS and BSA can help you quickly understand where the vulnerabilities exist in your environment, create remediation plans and execute against operational constraints.
- Through integration with BMC Discovery, you can better understand which applications and systems are critical to the business and therefore need to be patched quickly.
- BSA pulls in the Critical Vulnerability Exposures (CVEs) – CVE-2017-5754 for Meltdown and CVE-2017-5753, CVE-2017-5715 for Spectre – to help you quickly analyze and prepare remediation packages that include vendor patches.
- BSA gives you the tools to create custom steps such as pre- or post-processing, pre-flight checks, deployment, change ticket creation, documentation, closure, execution validation, and reporting.
- Compliance with regulations can be easily documented showing where the changes/patches have been applied and monitoring those systems for other changes that could impact compliance.
These postings are my own and do not necessarily represent BMC's position, strategies, or opinion.