The DevOps journey started with the idea of removing barriers between Development and Operations. Today DevOps has evolved into a set of practices and tools that are automating all aspects of application development as well as how the applications operate in production. Now companies are implementing advanced approaches using artificial intelligence (AI) techniques such as machine learning (ML) to maximize value. In this blog, I’ll share some new research from Enterprise Management Associates (EMA) and perspectives about the future of workload automation in DevOps. We’ll look at how enterprises are benefiting from combining AI/ML, DevOps and automation to reach new levels of productivity, quality, and reliability.

In its infancy, the Dev part of DevOps got most of the focus, with major investments made in continuous integration/continuous development (CI/CD) and infrastructure automation. In the last five years, the Ops side of the house has received its due focus, and the emergence of Workload Automation as part of DevOps practices is particularly noteworthy. This practice involves automating and orchestrating complex and interdependent application workflows and data pipelines to deliver critical business services in production. In fact, new research from Enterprise Management Associates (EMA) reveals 80 percent of enterprises have integrated the automation of application workflows into their DevOps processes instead of dealing with it in an ad-hoc style at the end of a release cycle.

If you are among the 80 percent of enterprises that have already integrated application and data workload automation into your DevOps initiatives, you don’t need me to explain the benefits. The results are clear. Building workload automation into applications, data pipelines, and business services when they are being created already has helped many organizations reduce their development cycles and time to production, increase reliability and performance, and free up time that can be spent creating new digital business capabilities instead of maintaining existing assets. Now we’re learning – from experience and new research – that those benefits can be enhanced through next-generation DevOps automation that infuses AI/ML into the process.

Let’s start by looking at the current state of workload automation in DevOps, which is itself pretty exciting. Today, 73 percent of companies are automating workflows while applications are being developed, according to the recent EMA study. This is often referred to as Jobs-as-Code. The automation of application workflows and data pipelines by treating jobs as code has become a mainstream practice in just a few years.  This is a testament to the proven value it provides, and how easy it is to adapt. Jobs-as-Code is practice that was pioneered by Control-M from BMC. Many of its early adopters such as Carfax were instrumental in proving its value to the market.

“DevOps is part of our culture and has been a very long time now. We’re a very time-to-market company, you really can’t afford to wait. It used to take weeks or months to get a process through development and into production. As we’ve moved more to the Jobs-as-Code, DevOps, open philosophy, we’ve really reduced that timeframe. We can actually go down to minutes.”
Robert Stinnett, Carfax

How will DevOps automation continue to evolve? It will become smarter, with help from AI/ML. While building application and data-pipeline automation into the DevOps process may have been a big leap for many organizations, introducing AI/ML should be more like a series of incremental steps. Leading workload automation products today provide policy-based automation, where automation engineers can set rules based on analytics from the workload automation tool’s own metadata.

For example, a policy for times of peak processing where compute utilization exceeds a threshold would have the workload automation product interface with an infrastructure automation solution to provision additional compute and process capacity there, and then terminate the provisioned resources when processing is complete. The natural evolution of this will be for workload automation solutions to have embedded AI/ML algorithms that can self-learn such patterns and then recommend remedial actions, which will take the system closer to a self-managed concept.

Sound too futuristic? Not to us, or to EMA. Analyst Steve Hendrick wrote: “ML will drive improvements in quality, which will drive wide-ranging benefits on application performance, reliability, OpEx efficiencies, cost savings, and customer satisfaction. AI will follow as a second wave of innovation built on ML that will enable expansive automation. This automation can mean improved speed in delivering new applications and updating applications, and improved employee productivity—all of which reduce SDLC cycle time and leave more time for addressing higher-order development work.”

The majority of IT operations professionals EMA surveyed believe introducing AI/ML to DevOps will improve application performance, application innovation, and design, and overall business revenue growth. Many developers agree, and the majority believe AI/ML will speed new application delivery. Both groups cited many other expected benefits.

The market isn’t quite there yet though. While many enterprises have placed high priority on using artificial intelligence and machine learning, those that are exploring their use for workload automation in DevOps are encountering some obstacles. Of course, there are the obvious suspects that typically tend to limit innovation – difficulties in setting strategy and finding skilled staff. But EMA’s research also found that, according to developers and IT operations staff,  a specific challenge to further automating DevOps is finding tools with AI/ML capability, and integrating them into the toolchain and processes. Look for future versions of workload automation solutions to have more built-in cognitive technology features to address these needs.

Bottom line, the application and data pipeline workload automation market is continually evolving to meet enterprise needs, and Control-M has been at the forefront. The current desire for artificial intelligence and machine learning provides a clue to where the market may go next. Stay tuned, and until then, visit the Control-M website to browse other analyst research reports, learn about the Jobs-as-Code approach, and see testimonials from customers that have brought workload automation to DevOps.

Want to learn more?

• Read EMA’s complete report – AI & ML: Driving the Next Generation in DevOps and Workload Automation

The “new normal” has accelerated our digital lifestyles at warp speed. Everything from the way we work, how we shop, and how we spend our leisure time has undergone a huge shift that may well be permanent.

Timelines for delivering new digital services that were months or years out—and already seemed tight—have shifted to a “get it done NOW!” schedule and frequently with a reduction, but certainly without any increase, in spend.

The Autonomous Digital Enterprise is a vision of the future state of business and the next evolution that will blend the use of automation with human roles. Companies that move toward it will most certainly thrive while those that fall short will fade or fail. There have been predictions that digital transformation laggards would be out of business by 2025 or that 50% of the Fortune 500 was going to disappear by 2030. The bad news is that in some cases they were wrong – for some firms it will be in 2020.

The good news is there’s still time to make changes. I’m pleased to announce Control‑M 20 is here and ready to accelerate your digital transformation now by:

• Delivering on the promise of faster time-to-market – Role-based administration lets IT delegate authority to developers and product teams so they can manage their applications independently. In addition, the continuing evolution of Automation API provides access for developers to new Control‑M capabilities using Jobs-as-Code.
• Managing costs and increase efficiency – Centralized connection profiles reduce time-consuming manual actions when deploying or modifying applications. Additionally, the web UI enhancements reduce training costs for new users.
• Enhancing business-to-business file exchange – The high availability solution now available with managed file transfer/enterprise (MFTe) ensures business continuity and improves external business partner data transfer reliability. MFTe LDAP integration further reduces administration costs when developing and deploying file transfer workflows.
• Generating business value from modern technologies –
  • Google Cloud services can be incorporated into application workflows with the Application Integrator, which now includes out-of-the-box service account authentication. This is in addition to existing support for AWS and Azure.
  • Additional highlights include Managed File Transfer support for bi‑directional transfers to and from AWS S3 buckets; usage of AWS Aurora DB for the Control-M database; and ongoing improvements for container support.

For more detailed information on the release, check out the release notes and the “what’s new” section for the complete list of features.

We’ve shared previews of Control-M 20 with customers during our pre-GA process; here’s what a few of them had to say about the latest version:

“RBA and AAPI will provide more freedom for customers to manage their connection (e.g. BANK users can update passwords without sharing them with our team).”
—Johann Vermeulen, IT Operations Analyst: ESFT, BMW Group South Africa

“Automation API enhancements will allow us to integrate Control-M with our enterprise applications. This will close the gap between all of our apps and data points and position Control-M as a central hub for process flows – as I feel it should be.   API Integration is the fast lane and Control-M 20 will make it easier to put the pedal to the metal.”
—Jonathan Spottswood, Tech Lead, Enterprise Workload Orchestration, GEHA 

“We plan to roll out Control-M usage to all our SAP personnel this year using the Control-M Web. I see that the expanded capabilities in Control-M 20 Web, especially in planning and monitoring, will enable us to accelerate and simplify on-boarding them all.”
—Bala Gampa, Enterprise Architect & SAP Technical Expert, NVIDIA 

“We can’t wait to increase automation with the new functions available via the API.”
—Martin Keey, IT-Automation Engineer, Foot Locker, Inc.

“As technology changes are constant, we continue to see that BMC Software is aligning Control-M to meet these fast-paced demands. We are excited to roll out more Application Integrator solutions and utilize Role-Based Administration to empower our internal customers with more control over their batch workflows”
—Senior IT Software Engineer, Fortune 500 Insurance Corporation

We’re continuing to innovate as we develop more functionality for you to leverage in Control-M.  We understand many of the challenges you face with changing technologies, focus on cost management, more demanding schedules, and a new set of critical initiatives. Continue to share these challenges with us and we’ll continue to deliver a solution that will help you meet them!