Workload Automation: Predicting Trends in 2016

In January 2014, I published a workload automation trends prediction for 2014 blog primarily focused on the impact of Gartner’s “Nexus of Forces.” These forces are often referred to as SMAC (Social, Mobile, Analytics and Cloud), widely recognized as the disruptive technologies driving the consumerization and industrialization “mega trends.” In 2016, some of these trends remain strong (with some fine-tuning). As expected there are some new trends emerging as well. The one theme that is reflected across all of my 2016 predictions is the evolution of workload automation to application workflow automation, accommodating the needs of both application developers and IT production control. Here’s my view on what you should be expecting.


Automate any application

In the 2014 prediction blog, I identified the trend of increasing the span of workload automation control to any application. This was mainly focused on the aspects of automatic workload discovery. This is still the first stage in the span of control evolution. But after you have discovered which applications are candidates for automation, you still need better and more powerful integrations to those applications to eliminate the pain, insufficient control, and poor visibility caused by the scripts being used to integrate those applications to your scheduler.

In 2016, the continuing explosion of applications driving digital transformation will increase stress on organizations to improve application workflow automation. Much of the source of this demand will come from the need to expose application specific workload information to business users and application developers via self-service web and mobile interfaces.

Workload automation vendors will answer this demand by providing application plug-ins or simple, intuitive tools to create such plug-ins. As organizations apply more DevOps processes to achieve greater agility and improve project time-to-value, they have attempted to fill the application to workload automation solution integration gaps by designing their own plug-in scripts. But using scripts to “teach” their workload automation solution how to start a new job, how to monitor it and how to determine if it completed successfully or failed is costly, risky and challenging to maintain, and still does not provide the access noted above for business users and developers. For this reason workload automation vendor-provided “plug-in designers” will become more important to help application developers and operations teams collaborate to deliver business value and project productivity.

Don’t fear, Big Data is here

With the increased number of big data projects going into production, and the need for them to be “production ready,” organizations will adopt enterprise-class workflow automation tools to improve project agility, speed delivery and enable ongoing change and innovation. Connecting mission critical big data applications with enterprise data sources requires the power and capability of an enterprise class workflow automation solution.

Directly analogous to the “any application” trend noted above, the ability to eliminate scripting in full production big data applications will not only improve agility, but will maintain that agility and flexibility as the ecosystem of emerging technologies continues to enable faster processing of data as well as simpler access to it.

Big data vendors – including the major Hadoop distribution leaders Cloudera, Hortonworks, and MapR, along with ETL and database providers, and big data systems integrators – will play an important role as they partner with workflow automation vendors to enable faster integration capabilities for their big data project. All evidence from successful big data projects to date indicates that when companies look ahead across the entire software development lifecycle stages to anticipate the demands of full production, the better they are able to deliver insights to drive business innovation.

In 2016 it will be easier for organizations to connect with others who have successfully “crossed the chasm” with successful big data use cases, and who will be willing to share their experience. I predict that many of those projects will include workflow automation as a critical success factor.

Bridging the gap between 2 speeds of IT

Automation is one of the main cornerstones of digital transformation as IT becomes a key driver behind the success of companies such as Uber, Airbnb, GoPro and many others. Competitive markets undergoing digital transformation demand organizations adopt agile and rapid development methods to support innovation. This has created the rise of DevOps.

Organizations express the urgency to go fast while simultaneously keeping good processes, standards, and other aspects of IT in place. Gartner calls these two seemingly opposing desires Bi-Modal IT. Mode 1 is the conservative, IT operations process and Mode 2 is the high speed innovation side of the coin. In 2016 there will be a greater need to bridge this gap.

From the workload automation perspective, this means organizations will be looking for collaborative ways to create and modify application workloads, and to promote them from development to production. They want the speed of continuous delivery with a process that is completely ‘industrialized’ into the enterprise workload automation solution.

To go fast, application developers adopt integrated development environment (IDE) plug-ins that will allow them to freely code and test their workloads without going through long and tiring approval processes. They will only submit their workloads to production control once they complete their coding and testing. Application developers will also look for the ability to easily migrate workloads that were developed and tested on a pubic cloud (AWS, Azure, etc.) to an on-premises production environment.

In 2016 application developers will be looking for workload automation solutions that allow them to create and control their workloads using standard APIs without the use of any graphical interfaces. At the same time, production control will be looking to manage the same workloads from traditional graphical client or web-based interfaces. Automation practices for microservices and containerization (such as Docker) will be adopted, starting with basic capabilities such as starting and stopping a container, and later on as a delivery vehicle for the workload automation solution components.

File transfers as a workflow

In 2015, many organizations still had dedicated teams that managed file transfers separately from “traditional” application workloads, even though both elements are part of the same business application workflow processes. This will probably remain the case for some companies over the next few years, but in 2016 the need to improve end-to-end visibility to the entire business service will grow. Organizations will look for integrated solutions that provide a single point of control and visibility with access to all stakeholders. Cost will play a major factor in the decision making process with the understanding that a single solution will reduce expenses (both from license and TCO perspectives) compared to separate products (for file transfer and for workload management), especially if they are offered by different vendors.

Additionally, organizations will look for best practices and tools to convert their existing file transfer definitions (in commercial, open source and homegrown products) to the integrated workload automation solution with minimal manual effort and risk. End users will want dashboards, reports, and viewpoints into their file transfer workflows from their integrated solution from a file perspective rather than from the traditional job basis.

In 2016, the automation of application workloads across the enterprise will become more critical. Traditional workload automation solutions will evolve to facilitate the different application workload requirements and needs of both application developers and IT production control. As the lines of responsibilities and areas of expertise between application development and IT operations continues to evolve, workload automation vendors must rise to this new business reality by creating an application workload automation solution as the collaboration platform, a bridge between the two speeds of IT, and as a key factor in a company’s ability to offer competitive digital services. It is going to be an exciting year!

BMC’s Complete Guide To Hadoop

  1. Introduction to Hadoop ›
  2. Hadoop Benefits ›
  3. Hadoop Ecosystem ›
  4. Hadoop Security ›
  5. Hadoop Hiring & Careers ›
  6. Hadoop Resources ›

These postings are my own and do not necessarily represent BMC's position, strategies, or opinion.

Share This Post

Tom Geva

Tom Geva

Tom Geva is an IT Workload Automation expert. Tom served as a Control-M product manager for more than 10 years and was responsible for translating the workload automation market requirements gathered from customers and analysts into product and business strategies, roadmap and specifications. Tom is deeply involved in the development process of all the Control-M releases. As a Sr. Solution Marketing Manager he frequently speaks in conferences and workload automation events. Tom holds 19 years of experience in the IT industry. Prior to joining BMC Software in 2001, Tom spent 4 years as a production control manager in the Israeli Defense Force and was responsible for workload automation and mainframe education services.