Big Data Creates Big Challenges

Big Data is the big topic in IT and business today.  In a report recently published by McKinsey & Company, they state that use of big data by a retailer could increase its operating margin by more than 60 percent.  And that the US healthcare sector could create more than $300 billion in value every year with the use of big data.  These are just a few examples of the big dollar numbers they associate with proper analysis and use of big data. 


Along with the promise for better margins, more revenue, and improved operations, Big  Data also brings new stresses to the IT infrastructure.  The challenge of “extreme data management” is already an issue.  But IT also needs to look at its workload automation and make sure it is robust enough to handle the demands associated with Big Data and the needs of Business Intelligence it is there to serve. 


So what are the automation considerations in working with Big Data?  File transfers need to be scheduled for moving data to a central database or data warehouse.  This typically involves an Extract, Transform and Load (ETL) workflow as data is typically gathered from a variety of database types.  Once the data is moved to a central database, queries are scheduled from a variety of users using various applications to determine patterns in the data.  The frequency of the queries varies from business to business – it can be once a day, once an hour or near continuous.  And of course, as data gets added to the database and moved to new databases, there is the routine task of database management that needs to occur.


Sounds simple enough as these are all tasks that IT has been use to performing for years, so what’s changed.  Big data is driving complexity.  Just as Big Data is creating new challenges in data management, the same is true for workload automation.  The Workload Automation software must be able to work with a variety of operating systems, databases, applications and platforms (both physical and virtual).  And it must also be able to handle a variety of tasks and be smart enough to handle the dependencies for successfully completing one task in a workflow and know when to start the next one.  It needs to be able to understand business priorities and identify problems accordingly and give you the ability to quickly fix problems when they occur.  And of course – do all of this and satisfy IT audit and compliance requirements at the same time.


The Big Data movement is not only pushing  IT to find ways to do extreme data management, but it also pushing IT to do extreme workload automation.  Just like big data pushes data management tools to a new level – it also pushes IT workload automation to a new level.  Are you prepared?

Share This Post