Workload Automation Blog

Common Challenges with Big Data Deployments

2 minute read
Raphael Chauvel

Common_Challenges_Big_Data_DeploymentsWith technology that enables the collection, storage, and analysis of an incredible quantity of data, business demand for big data deployments has moved from experimentation to production. At the same time, IT needs to keep integration with existing business systems in mind. For instance, structured data remains the main focus, so IT needs to keep traditional storage and analytics solutions in place while exploring big data. Read BMC’c Hadoop Tutorial for Beginners ›

We are at the beginning of mainstream adoption and eventually every organization will want to benefit as the lines of business push for better access to big data. Today, adoption varies across industry segments as some use cases are more predominant than others. IDG’s 2015 Big Data and Analytics study highlights data-driven initiatives and strategies that are driving these big data investments. This chart published by IDC illustrates users by sector and investment plans for 2015.

IDC Big Data

No matter which sector you’re in, these are some common challenges that IT teams face in managing big data deployments:

  • Avoiding too many siloed big data initiatives
  • Ensuring the team is staffed with the appropriate skills to address new challenges
  • Guaranteeing secure storage and access to unstructured data
  • Monitoring big data services and optimizing an ever-growing capacity
  • Collaborating across functions and businesses as the ownership of data is fragmented across the organization
  • Keeping up with a rapidly evolving technology portfolio

Discovery and dependency mapping solutions address these big data challenges to give IT confidence, knowing precisely the matrix of configurations. The ability to have in-depth information on every server, storage, network switch, load balancer, and firewall, along with software versioning visibility and knowing what is running and what is idle, will make any IT representative’s job much easier. Software license audits and identifying extraneous software that is running are prime examples for needing discovery and dependency mapping.

A robust discovery and dependency mapping solution will offer broad coverage of various components that need to be managed in a big data environment, including:

  • Technology frameworks
    • Apache: Ambari, Cassandra, Chukwa, Falcon, Flume, Hadoop, Hbase, Hive, Hue, Oozie, Pig, Solr, Spark, Sqoop, Tika
  • Infrastructure
    • Databases: VoltDB, MongoDB, Informatica Powercenter, Terrastore, Datastax, Basho, Oracle, MySQL, MSSQL, memSQL, Teradata, SAP HANA
    • Storage: EMC, NetApp, IBM, HP, Dell, Hitachi
    • Analytics: Cloudera, MapR, HortonWorks, Pivotal, IBM Netezza
  • Applications
    • Business intelligence: Hyperion, Business Objects, Autonomy, IBM Infosphere, Tibco
    • Operational intelligence: New Relic, Sumologic

Getting a holistic view of your IT systems and how they deliver business services is invaluable as you plan out your big data deployment. Application discovery and mapping along with powerful analytics will arm your team with the data needed to manage the inevitable challenges of such an immense project.

Automate workflows to simplify your big data lifecycle

In this e-book, you’ll learn how you can automate your entire big data lifecycle from end to end—and cloud to cloud—to deliver insights more quickly, easily, and reliably.

These postings are my own and do not necessarily represent BMC's position, strategies, or opinion.

See an error or have a suggestion? Please let us know by emailing

BMC Bring the A-Game

From core to cloud to edge, BMC delivers the software and services that enable nearly 10,000 global customers, including 84% of the Forbes Global 100, to thrive in their ongoing evolution to an Autonomous Digital Enterprise.
Learn more about BMC ›

About the author

Raphael Chauvel

Raphaël Chauvel is a Director of Product Management at BMC Software and has 20 years of experience in developing and marketing products in the areas of Data Center Discovery, IT Service Management, and Client Management. Raphaël is currently leading product management for BMC Atrium CMDB, BMC Helix Discovery, and BMC Helix Client Management. His team focuses on managing product strategy, ensuring product lifecycle aligns with customer and business needs, and that BMC and partners are enabled to these products. He joined BMC in February 2012 with the acquisition of Numara Software and currently resides in France.