In today’s fast-paced digital landscape, the ability to harness the power of artificial intelligence (AI) and machine learning (ML) is crucial for businesses aiming to gain a competitive edge. Amazon SageMaker is a game-changing ML platform that empowers businesses and data scientists to seamlessly navigate the development of complex AI models. One of its standout features is its end-to-end ML pipeline, which streamlines the entire process from data preparation to model deployment. Amazon SageMaker’s integrated Jupyter Notebook platform enables collaborative and interactive model development, while its data labeling service simplifies the often-labor-intensive task of data annotation.

It also boasts an extensive library of pre-built algorithms and deep learning frameworks, making it accessible to both newcomers and experienced ML practitioners. Amazon SageMaker’s managed training and inference capabilities provide the scalability and elasticity needed for real-world AI deployments. Moreover, its automatic model tuning, and robust monitoring tools enhance the efficiency and reliability of AI models, ensuring they remain accurate and up-to-date over time. Overall, Amazon SageMaker offers a comprehensive, scalable, and user-friendly ML environment, making it a top choice for organizations looking to leverage the potential of AI.

Bringing Amazon SageMaker and Control-M together

Amazon SageMaker simplifies the entire ML workflow, making it accessible to a broader range of users, including data scientists and developers. It provides a unified platform for building, training, and deploying ML models. However, to truly harness the power of Amazon SageMaker, businesses often require the ability to orchestrate and automate ML workflows and integrate them seamlessly with other business processes. This is where Control-M from BMC comes into play.

Control-M is a versatile application and data workflow orchestration platform that allows organizations to automate, monitor, and manage their data and AI-related processes efficiently. It can seamlessly integrate with SageMaker to create a bridge between AI modeling and deployment and business operations.

In this blog, we’ll explore the seamless integration between Amazon SageMaker and Control-M and the transformative impact it can have on businesses.

Amazon SageMaker empowers data scientists and developers to create, train, and deploy ML models across various environments—on-premises, in the cloud, or on edge devices. An end-to-end data pipeline will include more than just Amazon SageMaker’s AI and ML functionality, where data gets ingested from multiple sources, transformed, aggregated etc., before training a model and executing AI/ML pipelines with Amazon SageMaker. Control-M is often used for automating and orchestrating end-to-end data pipelines. A good example of end-to-end orchestration is covered in the blog, “Orchestrating a Predictive Maintenance Data Pipeline,” co-authored by Amazon Web Services (AWS) and BMC.

Here, we will specifically focus on integrating Amazon SageMaker with Control-M. When you have Amazon SageMaker jobs embedded in your data pipeline or complex workflow orchestrated by Control-M, you can harness the capabilities of Control-M for Amazon SageMaker to efficiently execute an end-to-end data pipeline that it also includes Amazon SageMaker pipelines.

Key capabilities

Control-M for Amazon SageMaker provides:

• Secure connectivity: Connect to any Amazon SageMaker endpoint securely, eliminating the need to provide authentication details explicitly
• Unified scheduling: Integrate Amazon SageMaker jobs seamlessly with other Control-M jobs within a single scheduling environment, streamlining your workflow management
• Pipeline execution: Execute Amazon SageMaker pipelines effortlessly, ensuring that your ML workflows run smoothly
• Monitoring and SLA management: Keep a close eye on the status, results, and output of Amazon SageMaker jobs within the Control-M Monitoring domain and attach service level agreement (SLA) jobs to your Amazon SageMaker jobs for precise control
• Advanced capabilities: Leverage all Control-M capabilities, including advanced scheduling criteria, complex dependencies, resource pools, lock resources, and variables to orchestrate your ML workflows effectively
• Parallel execution: Run up to 50 Amazon SageMaker jobs simultaneously per agent, allowing for efficient job execution at scale

Control-M for Amazon SageMaker compatibility

Before diving into how to set up Control-M for Amazon SageMaker, it’s essential to ensure that your environment meets the compatibility requirements:

• Control-M/EM: version 9.0.20.200 or higher
• Control-M/Agent: version 9.0.20.200 or higher
• Control-M Application Integrator: version 9.0.20.200 or higher
• Control-M Web: version 9.0.20.200 or higher
• Control-M Automation API: version 9.0.20.250 or higher

Please ensure you have the required installation files for each prerequisite available.

A real-world example:

The Abalone Dataset, sourced from the UCI Machine Learning Repository, has been frequently used in ML examples and tutorials to predict the age of abalones based on various attributes such as size, weight, and gender. The age of abalones is usually determined through a physical examination of their shells, which can be both tedious and intrusive. However, with ML, we can predict the age with considerable accuracy without resorting to physical examinations.

For this exercise, we used the Abalone tutorial provided by AWS. This tutorial efficiently walks users through the stages of data preprocessing, training, and model evaluation using Amazon SageMaker.

After understanding the tutorial’s nuances, we trained the Amazon SageMaker model with the Abalone Dataset, achieving satisfactory accuracy. Further, we created a comprehensive continuous integration and continuous delivery (CI/CD) pipeline that automates model retraining and endpoint updates. This not only streamlined the model deployment process but also ensured that the Amazon SageMaker endpoint for inference was always up-to-date with the latest trained model.

Setting up Control-M for Amazon SageMaker

Now, let’s walk through how to set up Control-M for Amazon SageMaker, which has three main steps:

1. Creating a connection profile that Control-M will use to connect to the Amazon SageMaker environment
2. Defining an Amazon SageMaker job in Control-M that will define what we want to run and monitor within Amazon SageMaker
3. Executing an Amazon SageMaker pipeline with Control-M

Step 1: Create a connection profile

To begin, you need to define a connection profile for Amazon SageMaker, which contains the necessary parameters for authentication and communication with SageMaker. Two authentication methods are commonly used, depending on your setup.

Example 1: Authentication with AWS access key and secret

Figure 1. Authentication with AWS access key and secret.

Example 2: Authentication with AWS IAM role from EC2 instance

Figure 2. Authentication with AWS IAM role.

Choose the authentication method that aligns with your environment. It is important to specify the Amazon SageMaker job type exactly as shown in the examples above. Please note that Amazon SageMaker is case-sensitive, so make sure to use the correct capitalization.

Step 2: Define an Amazon SageMaker job

Once you’ve set up the connection profile, you can define an Amazon SageMaker job type within Control-M, which type enables you to execute Amazon SageMaker pipelines effectively.

Figure 3. Example AWS SageMaker job definition.

In this example, we’ve defined an Amazon SageMaker job, specifying the connection profile to be used (“AWS-SAGEMAKER”). You can configure additional parameters such as the pipeline name, idempotency token, parameters to pass to the job, retry settings, and more. For a detailed understanding and code snippets, please refer to the BMC official documentation for Amazon SageMaker.

Step 3: Executing the Amazon SageMaker pipeline with Control-M

It’s essential to note that the pipeline name and endpoint are mandatory JSON objects within the pipeline configuration. By executing the “ctm run” command on the pipeline.json file, it activates the pipeline’s execution within AWS.

First, we run “ctm build sagemakerjob.json” to validate our JSON configuration and then the “ctm run sagemakerjob.json” command to execute the pipeline.

Figure 4. Launching Amazon SageMaker job.

As seen in the screenshot above the “ctm run” command has launched the Amazon SageMaker job. The next screenshot shows the pipeline running from the Amazon SageMaker console.

Figure 5. View of data pipeline running in Amazon SageMaker console.

In the Control-M monitoring domain, users have the ability to view job outputs. This allows for easy tracking of pipeline statuses and provides insights for troubleshooting any job failures.

Figure 6. View of Amazon SageMaker job output from Control-M Monitoring domain.

Summary

In this blog, we demonstrated how to integrate Control-M with Amazon SageMaker to unlock the full potential of AWS ML services, orchestrating them effortlessly into your existing application and data workflows. This fusion not only eases the management of ML jobs but also optimizes your overall automation processes.

Stay tuned for more blogs on Control-M and BMC Helix Control-M integrations! To learn more about Control-M integrations, visit our website.

Introducing a new Control-M and AWS integration

Business modernization is a strategic priority for almost every company that wasn’t born digital native. As executives look to transform their IT environments to deliver better customer and employee experiences, cloud adoption is soaring. The benefits are well documented and clear. However, many organizations have heavily invested in large mainframe environments with critical systems of record. According to Precisely, mainframes handle 68 percent of the world’s production IT workloads. Migrating application and data workflows across complex hybrid-cloud environments isn’t simple.

Because mainframes are deeply embedded into organizations’ tech stacks, changes require extensive examination of many objects. This must be carried out very carefully, generally over long periods of time. That puts companies in the difficult position of having to run old and new workflows simultaneously. Many organizations that use mainframes also operate in heavily regulated industries and are subject to strict governance. That means many, if not all, tasks must be checked, tested, and re-checked to ensure compliance and mitigate risk. And finally, mainframe environments have often been in production for many years. Lots of institutional processes and knowledge have built up over time. Mainframe modernization requires both procedural and cultural shifts. However, companies are not alone on this journey.

BMC works with organizations around the world to address these challenges and successfully deliver on their modernization projects. As an enabler of the Autonomous Digital Enterprise (ADE) framework, our solutions make it easier for companies to manage their constantly evolving tech stacks in an agile fashion to better support stakeholders. Control-M, BMC’s market-leading application and data workflow orchestration platform, gives developers, data engineers, and business users freedom to innovate securely across mainframe and cloud environments within a secure orchestration framework.

To help organizations better address their mainframe modernization challenges, we’ve partnered with Amazon to launch a new Control-M integration with the AWS Mainframe Modernization Service.

Workflow orchestration meets mainframe modernization

Companies can leverage Control-M’s deep operational capabilities with the AWS Mainframe Modernization Service to preserve the continuity of mission-critical business outcomes delivered by automating application and data workflows in production, across distributed, hybrid, and mainframe environments. Control-M easily integrates, automates, and orchestrates new applications and technologies with interfaces for IT operations, data engineers, developers, and business users across their on-premises and AWS platform.

Gur Steif, president of digital business automation at BMC, summed it up well: “Our Control-M customers are focused on driving modernization initiatives and delivering transformative digital experiences for their external and internal customers. We are proud to collaborate with AWS on this very important leg of that modernization journey.  With this new integration, our customers can modernize their mainframes and utilize the power of AWS to give stakeholders freedom to collaborate, to turn data into actionable insights faster, and to leverage the latest AWS services using the Control-M platform.”

The AWS Mainframe Modernization Service helps modernize mainframe applications to AWS cloud-native managed runtime environments. It provides tools and resources to help plan and implement migration and modernization. With the integration, job creation and monitoring can be performed entirely via any Control-M interface, which enables organizations to manage AWS Mainframe Modernization Service jobs just like any other Control-M workload.

In addition, users can submit or cancel batch jobs and review the details of batch job runs. Each time a user submits a batch job, the AWS Mainframe Modernization Service creates a separate batch job run, which can easily be monitored. Using AWS Mainframe Modernization Service web console, users can search for batch jobs by name, provide job control language (JCL), script files, and parameters to batch jobs.

The integration also helps companies:

• Reduce talent gaps
• Support rapid innovation with an agile DevOps approach
• Provide easier access to applications and data without significant changes
• Optimize the costs of running or extending applications
• Maximize business agility

Partnering for success

BMC’s Global Outsourcer System Integrator organization and global partner network will collaborate with companies to help them build a solid mainframe modernization strategy, and to determine the right type of migration for each organization. They’ll also help companies best leverage Control-M throughout the journey.

I recently spoke to Raul Ah Chu, BMC’s Global VP of Sales for the Global Outsourcer System Integrator organization. He is very excited to bring the power of our GSI partner community and their business modernization expertise to our customers.  BMC and AWS are working closely with our GSI partners to ensure that companies wishing to take advantage of the AWS Mainframe Modernization Service can do so while maintaining all the benefits of their Control-M platform.

In addition to our AWS partnership, companies will be able to leverage AWS’ Prescriptive Guide, which includes time-tested strategies, best practices, and guidance to help accelerate cloud migration, modernization, and optimization projects. In it, experts from AWS and its partners share practical real-world experience. The guide will help companies navigate the complex cloud landscape with specific approaches through how-to guides, step-by-step tasks, architecture, and code.

The path forward

With Control-M and the AWS Mainframe Modernization Service, companies have the right tools and partners to navigate the migration path at the speed their business requires (whether replatforming, refactoring, or both are required). Control-M will continue to manage all critical business workflows with all the deep operational capabilities organizations depend on. And once they start their cloud migration journey, Control-M will enable them to seamlessly manage all business workflows on a single pane of glass, mainframe to cloud. This gives them a single strategic platform to give all stakeholders the freedom to drive business outcomes faster within a secure orchestration framework.

Additional Control-M integrations with AWS

This adds to an already robust list of Control-M integrations available for the AWS ecosystem including AWS Lambda, AWS Step Functions, AWS Batch, S3 Buckets, AWS Glue, and AWS Databricks. Companies that are re-platforming mainframe applications using the AWS Mainframe Modernization Service can seamlessly orchestrate application and data workflows running in the re-platformed environment while managing dependencies with applications that may still be on-premises.

For more information about this integration please visit the AWS Prescriptive Guidance: Using Control-M workflow orchestrator integration with AWS Mainframe Modernization.