Part 1 of this 4-part series defined TDM and explained why it’s crucial to the success of your organization. Part 2 identified key stakeholders and Part 3 described some of the challenges associated with implementation of a TDM strategy. In the conclusion of the series, Principal Consultant Bill Clayton details BMC AMI DevX TDM solution and describes the benefits of implementing a TDM strategy.

BMC AMI DevX offers a comprehensive Test Data Management solution which includes an end-to-end combination of the technology, expertise, and best practices needed to support a data protection and optimization initiative across the enterprise. By implementing a TDM solution, an organization can reduce its risk of exposure, increase productivity, and reduce the overall test data footprint, all while lowering the cost of managing data.

BMC AMI DevX Test Data Management Best Practice is one of the key components of the solution. It provides the guiding principles to build or enhance the process of preparing secure data environments to support the IT functions related to the usage and delivery of data in production-like conditions such as testing, training, etc. It allows for right-sized, desensitized, on-demand data. The solution allows for the diverse technical, organizational and political challenges associated with the protection of proprietary information that project teams typically encounter. The methodology outlines the phases, activities, tasks, and deliverables that encompass the implementation of a complete data management solution within the context of an existing Quality Assurance Life Cycle.

The purpose of the BMC AMI DevX Data Management Best Practices is to facilitate the initiation and execution of a data management endeavor. This is accomplished by relieving the pains related to the scoping, planning, organizing, managing, controlling, and delivering data environments with fictionalized sensitive information. This approach is based upon the implementation of proven software technologies; in particular, with the BMC AMI DevX File-AID suite of data management products, as well as Topaz and BMC AMI DevX Data Studio. This suite of tools enables an organization to ensure that successful data subsetting and de-personalization procedures are employed across the enterprise.

Analysis

The key to a successful Test Data Management solution is to know your data. Having an up-to-date data model is crucial.

Data model analysis documents a representation of the information assets utilized by the business. It is a commonly used documentation tool that fully describes the data components of an application system. It identifies all of the physical data structures, their attributes, and their relationships to all of the other structures in the system. The goal of the data model analysis activities is to provide knowledge about the environment’s data, determine the elements that are considered sensitive, and to define their association to other data objects.

Function Model Analysis identifies and documents information about the application processes. It determines what business rules and logic apply to the data considered sensitive or private, and it outlines the expectations of how the affected data should be changed as a result. Function Model Analysis is also important in order to understand any/all data validations and checks done against sensitive fields within the application programs. This is a critical step so that any fields having undergone disguise treatment maintain the same validation results after being disguised. All enterprises have their own list of data elements to be considered for obfuscation, but some of the most often chosen candidates are:

1. Driver’s license number
2. Names
3. Address
4. Date of birth
5. Telephone number
6. Tax identifier

The analysis documents are peer and customer reviewed for acceptance and sign-off.

Design

The design describes the processes to define the strategies for the extraction of the data to be protected, the disguise techniques to be applied to the sensitive data included within the extract(s), and the considerations to be observed when loading the extracted and disguised data into its target environment(s).

The design should be generated by following the framework of the specific details identified and documented in the previous analysis phase. This should include details such as the physical locations of where all of the data and corresponding structural layout information are located, how that data is used by both the applications that interface with it and any testing procedures and processes that impact it, and most specifically, where all of the sensitive data elements are contained within the data necessary to build a fully functioning test data environment that emulates the production system(s) it supports.

The primary purpose of the design phase is to chart a strategy for the extract of the necessary test data, the disguise of the sensitive fields/elements contained therein, and finally the subsequent load of all of the combined data. This should be done while taking into consideration the specific requirements, dependencies and restrictions discovered during the analysis phase.

The deliverables of the design tasks are designed to provide written specifications that will serve as the basis for the development of specific data management rules and processes. In the context of BMC AMI DevX Test Data Management Best Practices, design information is documented to address the BMC AMI DevX File-AID and Topaz product suite specifically, which provides the data management capability.

Development

This is the process to build and test the extracting and/or subset of related data from a source environment, obfuscating sensitive elements, and loading the obfuscated data to a target environment.

The major source of input information for the tasks to be performed during the development phase of a TDM project is the documentation produced during the design phase. Such deliverables should serve as detailed instructions for programmer/analysts to build and test these processes.

Delivery

A critical success factor for delivery is to fully understand the application QA life cycle, the processes in place, the testing requirements, the roles of the QA organization, etc. This is part of the information collected during the analysis phase. To some extent, the delivery phase is similar to rolling out a business application to production. But in the case of a TDM solution, deployment is made to the different QA environments where data needs to be protected, and delivery of the extract, disguise, and load processes is fitted within the existing QA cycle.

It is critical to prepare for delivery of the solution by making sure that appropriate access to documentation and human and technical resources is readily available. This phase involves, in addition to the data management implementation group, the integration of a multi-disciplinary team including Security, Applications, Quality Assurance, Auditing, Systems, DBAs, Architecture, Technical Support, etc.

The Outcome

At the conclusion of the delivery phase then the ‘implementation’ is over. The new TDM solution must be owned and managed to keep it current. Following are some of the benefits you can expect from a robust TDM strategy:

1. The analysis document doubles as an inventory for where sensitive data is stored. Such an inventory is often required by legislation.

2. A better understanding of the enterprise data and how it all works together.

3. Knowledge of where unwanted and/or unneeded data resides.

4. A suite of jobs that can be run when required, on a schedule or on an on-demand basis.

5. Cost savings in the form of reduced DASD can be substantial. A major US bank reported savings of millions of dollars when they reduced their test data footprint down to 20% of its original size.

As you can see, proper Test Data Management, guided by a robust strategy and comprehensive solution, helps mitigate risk, ensures regulatory compliance, addresses the concerns of stakeholders, and increases productivity, all while helping to reduce cost.

Parts 1 and 2 of this 4-part series defined TDM, explained why it’s crucial to the success of your organization, and identified key stakeholders. Part 3 covers some of the challenges associated with implementation of a TDM strategy.

The challenges of implementing a TDM strategy come in many different forms. Following are some of the more common that I’ve seen.

Corporate Acquisitions. Many of the businesses that I have worked with have been banks, and banks especially have a lot of acquisition activity over the course of their evolvement. Some acquisitions may have happened many years or even decades prior to a TDM initiative and would have brought their own data conversion challenges at that time. Still existing, badly mapped data from conversions may be inevitable. Extensive data sampling needs to be completed as a part of an initiative to enable a strategy to handle this.

Naysayers. Among other roles, users especially, or application testers will express doubts about their ability to do their job if their data is obfuscated. In my years of working on TDM projects I have not come across anyone who was unable to do their job after a planned TDM initiative.

Names and Addresses. There may be many recommendations about what to do with names and addresses. Experience tells me that older enterprises inevitably have some anomalies in this area. Users may believe that all of their addresses on file are “valid,” but that is not usually true. Over time, application dBases will have added columns and tables, and names and addresses will have been keyed in by many different operatives across the business, in different formats. In the past little guidance or validation was employed at entry time and names and address fields often contain “difficult” data.

It is possible that there will be names on address lines, and it is probable that there will be multiple names on some name fields.

Analysis of the data contained in these columns/tables needs to be conducted at the start of a TDM initiative.

Proprietary Compressed Data. Many organizations chose to buy software packages to help run their business processes. Some of these software packages store data in a compressed state and that data needs to be uncompressed where necessary. Working with the vendors of such data may be a part of a TDM initiative.

Related Data and Referential Integrity. Data provisioning needs to be able to obtain subsets of related data on-demand. A subset may be driven by any defined element such as account number, customer number, or a range of numbers. Defining the tables or files needed in the extract should be part of a process and working with SME’s, DBA’s etc. is required. Knowing your data is key!

There are challenges to implementing a robust TDM strategy, but with proper planning and preparation, they can easily be overcome. In the conclusion of the series, I’ll discuss TDM solutions, how they can help you, and the results of implementation.

In Part 2 of this 4-part series, we discuss the stakeholders interested in implementing a TDM strategy.

In Part 1 of this series we discussed TDM and why it is crucial to your organization’s success. A robust TDM strategy includes more than consideration of risk mitigation and legal compliance—it must also address the concerns of stakeholders within your organization and those of your customers. Following are some of the roles that may have a stake in what an organization is doing with its test data.

The CIO/CTO. At this level, knowing that there is a positive initiative to always minimize risk and associated costs instills confidence that the organization is handling sensitive data correctly. A TDM strategy will prove accountability and will help to achieve compliance with the local, federal, or international mandates with respect to data management.

The Audit Professional. A TDM initiative verifies that the recommended procedures to achieve management of data are in place. This enables an auditor to easily manage reviews and substantiate the actions taken by the organization to protect sensitive information.

The Security Officer. A TDM system is part of the security mechanisms to protect data and can also provide an inventory of the enterprise assets to be managed under current security guidelines. A TDM system will document how specific data management rules and procedures are defined so that the Security Officer can validate policies at any time.

IT Personnel. Anyone having any kind of hands-on involvement with application development and test data (customer data). Subject matter experts will want to know what is happening to their data.

During a TDM initiative, a plan which describes in detail the phases, activities, and specific tasks and their deliverables to develop processes to protect sensitive data will be produced.

The Consumer. There is a growing realization amongst members of the public who have no tangible connection to the use of personal data that many organizations or enterprises store information about them. Their names, addresses, Social Security Numbers, and more… and they want to be confident and reassured that this data is not going to fall into the wrong hands.

A robust TDM strategy will meet the needs and concerns of each of these stakeholders, in addition to addressing risk of breach and ensuring compliance. In Part 3 of our series, we’ll cover the challenges faced when designing and implementing a TDM strategy.

In Part 1 of a 4-part series, we discuss what TDM is and why it is critical to your enterprise.

Defining a TDM strategy can be challenging—the growing number of systems and environments can even make it seem impossible. This series will address some of the common questions that customers have asked and discuss the delivery and implementation of a cross-platform mainframe and distributed TDM.

This series was written both to help enterprises understand the need for a TDM strategy and to share some tips on how to create a strategy that will obfuscate data in lower environments and minimize the cost associated with unneeded massive volumes of test data.

Posts focus on what TDM is and why it is needed, the stakeholders interested in undertaking a TDM initiative, the challenges of managing test data, solutions that will help you define and implement your strategy, and the benefits of such an implementation.

Part 1: What Is TDM and Why Do We Need It?

What are some compelling reasons for TDM?

Today’s Agile DevOps teams need the ability to go faster without sacrificing quality. Developers should not need to spend time on data provisioning tasks – they should spend more time developing new functionality and managing production problems. A robust TDM solution is a critical part of any enterprise and will ensure the provisioning of the rightsized test data to qualified users in a repeatable and timely manner. The data should contain all required test conditions and be anonymized.

The following factors emphasize the need for a TDM solution:

Legislation. Recent years have brought an increasing trend of local, federal, and international regulations towards the protection of sensitive data. Many of the legislations force businesses and organizations to scrutinize the way proprietary and confidential data is handled across a business enterprise. This legislation comes in the form of Europe’s GDPR, California’s CCPA, South Africa’s POPIA, Canada’s CPPA and others.

As a result, many companies are managing internal policy changes that demand the implementation of tighter security measures, strict audit controls, and radical changes to existing business practices. All of this is being done in an effort to meet compliance and demonstrate accountability.

Along with Covid-19, 2020 brought changes in working practices. More and more tech workers are working remotely – in their basement, in their bedroom, hallway, kitchen table, or any space in or outside of their home, maybe public places where wifi is available.

These changes create new challenge for IT leaders. Not only does an organization need to respond to its core business needs, but it also needs to adhere to strict legislative requirements associated with the exposure, theft, or misappropriation of customer, financial, corporate, or personal sensitive information.

Risk mitigation. In order to mitigate against the risk of a data breach from test data files, IT shops are taking different actions to safeguard their data assets. Much effort has been focused around the protection of production data from external threats by means of tighter security access, firewall, network, communications, storage and audit counter measures. However, studies conducted by research firms and industry analysts reveal that the largest percentage of data breaches occur internally, within the enterprise.

While organizations may think that their core data is immune from external Management threats, environments outside of the production perimeter such as testing, development, or quality assurance usually have far less robust security controls. Access to these areas is typically more widely exposed to a larger variety of resources, including in-house staff, consultants, partners, outsourcers, and offshore personnel.

Recent Breaches. As in every year since the turn of the millennium, 2020 has had hundreds of breaches across the globe. A simple search of “data breach 2020 headlines” in your favorite search engine will return headlines of breaches in all business sectors, from financial to healthcare, and from retail to cruise line.

Recent Fines. If you are an enterprise officer you should be afraid, very afraid. Massive fines and settlements have been levied on many household named businesses in the past 2 years. In July 2019 a leading credit agency agreed to pay over $575 million for “failure to take reasonable steps to secure its network.” Again, a simple web search of “data breach 2020 fines headlines” will return some of those fines.

Given these legislative requirements, internal and external risks, and recent breaches and fines, the need for and benefits of a TDM solution should be clear. Look for part two in our series for a discussion of the stakeholders within an organization who have an active interest in the security of test data and how a comprehensive solution benefits each.