The five D’s of data preparation

Data preparation is the task of blending, shaping and cleansing data to get it ready for analytics or other business purposes. But what exactly does data preparation involve? How does it intersect with or differ from other data management functions and data governance activities? How does doing it well help business and IT users – and the organization overall?

Data preparation is a formal component of many enterprise systems and applications maintained by IT, such as data warehousing and business intelligence. But it’s also an informal practice conducted by the business for ad hoc reporting and analytics, with IT and more tech-savvy business users (e.g., data scientists) routinely burdened by requests for customized data preparation. These days there’s growing interest in empowering business users with self-service tools for data preparation – so they can access and manipulate data sources on their own, without technical proficiency in SQL, Python or SAS® code.

One way to understand the ins and outs of data preparation is by looking at these five D’s: discover, detain, distill, document and deliver. Let’s examine these aspects in more detail.

1. Discover
Discover is about finding the data best-suited for a specific purpose. Many users cite this as a frustrating and time-consuming exercise. An essential enabler of efficient discovery is the creation and maintenance of a comprehensive, well-documented data catalog (i.e., metadata repository). In addition to data profiling statistics and other contents, the data catalog provides a descriptive index pointing to the location of available data.
Data profiling is key to unlocking a better understanding of this data, because it provides high-level statistics about the data’s quality (such as row counts, column data types, min, max and median column values, and null counts). That makes it easier to choose when multiple applicable data sets are discovered.

It’s important to note that discovery should not be just about finding the data that’s needed right now. It should also make it easier to find data later, when similar needs arise. As the enterprise encounters new data sources, especially those external to the organization, the data catalog should be updated. That’s true even if no further data preparation steps are done immediately.

Manual data preparation, often by spreadsheet wranglers, is not only time-consuming but often redundant. That’s because different users (or even the same user) may perform the same work – and they don’t necessarily generate the same results each time.

2. Detain
Detain is about collecting the data selected during discovery. The term “detain” conjures the image of temporarily imprisoning a copy of the data that feeds the rest of the preparation process. For too many organizations, the cells of a spreadsheet permanently detain data both during and after preparation. A temporary staging area or workspace is required for the processing that happens in the “distill” step of data preparation. When persistent detention of intermediate or delivered data is required, it should make use of shared and managed storage – a relational database, network file system or big data repository such as a Hadoop-enabled data lake. An emerging technique here involves the use of in-memory storage areas (or alternatively the cloud) for much faster real-time blending and shaping of the data before it’s sent on to other processes.

3. Distill
Distill is about refining the data collected during the detain phase of data preparation. In the process of refining data, you must determine how fit the data is for its intended purpose, or use.