Four Steps To Keep Data Scientists In Their Happy Place
Imagine someone has thrown the pieces of five different jigsaw puzzles into a single pile. Before you can start solving the puzzles, you must first separate all the pieces first. Then you discover key pieces are missing. You know they are somewhere in the house, but you must go hunting for them.
Data scientists working in traditional IT environments face this kind of challenge every day. Before they can even begin their analytical work, they must complete four basic tasks:
• Gather data: hunting for it across multiple systems.
• Validate data: ensuring they have the right data from the right data set, and that it is accurate, complete and up to date.
• Connect data: structuring and combining disparate data types in ways that answer business questions consistently and accurately.
• Select data: creatively combining data sets in the search for powerful predictive models (also known as feature engineering).
This process can be painstaking, repetitive and often highly manual: A survey of data scientists found that 76% thought of data preparation as the least enjoyable part of their job.
As a software engineer at Facebook, I faced exactly this challenge as we tried to stitch together predictive scores with customer profile information to optimize ad delivery. ETL-based data-wrangling solutions could help, but only to a certain extent. So I kept asking myself what capabilities would be required for data scientists to locate the latest, greatest data instantly and start modeling it right away. These are the steps that I believe are required as a result:
Finding data sources across multiple systems, each with its own protocol, is painstaking. Some businesses consolidate data in data lakes or enterprise data warehouses. However, both approaches are of limited help to data scientists gathering the specific data they need. Data lakes lack any inherent organization, so finding data is very difficult. By contrast, enterprise data warehouses store data with such rigid order that data becomes hard to find without a deep understanding of the warehouse’s design.
An alternate path begins with a schemaless database approach, allowing raw, granular data to be ingested at speed without requiring complex upfront ETL work. However, the data obviously requires some kind of order if data types are going to be transparent (and usable) for data scientists. So how do you balance these opposing requirements?
I believe it is possible to create a structure of data that is both highly flexible and transparent. In the case of customer data, it requires storing data in the same way marketers think about that data. You can make customer IDs serve as the prime logical unit, to which you can associate a clearly defined array of data types that help define that customer. This is an approach that makes it easy for data scientists to find and precisely access the data sets and attributes they need.
Businesses are constantly collecting new types of data and introducing new data sources. However, these changes are often invisible to a data scientist, making their data validation process a nightmare. They must be on guard for such changes, or risk having faulty findings.
However, you can sidestep this risk with a data store that can separate out business definitions from the raw data provided by source systems. For example, the business definition of a valid purchase typically requires two different pieces of source data: purchase information, of course, but also return information. However, this definition may change over time. If data scientists are not aware of these underlying definitional changes, they may end up comparing apples and oranges.
