why is effective data collaboration so elusive?
Traditionally, the job of gathering and integrating data for analytics fell on data warehouses. Data warehouses cleaned and aggregated the “operational exhaust” of business transactions.
Data warehouses created precision, accuracy and consistency out of messy transactional structures, so that business people could look back, and make sense of the past.
As data warehousing evolved, so did our operational applications. With the advent of the Internet, a whole new challenge to precision, accuracy and consistency emerged.
For business analysts and data scientists alike, their work now involves much more searching and discovery of data rather than navigating through steady structures. It is no longer a singular effort. Sharing results and persisting new views of data and models, and collaboratively governing their catalogs is the approach that works.
It is important to make the distinction between tools that provide “connectors” that operate on a physical level, either generating SQL from structural metadata or using API’s, etc., and those that have a rich understanding of the data based on its content, structure and how it is being used. Connectors are not sufficient. Someone, or some thing, has to understand the meaning of the data to provide a catalog for analysts to do their work.
Keep in mind that all of the data used by data scientists, analysts and other applications is essentially “used.” In other words, it was originally created for purposes other than to be analyzed: supporting operations, capturing events, recording transactions. The data at source, even when it is clear, consistent and error-free, which is rare in an integrated context, will still contain semantic errors, missing entries, or inconsistent formatting for the secondary context of analytics. It must be handled before it can be used. This is especially true if it is meant to be integrated with data from other sources or other previously managed data.
Even somewhat stable and understandable data like logs, especially weblogs, tend to drift over time. Can you think of a major website that hasn’t gone through a complete refresh in the past three years or so?
Data warehousing addressed this problem of dealing with “used” data long ago, with processes that executed before data was stored. This is where data warehousing and Hadoop differ. Tools such as ETL, methodologies and best practices ensured that any analyst working with that data accessed a “single source of truth” that was already cleaned and aggregated to produce a pre-defined business metric, often labeled a key performance indicator.
Though in fairness, applications from data warehouses are often quite creative. These solutions are also partially useful for data scientists, but because the data was pre-processed to fit a specific model or schema, the richness of the data is lost when it comes to hypothesis testing in a more exploratory manner and typically too slow to implement when experiments and discoveries are happening in an unplanned fashion (“Our competitor just released a press release about a new pricing model, how should we respond?”).
The innovation of widespread cloud computing and “cloud native” data warehouses introduced a better storage mechanism and best practices for hypothesis testing on rich, raw data – at low cost. These cloud data warehouses, as well as cloud native storage protocols such as JSON, evolved out of a system designed to capture “digital exhaust.”
Initially the byproduct of online activities, today, “clouds” also include a wealth of machine-generated data; events gathered from sensors in real-time. While data warehouses typically stored aggregated information based on application transactions, digital exhaust can be found in a multitude of forms like XML and JSON, typically referred to as “unstructured,” though more accurately defined as “not highly structured.
