How the data warehouse can stand between your data and your insights

You have a product that has taken off. Your daily active users metric has been growing exponentially. The number of events per day you’re logging is now in the 100’s of millions.

As a result you now find yourself with terabytes of data or if you have become really successful hundreds of terabytes.

You begin to wonder if you could use all of this data to improve your business. Maybe you can use the data to create a more personalized experience for the users of your product. Or maybe you can use the data to discover demand for new products.

You request that your data team come up with way to leverage this data to do just these types of things.

The data team that you have hired recommends that you develop a data pipeline. An end-point of that pipeline being the data warehouse.

You may get something like this:

But after months of work, and many dollars spent building the data warehouse, the data scientists that you hired can’t come up with the insights.

How could all of that data, all of those IT consulting hours, and those cloud computing resources be marshalled to not produce the insights?

The problem likely lies in one of the important components of your pipeline: the data warehouse

Here are some of the painful things you can experience in the data warehouse:

You data may be streaming in from multiple sources. When an analyst runs a on this data, it could result in a table that is inconsistent. Inconsistent data can manifest itself as missing columns that are required to properly identify each data item. Or the data may contain duplicates that take extra space and prevent from performing the aggregations necessary to achieve insights without extra work (meaning extra analyst time cleaning the data via interpolation, and extra compute hours deduplicating the data).

You have PhD’s on your analyst team. Why are they scratching their heads and shrugging their shoulders after looking at your data? It could be that the tables in the data warehouse are an enigma.

A lot of times, the data warehouse is built by a different team than the analysts. Both groups are trying to manage data but are not necessarily playing for the same data team.

Oftentimes the tables are created in a way that makes it easy to create the table and but not easy to be processed downstream. The table is created without taking the downstream requirements into consideration! Noone thought to begin the data warehouse design with the end goal in mind of quickly enabling insight generation.