Big Data vs. Data Warehouses. What’s the Difference?

Technology progresses at a pace that’s impossible to keep up with, and aging technology executives will soon find that all those undergraduate technology classes are becoming quickly outdated. If you’re a Chief Technology Officer (CTO) at a large firm, you don’t have a ton of time to learn about new technologies because you’re too busy fighting fires and making sure you look good at the next board meeting in front of all the other people who are trying to do the same thing. At some point in time, you may find yourself asking: what’s the difference between big data vs. data warehouses?

Any technology professional is going to be familiar with what a database is. It’s simply a collection of data that grows over time, and from which you learn interesting things from by querying. Then there’s the notion of a data warehouse which is what the name implies. Let’s not get into the whole “Kimball vs. Inmon” conversation and keep this real simple.

A data warehouse is a number of disparate databases in an organization that can be connected by a common key. For example, we might connect records across multiple databases using a unique field called CUSTOMER_ID. Here are databases in various departments where records exist that we may want to link using CUSTOMER_ID:

Using CUSTOMER_ID, you can then easily print out on a single page, a list of all invoices that haven’t been paid and a list of the 10 most recent service requests that a sales person can then take with them to a sales meeting. Of course, today we just use Salesforce for all of this, but this simple example gives you an idea of how useful it can be to connect disparate data sources. That’s what data warehouses are all about, except they take it a step further and use the connected data to make decisions at very high levels. When building a data warehouse, you usually know which questions you might want to answer because some C-level person is asking for certain Key Performance Indicators (KPIs) to be measured. You don’t just go building data warehouses for the sake of building them because it’s an expensive task. Now, let’s talk about “big data” and data warehouses.

The first thing we need to define is the term “big data” which pretty much defines itself. You’ve probably heard the often-cited statistic that 98% of all data has been created in the past 2 years. That’s big data. All the ginormous sets of data exhaust that are now being generated can be mined (remember data mining?) to extract insights. In today’s high-tech world, we might want to generate insights that we don’t know exist. Donald Rumsfeld cleverly referred to these as the “unknown unknowns,” things we don’t know we don’t know about. In the world of psychology, this concept is referred to as the Johari Window. You know that person in sales who is unaware of the fact that their mere existence makes everyone around them want to pull a Peter Pan off the nearest high-rise? The fact that the person is unaware of how annoying they are – and the fact that the people around that person can’t exactly put their finger on why – is an “unknown unknown” in that nobody knows why Rob in sales is just a big, fat, obnoxious prick. Anyways, moving on.