The Path to the Modern Data Warehouse is a Stream
Streaming data is becoming entrenched as a key part of modern application frameworks, but until recently hasn’t integrated well with the traditional database world. Streaming data is event-centric, distributed, variable, unbounded, and unordered, while relational data is batch-centric, centralized, structured, bounded, and ordered.
In her talk “ETL is Dead, Long Live Streams,” Neha Narkhede, the CTO and co-founder of Confluent, describes the disruption of traditional extract, transform, and load process (ETL) due to the needs of stream processing, as ETL infrastructure strains under a constantly growing volume and velocity of data from distributed sources. The big question now is whether the data warehouse can evolve to integrate those two worlds in order to get fresh data into the hands of analysts and data scientists quickly. Is there a path to unbounding the data warehouse to better meet the challenges of streaming data?
ETL is typically snapshot-driven based off the state of the database at a point in time. But in the time interval between snapshots, the state of the database is unknown and can’t be analyzed. The business typically has to live with only approximate results during these intervals. The challenge is that, historically, data warehouses were loaded on a predictable schedule (e.g., daily batches).
Driven by a desire to act faster with fresher insights, some companies began breaking the ETL process into smaller and smaller intervals—from daily, down to minutes. But as intervals decreased, the increased overhead of job tracking and maintenance diverted expert time and attention away from the business.
Streaming data, in particular, exposes the limitations of traditional ETL. Streaming is event-centric, and knowing when the event actually occurred is central to most streaming analytics. ETL, on the other hand, is process-centric, and batch systems have no concept of an event or the time it occurred until it is loaded into the target system. This introduces a potential problem matching event time (when an event actually occurs) to processing time (when an event becomes known to the data warehouse via a batch load). You can think of batches as windows of events, such as a series of customer browsing sessions on an online site. It can sometimes be tricky to associate events to the correct session in order to analyze customer behavior.
If we care about event-time in the analysis, and we’re using batch processing, events have to arrive as ordered in order to process time correctly. However, real-world data, especially from distributed systems, usually arrives unordered. We see examples of this in applications like monitoring or billing, or geo-distributed applications where there’s significant latency in gathering events from all sources.
It’s these streaming capabilities that inspired us to build a Kafka integration for our Pivotal Greenplum Database, so it can handle the demands of data analysts today and in the future.
