Modern Distributed Data Architecture with Event Streams, Stream Processing and Derived Data
Some of the most interesting projects I worked on at LinkedIn involved building large scale real-time pricing and machine learning products. They required crafting fault-tolerant distributed data architectures to support model training, forecasting and dynamic control systems. None of this work involved a traditional relational database. Instead event streams, derived data and stream processing became the core building blocks.
We built entire multi-billion dollar products on top of these architectures. The experience gave me a new perspective on how organizations can think about their data infrastructure in terms of data flows.
Over the past decade, distributed backends have grown considerably. Most products start out with a typical database centric architecture, then add replications and partitions to handle scale and growth. Periodic data snapshots are made available through a data-warehouse like HDFS for analytics and offline pre-computations. In some cases, nearline stream processing is introduced, perhaps through a lambda architecture. This evolutionary path that backends go through is quite familiar. The evolution is driven primarily by requirements to handle high requests per second to servers in the shortest acceptable time, typically within a few milliseconds.
The distributed backend is made possible thanks to log based data flow between the various replications and partitions. These streams of logs capture and transmit database writes. LinkedIn’s Brooklin is an example of this. In essence, each message in the stream represents a change event in the application state. With database writes externalized in a stream of logs, we no longer have to poll the database for changes. This is where stream processing has enabled us to react to change in application states in realtime. Also considering that streams can have multiple listeners, we are able to run orthogonal processes on a single stream of data.
We don’t have to be limited to firing change events only on database writes. We can also instrument both the application frontend and backend to fire events on various activities. For example we can fire click and UI interaction events from the frontend client. And we can also fire various backend events containing meta data to provide deeper insights into the processes within backend services.
And this creates an interesting opportunity to rethink the relationship between the applications and the database. If changes in states are available outside of the database, then we no longer have to poll the data tables for updates. In fact we can choose to build processes that listen to a stream of messages, and respond to change in state in realtime. Further, each process in turn can emit new messages that trigger other processes and so on.
In this way, the event streams become whatJay Kreps, creator of Apache Kafka, calls the central nervous system through which all data activities flow.
In essence, this paradigm takes system activities and converts them into externalized logs that can be analyzed, processed and acted upon by other systems. Logs have been around for as long as computing has been around. And this paradigm shows that it continues to be important in a distributed data processing world.
It’s also worth noting that a stream based data flow system can not only be processed in real-time, but can also be processed in batches as well by simply collecting the events into a data warehouse. However, the reverse is not possible – that is, a data system that collects in batches (for example periodic snapshots of databases to a data warehouse) cannot fulfil real-time computing needs. Therefore, stream based data flow architectures can seamlessly handle batch and realtime use cases.
Data flow based architecture promotes loose coupling of systems. A stream processor doesn’t need to know specifics of the implementation of the system that emits a stream of events.
