Embrace Diversity in Your Data Architecture
Over the last ten years, the data management landscape has changed dramatically — on that, I think we can all agree. The rise of big data and the new data management ecosystem has created an abundance of new patterns and tools, each of which is more specialized than the last. With each new iteration, engineers and architects face pressure from all sides to simplify and consolidate.
But counter-intuitively, the best data architects embrace infrastructure diversity rather than fight it. The reality is that all of these tools and patterns have important uses in enterprise data architecture, and that today Kafka is no more the cure to all that ails than MapReduce was five years ago. The most sophisticated enterprises enable each business unit to use best-of-breed technology to succeed while facilitating seamless integration between them, creating agility and avoiding the chaos that often arises across legacy environments.
When designing applications, there are many business and technology requirements that need to be considered. Rather than try to force an application into a singular architecture, it is important to ask the right questions to determine which architecture best fits a particular application.
If you have data that arrives in a stream, a streaming architecture may make perfect sense. Data coming in over REST interfaces, or log data and events being written to files, are perfect candidates for stream processing. Treating a relational table as a stream of rows is an elegant way to map an otherwise batch dataset onto a streaming application architecture, and in fact, this is how StreamSets processes relational data.
However, not all data is available in this way. Businesses often rely on data sourced from third-parties, partners, or customers. If that data is not made available as a stream, or if new data only becomes available intermittently, a streaming architecture may not make sense. If data arrives infrequently, the best architecture boils down to a choice between the convenience of having either an always-on streaming pipeline that sits idle for most of the day or a batch job that is started only when necessary, and stopped as soon as operation is finished. The choice ends up coming down to a matter of resource management and economics, and a streaming architecture may not always be feasible.
The urgency for data availability is a spectacular driver for continuous data movement and streaming ingest within the business. Users have come to expect immediate gratification and businesses know that they’re missing opportunities to improve “customer experience” if they don’t deal with data as it becomes available. This is true with both external and internal customers. Externally, an application might provide a “next best action” for a customer on a website, outside a storefront, or in front of a shelf of products. Internally, it might be a cybersecurity or operational insight application, whose customers want to be able to iterate quickly on the questions they’re able to ask and the data they’re able to analyze. In particular, data urgency requires you to answer the question “how will I manage my data operations?” For instance, the definition and continuous enforcement of SLAs for data latency will play a role in the technology you choose.
