How to build an effective streaming data architecture
Enterprises are being deluged with real-time data from a variety of sources, including web and mobile apps, IoT, market data and transactions. In theory, this should provide a wealth of information for improving customer experience, saving money and generating profits.
The challenge is figuring out what combinations of these data streams and types of analytics pan out in practice.
A proper real-time analytics architecture can help business managers and data scientists quickly test out new ideas to identify and scale up the best use cases.
Traditional analytics approaches rely on data being structured and stored in databases optimized for a particular class of query. Real-time analytics replicates this for data that is constantly changing and must be structured on the fly.
“Streaming analytics makes it possible to know and act upon events happening in a business at any given moment and use that information to make better business decisions,” said Erik Riedel, senior vice president of compute and storage solutions at ITRenew, an IT consultancy.
At the same time, it’s important to design a real-time analytics architecture that can respond and scale, rather than just building a one-off streaming analytics project.
Key questions that will shape your streaming data architecture All streaming architectures will have the same core components. These include a streaming data aggregator, a broker for managing access to this data and an analytics engine. But these components need to be customized for different kinds of enterprises and use cases.
“The specific data architecture needed for streaming will vary with data sizes, transaction frequency and the complexity of the questions that are being asked of the analytics,” Riedel said. For example, there can be big differences in the requirements for providing real-time analytics data to a small number of analysts at the company’s headquarters for quarterly trend spotting compared with providing different kinds of real-time analytics across a larger enterprise. Key questions to consider include the following:
What is the data size?
What is the update or transaction frequency?
What is the query complexity?
What is the number of analysts or applications it will serve?
Riedel said the answers to these questions can impact system design all the way down to the hardware level in terms of data storage, RAM and distributed memory access. Geographic distribution of stream ingestion can add additional pressure on the system, since even modest transaction rates require careful system design. Alex Bekker, head of the data analytics department at ScienceSoft, an international IT consulting and software development company, said the company uses the following blocks to structure IoT real-time analytics architectures: things — objects equipped with sensors to generate data for analysis; gateways — the linchpins between things and the cloud part of the architecture; data lake — a temporary reservoir to store data in its natural format; big data warehouse — a reservoir for storing processed and structured data to be further analyzed for meaningful insights; data analysis segment — where the analysis happens; and control applications — the blocks where automatic commands and alerts are sent to applications or real-world actuators in IoT apps.
Plan for growth of real-time analytics use cases Once a few real-time analytics applications pan out, data managers and data engineers can be inundated with requests for new types of analytics. “Once the business value of the analysis becomes clear, use of the system and richness of the analytics demanded may grow quickly,” Riedel said. He found that open standards and open frameworks can help solve key infrastructure scalability and adaptability challenges for both hardware and software. They also remove obstacles like vendor lock-in.
