Trends and top use cases for streaming data analytics
Enterprises are starting to adopt streaming data analytics to provide insight that adapts to new data, often in real time. This goes beyond traditional analytics that operate on batches of data to provide more reactive insight.
The technology is still in its early days, but enterprises are excited about providing more actionable insight to managers and frontline workers and better applications for users.
“Capturing, integrating, analyzing and archiving data streams is much more complex than creating pipelines for batch data at rest,” said Torsten Volk, managing research director at Enterprise Management Associates.
Streaming data analytics is like trying to analyze how cars work by standing next to a highway watching traffic, compared to examining those same cars in a parking lot. Not only do you not get a good long look at each car in traffic, you also need to constantly switch your attention from one car to another. “Enterprise data analysts face similar challenges when it comes to integrating and synchronizing streaming data, often from highly dynamic sources such as application containers, IoT sensors or edge devices,” Volk said. Pulling off this challenge requires a whole new set of technologies and human skill, as in addition to dealing with “moving cars” you also need to think about how to accommodate the much more dynamic character of these data streams. For example, containers can quickly spawn, move or shut down in different clouds or data center locations, while sensors can be added, enhanced, replaced or removed without notice. Additionally, an edge device could deal with patchy connectivity. “Building dynamic data pipelines to deal with this constant change is a nontrivial task,” Volk said.
Volk sees containers, IoT and edge computing as three key drivers to streaming analytics as all three disciplines continuously create increasingly broad streams of operational data. Capturing and correlating these data points often makes the difference between excellent and disastrous decisions or problem responses. For example, with Kubernetes containers, the more complex container clusters become, the harder it becomes for IT admins to get to the root cause of issues through static log analytics, as these logs often do not capture the early indicators of a problem. Streaming analytics catches these indicators by looking at everything that is going on between and during the time logs are created. Streaming analytics would enable teams to discover that while the app shows no symptoms of stress at all, under the covers there might be conflicting policy parameters causing containers to needlessly spin up and down or move around. This might be fine under standard conditions but lead to big problems under stress, Volk said.
Enterprises are adopting streaming analytics in response to the need for faster decision-making. “Organizations strive to reduce delays in analyzing data [at] rest by increasing their ability to analyze data in motion,” said Carlton Sapp, senior director analyst at Gartner. He also sees some tools for streaming analytics to help reduce noisy data and process only data that is relevant for answering or solving a business question or challenge. Streaming analytics platforms have become sophisticated filters for reducing an overall glut of data that is not useful to an application. These tools can also help analyze IoT data closer to where it is collected and appropriately summarized to drive deeper insight. Sapp believes that enterprises are increasingly looking for advanced analytics capabilities that use AI or machine learning on data in transit. But he noted that performing deep learning on streaming pipelines remains a mystery to many organizations. Another emerging trend is the development of tools to improve contextualization of streaming data by connecting streams to various data stores at rest.
