Operational intelligence and the new frontier of data
Always-on businesses such as global retailers, social media apps, transportation platforms, and financial marketplaces have mission-critical use cases that require real-time decisions on operational event streams.
Target’s supply chains must adapt to changes in store inventory, Snap’s new app launches must be debugged, Lyft’s drivers must be predictively routed to riders, and at Paypal, fraudulent payments must be flagged and blocked.
These use cases for logistics, app monitoring, and fraud detection aren’t science fiction, they’re real-world examples powered by an emerging technology stack that combines event stream processing, fast OLAP databases, interactive dashboards and machine-learning applications. Fueled by real-time data coming from instrumented products and services, this technology stack is driving a distinct category of analytics called operational intelligence, which is complementary to traditional business intelligence.
At Rill, we believe the need for operational intelligence will dramatically expand in the coming years. In this post we lay out why operational intelligence matters now, its salient differences with traditional business intelligence, and why it demands new technology architectures.
One consequence of ubiquitous computing and digital transformation is the promise of visibility into the previously invisible. There are now an estimated 20 billion connected devices on the planet, encompassing everything from smartphones and cars, to lamp posts and laundry machines. Previously unobservable actions in the analog world — a package delivery, a store purchase, a taxi ride — now throw off digital signals in the form of a barcode scan, a payment gateway request, or a stream of GPS heartbeats.
Collectively, these connected devices are the sensory scaffold for an emerging global digital nervous system, whose potential we are only just beginning to explore. However, some of the near-term opportunities are analogous to the advantages of our own nervous system: faster reaction times and better decision-making with richer sensory data.
While smart sensors have been widely deployed and digital services are more finely instrumented than ever, businesses are just now building the software infrastructure to capture and process the event streams from their products and services.
With this first wave of data plumbing complete, businesses now have an opportunity to build new kinds of workflows. For example, traditionally businesses compile and update a set of KPIs on a weekly or daily basis. What happens when it’s possible to track these metrics up-to-the-minute and down to an individual line item or customer? It means that small issues can be addressed before they become big challenges. Shipped software used to take months to detect and correct issues, but today Netflix continuously monitors the performance of new versions of its software and can revert software for specific users when issues are detected. Similarly, Lyft tracks rider conversion rates at a city block level and redirects drivers to a particular block when those rates drop.
Sensor data is generated on the edge by users on their devices, fleets of GPS-instrumented vehicles and servers in a data center. This data is then streamed over the cloud to a central processing system to synthesize KPIs. However, if the insights stopped here, with a downsloping graph on the dashboard of a business analyst, the value of these insights is limited.
One of the hallmarks of operational intelligence is the speed of feedback to the edge. In the real-life examples above, Netflix and Lyft aren’t waiting hours or days to make adjustments, they are acting within seconds to minutes. As operational intelligence systems mature, human analysts are excised from these decision loops, and these course-corrections are automated, powered by machine-learning algorithms.
