The Era Of Continuous Intelligence
The primary data challenge (and opportunity) presenting itself to many organizations, whether commercial enterprises, academic institutions or public sector bodies is not one of volume, but of speed.
That’s not to say that managing ever-increasing volumes of data isn’t important. Big Data still presents a challenge, but the tools, processes and skills needed to meet that challenge are commonly known and widely adopted. Big data has arguably been tamed; fast data is now, for many organizations, the holy grail.
Every event that occurs within an organization can be captured and turned into a data point; a temperature reading from a sensor in a piece of machinery, telemetry data from a delivery truck, phone calls in a contact center, visitors to a website. Each event has value, and that value can degrade depending on the length of time it takes to capture, process, analyze and act. Often framed in mere nanoseconds, this ‘window of opportunity’ is becoming a key metric for competitive differentiation for companies in sectors as diverse as financial services, space exploration, telecommunication networks and home energy providers.
It’s important to remember that when discussing the analysis of real time data, its value increases exponentially when you can bring it together with historic data to combine, compare and contrast ‘in the moment’.
Take for example temperature data from a sensor embedded in a machine. Understanding that data ‘in real time is useful for checking that the machine is operating efficiently or that a temperature threshold hasn’t been reached, but when you add it to historic data, mapped over many days and months, you not only get a richer understanding of how a machine is performing, but you can also build predictive models based on other machine performance profiles to understand when problems are likely to occur and take action in advance.
You can extrapolate this example to all manner of user scenarios and industries; track data from NASCAR cars alerting engineers to engine abnormalities, search queries from home shoppers telling Google or Microsoft what advert to serve when the results appear, location data from sensors on autonomous vehicles or drones warning of obstructions. The list is almost endless
Moving from From Big to Fast Data
At Kx, we call this the era of ‘continuous intelligence,’ namely the ability for organizations to make smarter decisions derived from insights gained from analysis of data – whether real time, historic or both in as short a time frame as possible. From machines in precision manufacturing that need to improve yield and reduce waste, to equipment monitoring in remote areas, optimizing 4G and 5G networks in real-time, or even improving vehicle performance for F1 racing teams, historical data must be able to inform and shape large volumes of incoming data – as that data is created – so that real-time information is immediately placed in the context of what a business knows already. This allows for faster, smarter decision-making and moves us beyond the age of data management and into the era of continuous intelligence.
The challenge for many organizations however is how to deliver continuous intelligence when their data management and analytics software stacks are often siloed and heavily fragmented.
To manage the vast increase in data volumes witnessed over the past decade – and to try and maximize the value inherent within it – organizations invested in large-scale data infrastructure solutions. Enterprise class databases that fed off data warehouses and data lakes all stitched together with integration and governance systems. As a result, many have ended up with a complex software stack with multiple applications from different vendors covering storage, analytics, visualization and more.
