Real-time data movement is no longer a “nice to have”
Sumeet Puri, CTO of Solace, discusses the role that event-driven architecture (EDA) can play in real-time data movement
Companies now know the basics of storing, managing and analysing data, but how this data moves is still a mystery to many. Yet, in the background, real-time, efficient data movement is the key to most industries’ operations – not least building and maintaining an efficient supply chain.
Products are not the only entities that travel through a supply chain; the movement of data which allows this travel is an often overlooked yet critical element. The IT infrastructure at the heart of any given supply chain can either boost or constrain a company’s bottom line and its customers’ experience, by either allowing a fast and effective movement of crucial information or throttling it.
The crux of the matter, and IT’s key challenge, lies in the accessibility of real-time data and communication both within the organisation and across its partners in the ecosystem, from retailers to suppliers, to warehouses to shipping lines, and more. And that too in real time, moving “events” as they happen at the same time to all systems who need them.
While batch-based, services-based and API-based ways to exchange data have been used over the decades, “event” and “event streaming” are a more natural way of thinking about real-time data movement. An “event” is an occurrence, which needs to be pushed in real time and in a guaranteed manner to whoever needs it, or subscribes to it.
For example, how does the US dollar price changing against the British Pound in London immediately get reflected in New York and Singapore? It’s an event that is being streamed. How do online shopping orders from London reach the warehouse in Oslo a millisecond later? And not just that, but how does it update inventory, warehouses, factories when needed as well as analytics and personalisation systems all in parallel at the same time, guaranteed? How do IoT sensors keep the energy grid running smoothly? How about a digital factory streaming events to correlate with quality and ERP systems and data lakes? All of this is made possible by events being streamed in real-time.
The way data can travel so fast is through an underlying event streaming layer called the event mesh, the infrastructure’s nervous system, which makes signals or data changes travel from point A to B, an C and D, all in real-time. Global capital markets have used this event-driven paradigm for years, and as other industries become more real-time, they can take inspiration from global capital markets event-driven architectures.
That’s why event-driven architecture (EDA), which pushes information as events happen, is a better architectural approach than waiting for systems to periodically update. EDA ensures that event information is sent to all of the systems and people that need it. It’s a simple but necessary concept.
Applications and systems can “publish” events to the mesh, while others can “subscribe” whatever they are interested in, irrespective of where they are deployed in the factory or data centre, or the cloud. This is essential for critical industries we rely on, such as capital markets, industry 4.0, and a functional supply chain. Indeed, there are few industries today who can do without as-it-happens updates on their systems. Businesses and consumers demand extreme responsiveness as a key part of a good customer experience, and many technologies depend on real-time updates to changes in the system.
However, many existing methods for ensuring absolute control and precision of such time-sensitive logistics don’t holistically operate in real-time, at scale, without data loss, and therefore open room for fatal error.
