Hazelcast Platform to Bring Historical, Real-Time Data Together
Hazelcast is best known as a developer of in-memory data grid (IMDB) technology, a RAM-loving layer for speeding up operational applications. But with the Hazelcast Platform launch currently slated for September, the San Mateo, California company is moving beyond the IMDG and into the realm of real-time applications that combine historical and real-time data for a range of use cases.
The Hazelcast Platform essentially is the combination of the Hazelcast IMDG with Hazelcast Jet, the real-time stream processing application that it introduced about four years ago, says Manish Devgan, Hazelcast’s new chief product officer. By combining the real-time and historical products together into a single offering, it will reduce integration headaches, minimize data movement, and streamline DevOps projects to unleash the power of data.
“The last thing you want to do in a distributed system is move data around,” Devgan tells Datanami. “So instead of the client saying, ‘Hey you’re pulling all that data into the client,’ you’re saying, I got this query or this compute function, and I’m going to send the compute to where the data is living. So that’s why we call it in situ data.”
This type of in-situ processing is very powerful because it enables customers to bring fresh, real-time data to bear on the historical and operational data they already are storing in the IMDB cluster. As Devgan sees it, that opens up a slew of new analytics use cases.
“You can now begin to see that the category of applications went from purely operational, transactional application to more applications which are now doing analytics as well,” Devgan says. “It’s a little bit of paradigm shift here where you have a lot of insights to be had in the operational data, which is going through your application, so why don’t I do analytics right there?”
Instead of building pipelines to move data from transactional systems to analytical systems–or to cloud data lakes like S3 or ADLS, which can then be queried using a variety of tools–Hazelcast is responding to customer demands for keeping the data movement to a minimum. That also helps to boost latency for time-critical decision making.
“You see that a lot of operational data stores have data pipelines, where they move the data from this expensive operational store to maybe a workload on S3 or ADLS on Azure, and then you bring do the processing,” Devgan says. “But we have customers saying, we don’t want to do that.”
Large companies, like Netflix, have the engineering resources to build these types of real-time applications, Devgan says. But smaller firms largely have struggled to deliver the types of compelling, data-driven experiences that customers are increasingly asking for during the current bout of COVID-fueled digitization.
“We are trying to lower the bar, or democratize this concept of building real time applications on the data which is going through to your system,” Devgan says. “You now access data fast.
