Streamlining Real-Time Data Integration and Multi-Cloud Management – DM Radio
Data Integration and real-time data are crucial to an organization doing online business and working towards digital transformation. The success of Big Data programs depends on how efficiently an organization can collect data, process and integrate it, and then analyze it. Building flexible Data Management architectures that can support the volume and variety of data steaming into organizations today is key. Data-driven businesses need tools for both Big Data Integration, and Cloud Data Integration. Dan Potter, the Vice President of Product Marketing at Attunity, remarked in a recent DATAVERSITY® interview that:
“Data Integration has been around as long as data’s been around. It’s evolved a lot over the years and when you look at the data infrastructure changes, relational databases were the thing back in the ’80s. Then the rise of data warehouses in the ’90s. Then all of a sudden we start to move to more distributed computing, and now cloud computing.”
The Data Management infrastructure kind of evolved as well, said Potter, with the advances in Hadoop. The big cloud vendors, specifically Google, had to figure out a way to manage their own data and be able to create and query data quickly, which led to the rise of Hadoop, which then led to commercial Hadoop distribution. “You had different ways to manage the data,” he commented. “Different integration requirements for Hadoop, different challenges with Hadoop, and now lately it’s all about the cloud, and Hadoop still sits on the cloud.” Data warehouses have found a new home in the cloud. The cloud has delivered both the elasticity and cost advantages.”
Data Integration is the process of extracting data from different sources, and then filtering and transforming it into a unified view, making it useful for actionable and valuable decisions. It can include replication, and will make data available throughout the organization, supporting organizational reports and analytics. A business wanting to maximize sales and supply chain efficiency will need to have warehouse data, point-of-sale data, and shipping data.
A business wanting to train and use Machine Learning or Artificial Intelligence will aim a steady flow of data through these systems (the more data they process, the more effective they become). This flow of data can come from a variety of sources, including the Internet of Things (IoT). Use of the IoT often requires integrating databases, devices, and business systems. However, it should be noted, integration has been described as one of the leading and most expensive barriers to embracing IoT analytics. According to Gartner, half the expense of implementing an IoT program will be spent on integration.
Real-time data streaming involves the processing of massive amounts of data quickly enough so an organization’s decision-makers can react, in real time, to changing conditions, said Potter. Massive amounts of data can be stream processed, allowing organizations to respond immediately to potential security threats or fraudulent activity, and to promote profits. Data from the IoT comes with a significant amount of volume and dealing with it efficiently requires real-time data streaming. Potter commented:
“If you think about what’s happening in the analytics world, you’ve got AI and Machine Learning. The more data that you can feed into those models the more effective they are. You’ve got IoT data, which is lots of volume. You’ve got predictive and prescriptive analytics. You’ve got automation happening in terms of decision-making. The faster I can have an insight and make a good decision to lower my risk, the lower my cost, the greater the opportunities I’m finding. There’s a lot on the analytics side that’s driving the need for the new data infrastructure, and the cloud is helping to address those challenges.
There’s also a lot of disruption and the disruption’s being caused by startups who are more agile. “It’s the fast eating the slow,” he said.
