The Future of DataOps: Four Trends to Expect
What’s ahead for DataOps? From automated data analysis to the transformation of subject matter experts into data curators, we look at what’s next in the last article in our four-part series.
In this series on DataOps, we’ve covered the three key functions necessary to build a DataOps team and the need to think about agility. We’ve also examined real-world examples of how taking a DataOps approach to data can bring tremendous advantages to organizations.
What does the future hold? Here are four trends you can expect.
Trend #1: Ever-increasing sources of valuable data will intensify the need for smart, automated data analysis
IoT devices will generate enormous volumes of data that must be analyzed if organizations want to gain insights — such as when crops need water or heavy equipment needs service. John Chambers, former CEO of Cisco, declared there will be 500 billion connected devices by 2025. That’s nearly 100 times the number of people on the planet. These devices are going to create a data tsunami.
People typically enter data into apps using keyboards, mice, or finger swipes. IoT devices have many more ways to communicate data. A typical mobile phone has nearly 14 sensors, including an accelerometer, GPS, and even a radiation detector. Industrial machines such as wind turbines and gene sequencers can easily have 100 sensors. A utility grid power sensor can send data 60 times per second — a construction forklift, once per minute.
IoT devices are just one factor driving this massive increase in the amount of data enterprises can leverage. The end result of all this new data is that its management and analysis will become harder and continue to strain or break traditional data management processes and tools. Only through increased automation via artificial intelligence and machine learning will this diverse and dynamic data be manageable economically.
Trend #2: You’ll want to stitch together your own custom solution from purpose-built components
The explosion of new types of data in great volumes has demolished the (erroneous) assumption that you can master big data through a single platform (assuming you’d even want to). The attraction of an integrated, single-vendor platform that turns dirty data into valuable information lies in its ability to avoid integration costs and risks.
The truth is, no one vendor can keep up with the ever-evolving landscape of tools to build enterprise data management pipelines and package the best ones into a unified solution. You end up with an assembly of second-tier approaches rather than a platform composed of best-of-breed components. When someone comes up with a better mousetrap, you won’t be able to swap it in.
Many companies are buying applications designed specifically to acquire, organize, prepare, and analyze/visualize their own unique types of data. (Part 2 of this series discusses the kinds of tools needed for every DataOps team.) In the future, the need to stitch together purpose-built, interoperable technologies will become increasingly important for success with big data, and we will see some reference architectures coalesce, just as we’ve seen historically with LAMP, ELK, etc.
Organizations will have to turn to both open source and commercial components that can address the complexity of the modern data supply chain.
