The year 2020 is seared into CIOs’ collective memory as one of the most cataclysmic, consequential years this century. But while the pandemic helped drive digital transformation far beyond anyone’s expectations, make no mistake: This was not the perfect storm we’ve all envisioned.
If anything, the unprecedented wave of data-driven deployments last year shined a light on shortcomings in existing frameworks that are muting Big Data’s potential. Many organizations, for example, have been frustrated by how long it can take just to prepare data before analysts and scientists can even start a new project.
As well, the process for migrating the resulting models into business processes has been just as vexing. And without the proper security and governance controls across the entire data lifecycle, analysts and data scientists run the risk of exposing and permanently sullying critical data stores.
“It’s really about expanding the data management purview,” said Richard White, Chief Data Officer at New York Life Insurance Company. “What we’re seeing is a triangulation of the data, data protection and cloud strategies. And they’re all coming together, with more pervasive governance.”
The need for that triangulation isn’t new. But the requirements became vastly more complicated during the pandemic, primarily for two reasons. First, the number of data science projects inside many organizations mushroomed. And second, the constellation of data sources fueling those projects greatly expanded.
Which helps explain why more than 37 percent of organizations view security and governance as their top cloud challenge, according to the just-released Global Cloud Survey 2021 from data virtualization provider Denodo.
But perhaps the greatest data management lesson to come out of the pandemic is that many more organizations now understand that there’s a right way to do data, and a wrong way. And while there are tools to make the job easier, there are no shortcuts.
“There’s legitimate, difficult work to do to bring data together, and then to do science on that data,” Chris Wright, Senior Vice President and CTO of Red Hat, told me. “So you need to have the right tool set in place.”
Indeed. Which is why many CIOs and CDOs are making it a priority to build, rent or buy what some are calling a cloud data platform – that is, all the ground-level management stuff that must happen before, during and after the glitzy work of digging into the data lake in search of answers.
Perfect storms form when, at long last, all the requisite ingredients needed to drive a new market skyward are not only available, but also mature enough to scale. That sounds a lot cleaner than what happens in real life, of course. Because the individual sections of scaffolding rarely develop at the same pace. Worse, it’s sometimes difficult to see what’s ready and what’s not without deploying what’s available in the wild.
Which, as we all know, is exactly what happened during the pandemic.
“We were already headed this way,” JG Chirapurath, VP of Azure Data, AI and Edge at Microsoft, said. “But when the pandemic shut everything down, the evolution of using data to unlock outcomes became more of a revolution.”
To be sure, the shutdown effectively scrapped existing models and dashboards, rendering finely tuned supply, demand and cost forecasts worthless. Enterprises suddenly were flying blind. They urgently needed new systems to operate in a reality that only just started taking shape in March 2020.
That sent businesses scurrying in search of new data sources, giving rise to the demand for hybrid and multi-cloud. Many companies, for example, folded infection rates, pandemic sentiment trends and other COVID-related information into their budding new models. At the same time, some large, established organizations found cost-cutting religion during the shutdown, and pulled stable, regular workloads back from the cloud.
