Kinks in the Data Supply Chain
It goes without saying that having a good data supply chain is instrumental in being a data-driven organization. Without a smooth and repeatable process for acquiring, validating, and preparing data for downstream consumption, organizations don’t have much hope in achieving their data-driven goals. Unfortunately, new research suggests sizable constraints are emerging in organization’s data supply chains.
The research comes in the form of a 451 Research report titled “DataOps Dilemma: Survey Reveals Gap in the Data Supply Chain.” Commissioned by Immuta, the survey, which was unveiled earlier this week, reveals that many organizations are struggling to balance competing needs when it comes to data operations.
The survey, which involved 525 enterprise practitioners in the US, Canada, UK, Germany and France and was written by Senior Research Analyst Paige Bartley, looked at various aspects of data culture in organizations today. The survey concluded, not surprisingly, that data is playing an increasingly important role.
For example, the survey found that 71% of respondents agreed that data will become more important to their organization’s decision-making in the future. A similar percentage (72%) said that the number of data consumers in their organizations is steadily increasing. Clearly, organizations are buying into the need to use data as a competitive differentiator, if not a means for basic survival.
But it’s not all puppies and rainbows in data-land, according to 451 Research, which pointed to three main sources of the data supply chain problems.
For starters, industry regulations such as GDPR and HIPAA, while important for protecting the privacy of sensitive personal data, are also taking their toll on the data keepers and their downstream customers.
451 Research found that 84% of survey respondents believe data privacy and security requirements “will limit access to data at their organizations” over the next 24 months. Slightly more respondents (86%) say that it’s true that “security and privacy rules [are] making it harder to access and use data,” the report says.
Secondly, a skills shortage among people threatens to disrupt upstream supplies of good quality data to downstream consumers, including people and applications.
451 Research found that 38% of folks in the data supplier role cited a lack of skills as the biggest challenge or pain points. That hints at an important change in the makeup of the data team, according to Bartley.
“Often, the skills discussion in organizations is focused on the data ‘consumer’ side of the equation, with emphasis on employee data literacy programs and ease of use for self-service software such as data visualization tools,” she writes. “However, as the number of trained data consumers grows and data visualization tools become easier to use, the tide seems to have turned. Now, the traditionally more technical data ‘supplier’ roles are falling behind the consumer demand.”
Lastly, the report cites a lack of automation at the technological and process levels are hampering the ability of the overworked staff to keep up with the building demands placed on the data supply chain.
Too much complexity, various bottlenecks, a lack of real-time streaming, and a general lack of automation were cited as by 29% or more of data engineers, architects, and developers, the survey found.
Bartley pointed at one unexpected culprit as hurting the data supply chain: self-service business models. Apparently, self-service programs may not be the panacea that they are often made out to be.
