How to Combat Shadow Data Management
Data scientists and their IT counterparts must work together to develop data management strategies that work for both sides.
The term “shadow IT,” which refers to technologies that are adopted without the involvement of IT, invokes a different reaction depending on who you ask. The IT department might shudder, while developers may shrug it off as a necessary shortcut to achieving agile workflows.
To close this gap in sentiment, companies have worked hard in recent years to increase awareness on both sides — developers have learned more about the security risks of going outside the IT lines, while IT has learned more about how to deliver tools and processes that match developers’ needs.
Today, we must have a similar conversation on a related topic: how to handle shadow data management. Left unchecked, this is an issue that will create many headaches, ranging from security vulnerabilities to inadequate return on investment (ROI) for AI projects.
Shadow data management happens whenever data scientists or analysts copy data from the primary, IT-approved database to run analysis with their own tools in their own environments — many of which have not gone through the rigor of IT approvals. Shadow data management carries many of the same risks as shadow IT (like increased attack vectors or disrupted workflows), and also amplifies a company’s liability if sensitive user information is included in the “shadow” dataset.
To understand how to address the challenge of shadow data management, it’s important to first explore the underlying root causes. After all, data scientists and machine learning (ML) engineers aren’t devious rule-breakers devoted to causing chaos. Rather, they’re driven to shadow data management practices because their technology needs aren’t being met by their IT department’s approaches to data governance, management and risk.
For example, I’ve heard of large organizations spending lots of money to stand up massive databases that were actually too slow to use for the models their teams needed. Frustrated by the primary database, the data scientists found a simple workaround: running a query to make the full table dump into a CSV file and then bringing in other scripts from there.
Speed isn’t the only pain point that drives data scientists to shadow data management. Another challenge is a lack of flexibility in data tooling and processes, which have traditionally centered around SQL databases. Additionally, data is often treated as artifact management, which has worked fine for many traditional business analytics workflows. However, what differentiates data science from business intelligence is the ability to experiment — simply pulling queries is not the same as experimentation and ML modeling.
What’s more, the types of data that businesses find valuable today are more varied than ever before, which means that one-size-fits-all data management solutions and tools are often insufficient. Arming data scientists with only spreadsheets and SQL and asking them to do their work is like asking them to build a jet engine with a chisel and stone. Shadow data management is invariably what results when data scientists break out of the restrictive silos of traditional data platforms.
The shadow data problem, then, is a view into the tension between the unmet needs of data scientists and the way traditional IT approaches data management. To solve this challenge, both sides need to be willing to work together on new solutions that can meet each team’s key needs.
Here are a few action steps that IT departments and data scientists can collectively take to help address the challenge of shadow data management.
