Getting the Data You Need: Connecting Data Producers and Data Consumers

Data holds great value, but only if you can find it and access it. That may sound simple, yet in the real world, it’s not.

Some people think they’ve solved the problem if they’ve asked permission to use someone else’s data and have gotten a “yes” in response. But getting permission is just scratching the surface of the challenges that arise in data sharing, whether it is between departments within an organization or between organizations. How do you find the right data? How do you even know it exists? Can you trust it? Is it stored in a form your applications can access?

The challenges for getting the data you need can form a significant barrier to data getting used, and as a result, a lot of data lies dormant. “Finding, accessing, and using data effectively is one of the biggest challenges companies will face,” says Janice Zdankus, VP of Innovation for Social Impact in Hewlett Packard Enterprise’s Office of the CTO.

One reason just getting permission isn’t enough is that often data wasn’t collected specifically for the data consumer’s project. Data science – from analytics to AI – frequently makes use of data in different ways than its original purpose. Customer transaction data, for instance, may have been collected for billing purposes, but it also provides a rich source of training data for online recommendation systems, fraud detection, and supply chain predictive analytics. That’s one reason it’s useful to run AI and analytics applications together, on the same system.

These examples illustrate the fact that context gives data value, as pointed out by Joanna McKenzie, Principal Data Scientist at The Data Lab – Innovation Centre in Scotland, in a recent AI and data panel “Data Sharing for Data Science.”  Weather data, for instance, is obviously useful in making predictions about agricultural conditions. But one might produce better results by combining weather data with socio-economic or other human behavioral data in use cases such as predictive analytics for retail marketing or healthcare and disease prevention.

But to take advantage of the added value of combining diverse data sets, you may have to deal with mismatches in data type, in granularity, or many other parameters. Let’s look a bit deeper at some of the common challenges encountered in data sharing.

The gap between data producers and data consumers underlies a major challenge in getting the data you need: data discovery. To find and use the data you need, you first have to know it exists. That’s not a trivial issue, as data generated by data producers often is siloed, whether within an organization or even from open-source data sets. Data discovery is even harder given the huge proliferation of different types and different sources of data.

Data trustworthiness is another fundamental challenge, particularly when there is a gap between the data consumer and the data producer. Even assuming all parties are responsible and have good intentions, it still may be difficult to find out exactly what a particular data set really is, how and when it was collected, who else is using it, and whether you can trust that it is a reliable source with an assured lifetime that fits your requirements. Suppose you need to revisit a data set for retraining AI models or to respond to regulatory audit requirements? Has the data been changed or “cleaned up” by someone using it for other reasons? Proper data curation and governance is challenging, especially when data is collected without knowing how it will be needed by multiple data users.

Data accessibility by multiple users and a variety of applications is a fundamental requirement for practical data sharing.