The Biggest Problem with Big Data: Search… or Find?

With the continuing exponential rise of data volumes one of the biggest problems that doesn’t get an airing is the continuing challenge, not of munging, prepping, and analysing, but a far more basic one – searching, and then finding.

Never mind ‘you can’t manage what you can’t measure’ – if a business can’t find the right data in the time available, it will use the data that it can find – and that may lead to outcomes ranging from less precise decisions to out and out poor decision-making.

According to a landmark 2012 McKinsey research study showed that 19% of the average work week is spent unproductively searching and gathering information. McKinsey found that social technologies, which create value by improving productivity could potentially contribute $900 billion to $1.3 trillion in annual value across four sectors they reviewed: Consumer packaged goods, retail financial services, advanced manufacturing, and professional services. McKinsey estimated that the average interaction worker spends 28 percent of the workweek managing e-mail and nearly 20 percent looking for internal information or tracking down colleagues who can help with specific tasks.

A statistic by Outsell showed that an engineer’s time spent searching for information has increased 13 percent since 2002.

And a survey by SearchYourCloud revealed that workers can take up to eight searches to find the right document and information they need.

One final stat: An IDC research paper has shown that “the knowledge worker spends about 2.5 hours per day, or roughly 30% of the workday, searching for information….60% [of company executives] felt that time constraints and lack of understanding of how to find information were preventing their employees from finding the information they needed.”

There are a multitude of such studies repeated over the years. It all goes to show that data is unambiguously a respected resource, but that the search and find aspect of managing it has become a troublesome part of the knowledge worker’s role.

Part of the problem is purely an issue of volume. Many organisations keep so much data that of course sifting through it becomes a challenge. And yet there are simple ways to help alleviate the chore that finding the right data has become in the minds of many business analysts, simply trying to find the data they need to get their jobs done.

Taking inspiration from social tools, there’s no reason that means that analysts have to reinvent the wheel, and investigate the same data sets every time in their search. In the same way that tools like Slack and Asana have revolutionised the way distributed teams can collaborate, there are ways the world of data can draw on similar functionality to help them rate the reliability and utility of the data at hand.

In the same way that popular posts can be uprated on Reddit, liked on Facebook, or commented on in Asana, valuable data sets, trusted and clean numbers can be preserved and assigned trust ratings. That way the business preserves knowledge and speeds the process of getting to the right answers faster, time after time.

This is important.