Why Is Big Data Analysis So Challenging?

There’s gold to find in the big data forest, but most companies have no map and no crew.

A new research report from TDWI, titled Data Science and Big Data Enterprise Paths to Success, outlines the state of big data and data science: In short, it’s getting bigger and more difficult. On a scale from 1 to 5, with 5 meaning “completely satisfied” with the current data management strategy, only 3 percent of respondents gave a 5 answer.

Roughly 43 percent were right in the middle, and nearly 40 percent offered a 1 or 2.

Part of that dissatisfaction might be because of the sheer amount of data being collected. Twenty percent of the survey respondents are trying to work with 10-100 terabytes, and 17 percent have anywhere from 100 terabytes to more than a petabyte. Most of this data is structured data right now, but companies understand the need to quickly figure out plans for integrating that reliable data with the more unpredictable new inputs. And Hadoopis the big data platform of choice, generally—30 percent of all respondents use Hadoop on-premises today, but for those managing more than 10TB of data, that jumps to 50 percent.

Among the types of data being managed, some are growing far more rapidly than others. Text/content data from emails, call center notes, and claims is growing extremely fast, as is external social media text data.

While most of the respondents are using data science to make traditional reporting and analysis queries, a solid 53 percent are also using it for visual analytics. Predictive analytics is rising quickly as well—collecting text/content data from emails, call centers, and social media is growing rapidly, and will likely create the foundation necessary better understand how customers will react to a new product or a response from customer service.

The data scientist has existed for quite some time now, but that role has recently become much more complex as companies try to convert their big data assets into real value. In the past, data scientists have been predictive modeling professionals—part computer scientist, part statistician, part mathematician, and part business analyst.

That role is changing for a number of reasons, one of which is the advent of what Fern Halper, a VP and senior research director for advanced analytics at TDWI, is calling the “citizen data scientist.” These people are the “next generation of statistical explorers” who are generally self-taught and want self-service access to the tools and data they need to make decisions. Being business users, they tend to not have formal training in statistics, but are taking advantage of easy-to-use analytics platforms.