The explosion of data, ranging from social sources like Facebook to sensors in the Internet of Things, has created a need to stem the flood and make sense of what is in there. There is also the promise of new insights gleaned from patterns in the data that will predict scenarios and not just report on what happened in the past.
This has compelled many organizations to hire data scientists to cast light on the mysteries found in NoSQL databases and data lakes where all this data is being accumulated. However, the results have often been rather disappointing. In a white paper onData Science, Pivotal reports that $41 billion was spent on data science software and infrastructure in 2015, according to research by IDC. This excludes staffing and other costs. However, the ROI for all this expenditure exceeded three percent for only 27 percent of a global survey conducted by Teradata for Forbes Insights, in which they interviewed 316 executives of businesses with a minimum turnover of $500 million.
So why the big discrepancy between expectations and delivery? There are a number of factors contributing to this disappointing state, such as the following:
Traditionally, business intelligence has been restricted to the IT division, which produced standard reports, using historical data from structured databases and data marts. Ad-hoc requests could take weeks or even months to be delivered. In this era of big data, this is not good enough or fast enough.
Becoming a data-driven organization requires everyone in the company to have an unfettered access to data, limited only by security considerations. Once they have the access, they should be encouraged to use it and familiarize themselves with what is available. This should start as a top-down initiative, with executives showing the way. This is a big cultural change and is probably the hardest part of the data-driven journey. The Forbes survey found cultural issues to be prominent in most companies, even those who were succeeding in their transformation.
There are many aspects to becoming data-driven, and not the least of them is how to manage the tsunami of data that you will need. Master data management in the age of big data is complex and we discuss it in a separate article. What is more, analyzing data is not an end in itself, since someone needs to apply the analytics if they are to have any value. While there is much buzz about AI and machine learning, in the short- to medium-term this job is still reserved for humans.
You also need to determine where you will apply analysis first; maybe risk management, for example, or demand forecasting. Build an attainable roadmap, engaging as many of your employees as possible, as the start of your change management process, and implement a communication program to report progress to all employees and stakeholders. You will also need to make adjustments to your business model and organizational design if you want to embed data science in your business.
There are two schools of thought about how you deploy data science and analytics – centralized or decentralized. The centralized one is closest to the traditional business intelligence set-up. Most BI business units are located in IT and report to the CIO. Ideally, you need a CDO (Chief Data Officer) who is on par with your CIO, to whom the data science/analytics team will report. The analytics team should include business users, who will both initiate the business need and be responsible for applying the output.
This is why many companies have a decentralized approach, where data scientists and analysts are deployed to different business units. Obviously, this is viable only for large-scale enterprises who can afford to appoint enough data scientists. You could look at a long-term co-sourcing option, where you retain the services of a company that has the necessary big data experience to perform the analytics.
