Lost in Translation: Big Data Innovation is Missing the Point

Business Intelligence (BI) as currently practiced, on premise or in the cloud, with new tools or old, does not work. Why? Recent innovation in Big Data has focused on the wrong problem: building a better mouse trap, rather than innovating the entire approach.

For example, take Tableau or Qlik. Both offer significantly better Data Discovery than the previous generation of tools; however, despite marketing claims around ease of use and a low level of technical expertise needed, these tools are still designed for experienced analysts and data scientists. Unless highly technical, a business user cannot put them to good use.

This problem exists across the entire BI project lifecycle:

First, someone very technical needs to create a data mart (or warehouse or lake). That person is, by definition, a technologist, not a business expert. Therefore, interpretation of requirements is an essential step in this process. Much is lost in translation during this step. The usual answer is to load everything including the kitchen sink into the data mart, which then makes it unwieldy, even more expensive to maintain, and costs a mint to store and to keep current. Creating a data mart takes time, and business requirements change constantly. Therefore, data mart design is usually not current.

Second, extracting insights from the data mart is an iterative process. An insight leads to unanticipated questions, requiring a new insight. Each insight is a result of a number of queries, metrics, visualizations, and stat models. Each insight requires a sophisticated application, one that typically depends on a technical person to create. Since the technical person creating the application is usually also not a business expert, much is lost in translation here as well. And, because many business users lack the analytical or statistical training to define a productive discovery process, the iterations are ad hoc, lead to blind alleys, and lack of understanding of the result.

So what to do?

Like most technologies, BI is a tool on which someone needs to build a solution for practical daily use and consumption by non-technical business users. The future of BI lies in creation of Applied Business Analytics solutions – solutions built to meet specific business needs, to answer specific business questions, which are delivered as a service and at scale to businesses and end users.

For most business processes, there are documented best practices, with differences by industry sector, geography, or size of the organization. These best practices can be interpreted and converted into analytics that measure, forecast, and model future outcomes based on the understanding of the process parameters which influence a given outcome.