5 Reasons That Business Intelligence on Hadoop Projects Fails
Hadoop has its own set of constraints and problems, and when you throw BI into the mix, it’s a whole other ballgame. As companies move their EDW to Hadoop, they expect to be able to migrate their analytical applications as well. While workloads such as ETL, predictive analytics, and Machine Learning work great on Hadoop, the more performance-sensitive BI applications suffer from Hadoop’s inherent low latency.
We’ll discuss these five harbingers of Hadoop doom and gloom and look at what can you do to avoid those pitfalls and be successful when setting up your own BI on Hadoop endeavor. To begin with, good planning, as is often the case, is an excellent starting point. You need to ensure that you have buy-in from management and that your internal domain experts are actively involved. Your computer hardware and network also need to be able to handle Big Data otherwise your system might be painfully slow or unreliable. Finally, in order to make efficient use of your big data you will need Business Intelligence (BI) capabilities, SQL, and the ability to run complex queries.
The first factor that can result in the failure of your BI on Hadoop project is a lack of good planning, in terms of project scope, budget, time frame, hardware (server storage, and computing power), and network requirements. Another reason is a lack of buy-in from top management. Make sure management appreciates the value of your Big Data project and your Business Intelligence (BI) framework and integrates it into their decision-making process. Historically isolated data silos in your organization also pose a barrier and need to be taken into account during planning. “60% of big data projects will fail to make it into production either due to an inability to demonstrate value or because they cannot evolve into existing EIM processes,” concluded Roxane Edjlali, Chief Data Officer Research and Advisory at Gartner.
Project costs are often underestimated, or not fully understood, resulting in poor scope and planning. A useful reference when calculating the cost of big data projects is the Forbes article The Big Cost Of Big Data. According to the authors, a petabyte Hadoop Big Data project can cost around USD 1 million, including up to 250 nodes, compared to tens or even hundreds of millions of US Dollars for an enterprise data warehouse project.
Begin with small, iterative steps, and let your organization’s internal domain and data experts experiment with Hadoop using a flexible, and open data infrastructure. Select server hardware, and network infrastructure that is up to the task. Pay attention to your Java Virtual Machine (JVM) and garbage collector, memory settings, and the Input/Output (disk and network input and output) speeds of your server hardware. Also, consider investing in NameNode High Availability (HA and secondary NameNode) and Disaster Recovery (DR) hardware.
Even if the costs are well understood, the use-case may be incorrect. This can mislead you into building a solution that meets all the requirements of the plan, but does not fix the problem you intended to solve in the first place. To prevent this, focus on the goals, and document a number of detailed use cases that will help you configure your system to them.
Figure 1: This diagram shows the relationship between the different CRISP-DM phases in a data mining project (Kenneth Jensen via Wikimedia Commons).
Over-reliance on only data experts without the active involvement of your organization’s own domain knowledge experts is also cited as a common reason for failure. You may end up building an impressive project, but it also needs to fit into your organization’s EIM model from the organizational vision and strategy, to the existing data infrastructure. You will need to adjust existing business models, and the business culture in your organization to enable improved internal data sharing and utilization.
