The Data Warehousing Sanity Check
The “big data” era is still very much upon us, ushering in an age of constantly evolving technologies and techniques. Many wonder whether the enterprise data warehouse (EDW) still has relevance in the industry, particularly since many new alternatives exceed the technical capabilities of the traditional EDW at a drastically reduced cost. A year ago, I wrote that the EDW is still a sound concept, albeit one that needs to evolve. That sentiment is still true today; however, the rise of several groundbreaking technologies in the last year makes it clear that meaningful evolution is occurring.
At the core, all EDWs share the central concepts of integration and consolidation of data from disparate sources while governing that data to provide reliability and trust, enabling credible reporting and analytics. But in today’s world of high demand for analytics-driven business decisions, is credible reporting enough?
Over the past few decades, use of the EDW has proven to be a worthy but insurmountable undertaking, with a relatively low success rate. In fact, generally accepted survey data indicates that 70% of data warehouses ultimately fail. Of the 30% of “successful”
EDWs, many will never achieve ROI or strong user acceptance. EDW failures can largely be attributed to legacy interpretations of the design and traditional waterfall software development lifecycle (SDLC) approach. A current trend that is helping EDW projects succeed is the use of more modern, agile techniques.
