Building a Modern Data Stack
There are numerous reasons why organizations pursue a “modern data stack.” Some aim to move away from a “silos and spaghetti architecture” described in “Future Ready,” while others aim to establish an AI factory and its supporting data marketplace, as described by Marco Iansiti and Karim Lakhani. CMOs are interested in creating omnichannel engagement with customers for their desired products and services. Data practitioners and CIOs are left to ponder the implications in Andreessen Horowitz’s 2020 analysis of modern data infrastructures and what the impact will be on businesses.
The authors of “Future Ready” describe creating a modern data stack as “industrializing data” and see it as crucial in making data a strategic asset that is accessible to all who need it, resulting in a modular and agile organization. A reusable and modular data platform that fixes integration, cleans data and provides a single view of the customer is necessary for success and requires an industrialized data foundation.
As a goal, modern tools should get organizations out of the tedious, expensive data governance, mapping and classification work. It should at the same time eliminate data being siloed in multiple and proprietary data management systems. According to Craig Milroy, former chief data architect for TD Bank, “The data lake architecture on Hadoop was the modern data stack not too long ago; now it is data mesh and data lake houses. While I am all for the decommissioning of Hadoop, I think organizations should think through the business capability enablement in selecting the next data stack.” Miami University CIO, David Seidl, goes on to suggest, “This all really depends upon the organization, its scale, maturity and business needs. But a good general answer remains data culture. Data culture still is one of things you need to build to guide the data stack you build.”
Many CIOs believe that starting with clearly defined business outcomes is essential. To attain business outcomes from digital transformation and AI, a modern data stack is foundational. It’s the central nervous system for an organization. Undoubtedly, attempting to digitally transform with legacy data stacks will impede the transformation processand increase technical debt within the organization. Put simply, a modern data stack should enable a business to be data driven, to gain insights faster and to unlock the value of digital assets and enable innovation. It is without question the starting point for digital transformation. Milroy says, therefore, “a modern data stack should result in less data silos, less tech debt, more data exchange (internal/external), self-service data access, and data governance (understood data including data quality); and exceed business expectations.”
The Andreessen Horowitz architecture clearly is slanted toward cloud solutions. However, former BusinessWeek CIO, Isaac Sacolick says, “There isn’t a universal answer to a CIO’s data management strategy. That said, in my opinion, most companies will use public clouds for analytics and edge when there’s a performance and cost benefit. Large enterprises will shift consistent workloads, ETLs for example to data clouds when it’s cheaper.
