Data Models: Beauty Is in the Eye of the Implementer
The data vault model and data warehouse automation are worth investigating if you are about to embark on a new data warehouse project.
In a recent TDWI Upside article, I suggested that data models will be beautiful again, pointing to new techniques at the conceptual and logical levels that connect users’ requirements with the data representations needed to support them. These new modeling approaches allow business users and IT to collaborate closely on valid data definitions and provide agility to finesse the model or substantially rework it as the business changes.
Readers engaged in the more hands-on aspects of implementation of such models, whether data warehouse, mart, or lake, may well ask: that’s all very well, but are there any advances at the logical-physical level to make our lives easier? What about actual implementation?
One important advance that has been growing in popularity over the past decade is the “Data Vault” approach, pioneered by Dan Linstedt and now at version 2.0. This approach, especially when combined with data warehouse automation, offers several advantages, particularly in the areas of enterprise scope and agility, two aspects often considered irreconcilable.
Support for scope and agility, as well as for temporal data, is vital in modern data warehousing. However, understanding how these issues have been addressed takes us back to the old wars between the “Inmonites” and “Kimballites” of the 1990s.
Per Bill Inmon, a data warehouse should be “subject-oriented,” implying tables that are enterprisewide in scope — highly normalized implementations of the main entities of an enterprise data model. They therefore require up-front, enterprisewide negotiation and definition. This typically delays initial implementation, although this may be mitigated by staged implementation methodologies. These models are also notoriously difficult to change afterwards as business needs evolve.
Furthermore, Inmon’s suggested snapshot update approach fails to properly address temporal data, although bitemporal data structures combined with incremental update strategies can address this problem, as I showed in my 1997 book, Data Warehouse — From Architecture to Implementation.
Ralph Kimball’s star schema dimensional model approach, on the other hand, focuses first on quick-win, departmental solutions (data marts) optimized for a common set of slice-and-dice analysis needs. Although later extended with enterprise bus architecture, full cross-enterprise support remains a challenge, as does ongoing modification in the light of business change. In addition, Tom Johnston challenged the validity of the approach’s support for temporal data in his 2014 book, Bitemporal Data: Theory and Practice.
