The rise of modern architectures based on change data capture technology
Data is creating massive waves of change and giving rise to a new data-driven economy that is only in its infancy. Organizations in all industries are shifting their business models to monetize data, understanding that doing so is critical to competition and even survival. There is tremendous opportunity as applications, instrumented devices and web traffic are throwing off reams of 1s and 0s, rich in analytics potential.
These analytics initiatives can reshape sales, operations and strategy on many fronts. Real-time processing of customer data can create new revenue opportunities. Tracking devices with Internet of Things (IoT) sensors can improve operational efficiency, reduce risk and yield new analytics insights. New artificial intelligence (AI) approaches such as machine learning can accelerate and improve the accuracy of business predictions. Such is the promise of modern analytics.
However, these opportunities change how data needs to be moved, stored, processed and analyzed, and it’s easy to underestimate the resulting organizational and technical challenges. From a technology perspective, to achieve the promise of analytics, underlying data architectures need to efficiently process high volumes of fast-moving data from many sources. They also need to accommodate evolving business needs and multiplying data sources.
To adapt, IT organizations are embracing data lake, streaming and cloud architectures. These platforms are complementing and even replacing the enterprise data warehouse (EDW), the traditional structured system of record for analytics.
Enterprise architects and other data managers know firsthand that we are in the early phases of this transition, and it is tricky stuff. A primary challenge is data integration—the second most likely barrier to Hadoop Data Lake implementations, right behind data governance, according to a recent TDWI survey, “Data Lakes: Purposes, Practices, Patterns and Platforms.”
IT organizations must copy data to analytics platforms, often continuously, without disrupting production applications (a trait known as zero-impact). Data integration processes must be scalable, efficient and able to absorb high data volumes from many sources without a prohibitive increase in labor or complexity. The table below summarizes the key data integration requirements of modern analytics initiatives.
All this entails careful planning and new technologies because traditional batch-oriented data integration tools do not meet these requirements.
