How to Build Real-Time Data Pipelines from Traditional Systems
Most modern enterprises are running multiple complex systems, including a mix of on-premises and cloud software, often deployed in various locations around the world. Typically, organizations must grapple with a host of different data models and nomenclatures, as well as adjust to a constantly changing landscape in which new systems are brought online, outdated systems are retired, or corporate acquisitions or mergers call for the integration of existing ERP, CRM, or similar business management systems. Building real-time data pipelines can help address these challenges.
For established organizations, traditional systems such as the mainframe present a particular challenge. Very often, the businesses running such systems have invested heavily in existing code. Understandably, many in those organizations are hesitant to walk away from those established investments. After all, the process of recreating complex systems from scratch is risky, expensive, and potentially highly disruptive.
Yet systems like the mainframe present a particular challenge because they often operate as functional silos. Integration is difficult given the mix of complex data formats, hierarchical IMS databases, VSAM files, and COBOL copybooks, among others. Most integration tools are poorly suited to tackle the data access challenges that are unique to mainframe environments.
This leads to a situation in which organizations must struggle to roll out new innovations and access to information may be limited. Connecting cloud-based systems to the mainframe can be cumbersome and time-consuming. Efforts to build out advanced analytics or AI and machine learning may be hampered by integration bottlenecks that make it difficult to gain real-time visibility.
Real-time access to information is becoming increasingly important. In an age of connected devices, clickstream analysis, and the rapid flow of information, it is no longer acceptable to wait for an overnight update to get the analysis you need. Fraud detection in payment processing, for example, relies upon the early identification of anomalies and quick action to resolve questionable charges. Too many card declines will lead to customer dissatisfaction. Too few will result in fraud losses.
Monitoring critical systems, likewise, relies on real-time visibility to large volumes of data. In many cases, it is essential to identify potential problems quickly, especially when a rapid response may be necessary to avoid system failure.
To achieve real-time visibility, organizations must have tools that can keep data in sync without overloading networks or adversely affecting database performance. Integration tools must be robust enough to handle situations in which connectivity fails. This calls for automatic restart capability, with a guarantee that no data will be lost in the process.
The primary challenge of integrating traditional systems with modern cloud environments arises from the complex data formats previously mentioned (ISM, VSAM, COBOL copybooks, etc.). Experienced mainframe programmers who understand these formats are getting harder to find, particularly as many of them are approaching retirement age. Without a workforce to replace them, the talent shortage is growing in severity.
Integrating a wide array of modern systems, likewise, presents a challenge when it comes to hiring and retaining talent with the right mix of skills.
