The streaming data paradigm shift for legacy systems
No doubt about it – there’s a lot of streaming data these days, especially when you consider web activity data streams, Internet of Things (IoT) devices, self-driving cars that generate loads of data, smart cities, etc. In fact, documenting thoughts about the need for improved information management processes and procedures to handle the volume, velocity and variety of the myriad numbers of streaming data sources is sort of like shooting fish in a barrel.
I was in a strategy meeting recently that made me rethink the concept of streaming data in the context of much more mundane (and much more “legacy”) systems. Here is the scenario. The customer is reviewing their current products used for data integration, entity resolution and master data management. One question that was raised had to do with reconsidering the approach used for identity resolution. The current approach extracts and then matches collections of records in batches, assigns unique identifiers to matched records, and then creates a master index that links all source records together that represent the same entity.
Apparently, the client had been introduced to a number of newer MDM products that did not build a traditional master data index. Instead, these products combined some of the conventional methods of similarity scoring (that is, determining how closely related any pair of records are) with more innovative big data platform tools for distributing data, indexing objects and parallel searching to effectively create a dynamic capability for approximate record linkage. In other words, as opposed to processing batches of data to form a master index, the system maintains an inventory of entity data objects, and searches and matches records on demand.
If you work for an organization whose systems largely operate in batch, this approach to master data management is not only foreign, it does not make sense at all. For example, how could you facilitate the population of the customer domain for a static data warehouse if you are not able to extract that domain from your MDM system?
To what would one attribute this alternative take on MDM? A lot of it boils down to use cases and consumption patterns.
