can machine learning transform data management?

Do data scientists really spend 80% of their time wrangling data? Now it’s time for the follow-up: can machine learning make a difference in data management? Can it alter that 80/20 data cleansing ratio?

Machine Learning (ML) is a term that can mean just about anything. In evaluating a (proprietary) tool for data management that claims to use machine learning, you should understand what that means. It isn’t necessary to see the math or even the code that implements the algorithm.

It should suffice to understand what the algorithm evaluates, at least a high-level explanation of how it operates and what it produces.

Keep in mind that the fundamental workings of the algorithms are usually proprietary, so the explanations, if given, will be pretty high level. How coherent the explanation is, though, should help you understand what is real.

Despite its lofty name, machine learning isn’t that mysterious. The most popular algorithms in use today are pretty mature. What makes them “machine learning” instead of just statistical models is the use of massive amounts of data, which was not previously possible. Some machine learning algorithms that are common in use are:

There is a tendency to conflate machine learning with Artificial Intelligence (AI). There are two general fields of AI. The first, Artificial General Intelligence (AGI), is about machines having human-like cognition and human intelligence, but there is some disagreement about when or if we will reach that threshold. Each new bold advance in what appears to be AGI demonstrates that what was assumed to be intelligence turns out not to be. Facial recognition is a good example. The other is what is in place now: non-sentient machine intelligence, typically focused on a narrow task. This is where machine learning and AI get mixed up.

For an ML algorithm to learn, it sifts through lots of data using a variety of statistical, non-parametric and other quantitative algorithms to find relationships, patterns and connections in the data. According to Judea Pearl, the Turing Award winner and author of “The Book of Why: The New Science of Cause and Effect,” ML cannot understand cause and effect. ML without causal capabilities, as Pearl derisively claims, “is just curve fitting.”

Pearl has led the field in the issue of cause and effect, and while there is some truth in his comment, there are many applications for ML that are “just curve fitting.” For example, sifting through billions of records to find what relates to what and how strongly, and then having analysts or data stewards the opportunity to edit those findings. That’s how ML actually “learns.”

For a data discovery/relationship discovery process to tie to a data catalog, the essential abilities are:

The real magic in applying machine learning models to a software product is producing the right mix of things that are general enough to work with a wide range of situations and powerful enough to produce non-trivial results repeatedly (useless example, “Most auto injury accidents occur when the driver is at least 16 years old.”)  Supporting data science with Integrated (no code) tools requires creating and maintaining a comprehensive data catalog, but a few steps precede it.

If you think about it, the most crucial part of managing collections of unalike data is finding relationships. Finding relationships between so many forms of data is practically impossible to do by hand. When dealing with tabular/columnar data, figuring out what names are likely to point to similar kinds of data (though not consistently accurate). Instead, the magic investigates the actual data to determine what it is.

To put this in perspective, if you have a few billion instances to compare, this can be a computationally expensive (read, slow) process. Here is the first example of machine learning boosting the process.