Solving One ML and AI’s Biggest Challenges: Exploratory Data Analysis
As organizations invest in Artificial Intelligence (AI) and Machine Learning (ML), they inevitably run into a paradox. The automation that ML models offer can be a critical advantage for many businesses. In theory, automation should drive down labor costs, but with each additional model comes a maintenance overhead burden many organizations underestimate.
While a lot of emphasis gets put on model building, data scientists must continually explore and analyze the underlying data for shifting patterns, data quality issues, and unforeseen changes to the business, which will erode the performance of even the most sophisticated pipelines. And it’s in this exploratory data analysis that things often go wrong. The work is often done manually and must be done frequently to catch potential changes in the data, which is time-consuming and arduous.
Organizations underestimate just how much effort will need to be spent on exploratory data analysis and as such, it often simply doesn’t get done. Ultimately, most AI and ML initiatives fail because stakeholders don’t catch and remediate data issues. One survey puts the failure rate at 87%.
The prudent scientist must interrogate the data with a laundry list of statistical questions to determine the data’s fit-for-use in AI and ML projects. Are there enough data points? How many data points are good and actually usable? Are there missing values or bad values? Are there any outliers? And on and on, each question branching out into a multitude of questions that need to be answered to preempt most if not all cases that could derail the project.
Worst of all, this process isn’t a one-and-done activity! The business processes generating the data are inherently dynamic. As a result, a significant amount of time must be spent revisiting the data pipeline to ensure the data feeding into ML models are consistent. Airbnb found that nearly 70% of the time that data scientists spend developing ML models is allocated toward this crucial and thorough but lengthy exploratory analysis. This oceanic wave of unchecked overhead can easily drown out the automation benefits we strive to achieve with our projects.
Conventional data testing can help, but the usual test coverage won’t catch important changes in the data intimately related to your business. For instance, tests won’t shed light on the “unknown, unknowns.” If you aren’t aware that an issue could occur, then it is impossible to create a test to catch that issue. As the saying goes in boxing, “it’s the punch that you don’t see coming that knocks you out.”
Throwing more data scientists at the problem is rarely the answer for most companies. It isn’t cost-effective, and data scientists, and everyone else for that matter, simply don’t like this kind of work.
