The Data Dilemma: Four Common Barriers To AI Success
With all the incredible results in today’s generative AI, is this when robots finally take our jobs?
Historically, new technologies become a tool for people to change and improve the jobs that need doing, and AI is no different. Many organizations are investing in AI-powered solutions to enable faster problem-solving and better decision making, yet many struggle to see positive results.
Done right, AI can make companies more efficient by unlocking data-driven insights to save costs or help generate more revenue. AI is a disruptive force and can empower organizations to provide new services or enter new industries. Done wrong, it can lead to disgruntled employees, loss of revenue, regulatory issues and even brand damage.
Given these potential benefits, I get asked by executives if there is a surefire way to achieve AI success. While there are steps that companies can take to best understand and use their data in the right way, it is not necessarily the solution they want to hear.
But first, we need to unpack why companies encounter barriers to AI success. It all starts with data, because without it, our ambitious AI tends to get hangry.
When organizations first adopt AI, they typically invest in a state-of-the-art data platform or a new data science widget. They’ll hire Ph.D. data scientists to turn data into models and dollars.
AI needs data to train on, so as these data scientists eagerly open their computers, they’ll hit the first obstacle: What data is available, and how can I get it?
The data may simply not exist, meaning it must be created or gathered from scratch. Or it exists, but it is not accessible. Even if the data is accessible, there may be privacy or regulatory issues that prevent it from being used.
The second obstacle concerns data quality. If the data is of poor quality (missing key fields, mixed data types, outdated data), the AI it is fed to will inevitably be trained poorly. The data patterns it learns will be skewed or outright wrong, leading to poor results.
The third obstacle—one that is less often spoken about—concerns data observability, which is the ability to understand how well an AI product is running.
Imagine an AI model that predicts consumer behavior. Initially, it seems accurate, but behaviors can change over time. For instance, what if customer behavior patterns change due to inflation or a recession? Or what if the data pipeline feeding the AI model breaks, meaning it is starved of its precious data nutrients?
You need real-time data that can tell you whether the model is shifting or if the quality of data is falling below a necessary threshold. Data observability requires establishing a system to warn you about any issues and allows a human to inspect any problems. Given that an AI could be making thousands of decisions at any moment, it is essential to have guardrails in place.
Let’s imagine that the data scientists overcome these three obstacles.
