It’s time for AI to explain itself
AI models get more accurate all the time, but even the data scientists who built them can’t explain why—and that’s a problem.
AI-driven algorithms are now a daily part of nearly everyone’s lives. We’ve all grown used to machines suggesting a new series to binge on Netflix, another person to follow on Facebook, or the next thing we need to order from Amazon.
They’re also driving far more important decisions, like what stocks to invest in, which medical procedures to consider, or whether you qualify for a mortgage. In some cases, an AI system might offer better advice than a human financial adviser, doctor, or banker.
But if you end up on the wrong side of an algorithm, there’s no person you can buttonhole to explain why your loan application was rejected or your resumé discarded.
And all too often, the companies that created and deployed these algorithms can’t explain them either.
According to The State of Responsible AI: 2021, a survey sponsored by financial services firm FICO, two-thirds of companies are unable to explain how the AI models they’ve deployed arrive at decisions. And only one in five actively monitor these models to ensure decisions are made ethically and fairly.
AI has a transparency problem. Organizations that want to earn the trust of consumers, avoid the ire of regulators, or simply determine how well their machine learning models are actually working will need to adopt explainable AI (XAI) moving forward.
While no one is suggesting that AI is about to become self-aware and start waging war on humanity, the negative impacts of automated decision-making are well documented.
The most common problem is bias introduced while the AI model is being trained. For example, facial recognition algorithms are notoriously inaccurate when detecting darker skinned individuals, most likely because the training data included fewer people of color. Predictive policing models rely on existing criminal records, reinforcing decades of racial injustice. Resume-screening algorithms based on a company’s historical hiring patterns can discriminate against women, older applicants, or people of color.
Skewed or insufficient data can also lead to inaccurate predictions, making a model useless and potentially dangerous. For example, when a Florida hospital acquired an IBM Watson system designed to help oncologists treat cancer patients, the AI recommended procedures that would have made them worse. The reason? It relied on data from hypothetical patients, not real ones. The $62 million system was scrapped.
But eliminating bias from AI models involves more than merely excluding data from protected categories such as race, gender, and age. You also need to account for data that acts as a proxy for these categories. For example, if you live in Florida and have an AOL email address, a model might assume you’re more likely to be a senior citizen.
And you need the ability to explain how an AI model arrived at a particular decision, especially when it has a significant impact on people’s lives, notes Kirk Bresniker, a fellow at Hewlett Packard Enterprise who was involved in the creation of the company’s AI Ethical Principles.
Most human-driven decisions can be traced back to their origins, adds Bresniker. With AI, that’s usually not the case.
“You can go back and ask the lawmaker what they were thinking, or you can look at lines of code,” he says. “But with AI systems, you can’t go in and audit all the way back to the source data. If we can get to explainable AI technologies you can audit, there may be more areas where we can employ them.”
Many current AI models were built using deep learning neural networks, which devise their own methods for interpreting data and whose internal operations are a mystery even to the people who designed them—hence the term “black box AI.”
These networks may contain dozens of layers of mathematical functions.
