In Automation We Trust: How to Build an Explainable AI Model

As AI becomes more advanced and complex, the algorithms and logic powering it become less transparent. This lack of clarity can be unnerving for some

people. Recent high-profile AI failures illustrate this. For example, Amazon built AI-powered recruiting software to help review and recommend applicant resumes, but failed to recognize that the data feeding the algorithm was biased toward male job seekers. Apple and Goldman Sachs used machine learning to determine creditworthiness of Apple Card applicants—only to realize that the credit history data the algorithm was based on is inherently sexist.

Amazon and Apple learned in a very public way that AI models are only as good as the data that you feed into them, and if the data is compromised, discriminatory or otherwise flawed, you’re not going to get accurate results.

Consumers and regulators are catching on. Research suggests that consumers are more likely to use AI if they are able to even slightly modify its algorithms, and they’ll be more likely to trust the algorithms if they are able to play a role in its decision-making processes and have some control over its outcomes—especially when those decisions are derived from their own user data. They’re pressuring for a right to explanation to be given for the outcome of an algorithm and investigating new methods for delivering explainable AI or bringing the human-in-the-loop to get deeper control of the algorithms and the data that drives their outcomes.

This change is crucial. It’s human nature to forgive human error, and hold AI to a much higher standard. Machines are supposed to be infallible, and they’re often marketed that way. Autonomous driving is a great example.

According to the National Highway Traffic Safety Administration, 37,000 Americans died in car accidents in 2018—only one of which involved an autonomous vehicle. Yet, Americans continue to show distrust in self-driving technologies because they’re unfamiliar with how they work. Even more importantly, today, when a car accident happens, the driver is the primary responsible party. Until manufacturers can explain what happened in each and every accident with their cars and take accountability when they happen, the dream of a self-driving car cannot become a reality.

In order for AI to truly succeed on a large scale, consumers need to trust the algorithms that power the decision-making process. This requires transparency into AI models and the data that is being fed into the algorithms, giving users peace of mind that their personal data is being used appropriately to inform and improve decisions.

This can be achieved using explainable AI. From a backend development perspective, explainable AI allows model developers, business users, regulators and end users to better understand why certain predictions are made and to course correct as needed. Further, it enables them to correct and improve their models before they’re deployed at scale.

If Amazon or Apple had built explainable AI into their development processes, they likely would have identified the inherent bias in their data and been able to amend the weight of certain criteria to get a more equitable, desired result. They didn’t, and so faced very public, very embarrassing AI fails.

Here are five things to keep in mind when implementing explainable AI in your AI models to gain user trust:

A car manufacturer is not sued for injuries or deaths as a result of speeding. A maker of kitchen knives is not held accountable if one of their products results in a chopping accident during dinner prep. Humans make the poor decisions that lead to those actions, not the manufacturer or designer.

AI changes that dynamic. If AI is providing guidance to the user, the responsibility continues to sit with the user. If AI is automating the decision without human input, there is a shift of responsibility.