Explainable artificial intelligence: Easier said than done
The growing use of artificial intelligence in medicine is paralleled by growing concern among many policymakers, patients, and physicians about the use of black-box algorithms. In a nutshell, it’s this: We don’t know what these algorithms are doing or how they are doing it, and since we aren’t in a position to understand them, they can’t be trusted and shouldn’t be relied upon.
A new field of research, dubbed explainable artificial intelligence (XAI), aims to address these concerns. As we argue in Science magazine, together with our colleagues I. Glenn Cohen and Theodoros Evgeniou, this approach may not help and, in some instances, can hurt.
Artificial intelligence (AI) systems, especially machine learning (ML) algorithms, are increasingly pervasive in health care. They are used for things like evaluating cardiovascular images, identifying eye disease, and detecting bone fractures. Many of these systems, and most of those cleared or approved for use by the Food and Drug Administration, rely on so-called black-box algorithms. While the notion of what constitutes a black-box algorithm is somewhat fluid, we think of it as an algorithm that is exceedingly difficult, or even impossible, for ordinary humans to understand.
Examples of black-box AI models would be any of a class of algorithms ordinarily labeled as “deep learning,” such as neural networks with many layers, convolutions, back propagation, and the like.
There are two key ways to understand how an AI system operates. The first is simple and intuitive: The system’s maker can stop using the black box for making predictions and use a transparent system — a white-box model — instead. While white-box models are also a fluid concept, examples include simple decision trees or ordinary regression with a few variables, where it is easy to tell how the variables combine to form the system’s predictions. For example, many doctors use a point scoring system for calculating patients’ heart disease or stroke risk based on their blood pressure, cholesterol levels, age, and other characteristics. Let’s call these white-box systems interpretable AI (IAI).
Interpretable AI is great for increasing transparency and helping one understand how a model works. It is simple, intuitive, and easy to grasp. And to the extent that such a simple white box can be substituted for a complex black-box, we are all for it. But herein lies the problem: For many medical applications, developers need to use a more complicated model.
One example is an application that relies on image recognition, in which the number of predictor variables is extremely large and the features are often highly engineered. Another example is an application that relies on genetic data. In such cases, developers generally won’t want to substitute an advanced deep-learning system with, for example, a simple decision tree. IAI is therefore not an adequate alternative, as it may not reach the necessary levels of accuracy that more complex black-box models can achieve.
To placate those who are worried about trust and transparency, developers who insist on using black-box systems turn to the second alternative, namely XAI. Here’s how it works: Given a black-box model that is used to make predictions or diagnoses, a second explanatory algorithm is developed that approximates the outputs of the black box.
