The Question Medical AI Can’t Answer
Artificial intelligence (AI) is at an inflection point in health care. A 50-year span of algorithm and software development has produced some powerful approaches to extracting patterns from big data. For example, deep-learning neural networks have been shown to be effective for image analysis, resulting in the first FDA-approved AI-aided diagnosis of an eye disease called diabetic retinopathy, using only photos of a patient’s eye.
However, the application of AI in the health care domain has also revealed many of its weaknesses, outlined in a recent guidance document from the World Health Organization (WHO). The document covers a lengthy list of topics, each of which are just as important as the last: responsible, accountable, inclusive, equitable, ethical, unbiased, responsive, sustainable, transparent, trustworthy and explainable AI. These are all vital to the delivery of health care and consistent with how we approach medicine when the best interest of the patient is paramount.
It is not hard to understand how an algorithm might be biased, exclusive, inequitable or unethical. That could be explained by the possibility of its developers not giving it the ability to discern good data from bad, or that they hadn’t been aware of data problems because these often arise from discriminatory human behavior. One example is unconsciously triaging emergency room patients differently based on the color of their skin. Algorithms are good at exploiting these kinds of biases, and making them aware of them can be challenging. As the WHO guidance document suggests, we must weigh the risks of AI carefully with the potential benefits.
But what is more difficult to understand is why AI algorithms may not be transparent, trustworthy and explainable. Transparency means that it is easy to understand the AI algorithm, how it works and the computer code doing the work behind the scenes. This kind of transparency, in addition to rigorous validation, builds trust in the software, which is vital for patient care. Unfortunately, most AI software used in the health care industry comes from commercial entities who need to protect intellectual property and thus are not willing to divulge their algorithms and code. This likely results in a lack of trust of the AI and its work.
Trust and transparency are, of course, worthy goals. But what about explanation? One of the best ways to understand AI, or what AI aspires to be, is to think about how humans solve health care challenges and make decisions.
