As the hype cycle for AI continues it is going to be increasingly important for users to ask more questions about the limitations of these systems as well as ask what types of problems will machine learning models such as Deep Learning be best suited. The market is filled with claims of both miracle cures and the dangers of AI and we need to start asking more targeted questions to cut through the clutter and avoid decisions that could have a negative impact on health outcomes. One often-cited case below illustrates the need to understand more about causal inference in AI/ML so that better assessments can be made.
A study from the 1990s illustrates how AI can go wrong and become potentially dangerous in healthcare. In the 1990s researchers sought to create a pneumonia risk model that would classify patients in either “high risk” or “low risk” categories. Low-risk patients were to be treated with antibiotics, chicken soup, and told to call back in three days if they were not significantly better. High-risk patients were to be hospitalized. Researchers trained a neural network on the data and an early version of the model suggested sending asthmatic patients home. A graduate student at the time, Rich Caruana, picked up on the fact that this recommendation was likely flawed. The algorithms “learned” that asthma patients presented earlier and the hospitals hospitalized them in intensive care and improved their outcomes faster. The AI/ML algorithm saw the improved outcome earlier in the progression and interpreted this as cause for lower risk and less likely to die, when in fact, they are at greater risk of death.
This example, albeit on less sophisticated neural networks than now in use, is still extremely relevant. With an aging population with co-morbidities, there is a lot of room for confounding variables to trip up algorithms and make poor recommendations. The vast majority of AI/ML tools in the marketplace are based on regression analyses. Regression analysis is great for understanding associations, but less powerful in helping understand causal mechanisms. Whether it is a policy issue or a clinical issue, causal mechanisms are essential to most cases in understanding where and how to intervene. The failure of many AI/ML applications will likely be found in the murk of causality. Furthermore, many of our most complex health policy challenges can be described as wicked problems lacking clear solutions and we should only expect Deep Learning to provide a tool for analyzing components of the problems rather than some sort of magic wand.
We need to stay focused on the issue of causal inference and work with data scientists who are developing AI/ML approaches to health and medicine that incorporate this into their work and what comes after the current craze for Deep Learning. We may also need to find mechanisms to open up black-box algorithms for greater transparency and analysis, or alternatively use white box algorithms where models can be assessed more transparently for patient safety reasons. For example, if we use historical data to train algorithms used to predict or prescribe an individual patient journey from data based on treatments up to 2017 and learned that those past treatments were not the optimal treatments and that clinical practices had changed, this ML model may expose patients to sub-optimal treatment or harm.
