AI and the question of explainability
Do we need to understand our models in order to trust them?
The development of artificial intelligence and neural networks has greatly impacted the field of Computer Science on the one hand, but it has also drawn significant attention to the field of Neuroscience. The concept of artificial neural networks is inspired by the biology of neurons, in that artificial neurons were designed to rudimentarily mimic how neurons take in and transform information.
As a Neuroscience student, I was initially attracted to neural networks from the perspective of understanding our own cognition better via computational models. However, the fundamental qualitative differences between human and artificial intelligence make it very challenging for us to understand a model’s decision making process in human terms.
AI that is currently in use operates as a “black box” — we can test levels of accuracy on a test set, and then deploy our model on new data with a certain confidence in its outputs, but if challenged, we would struggle to actually explain a model’s decision in individual instances. As a response, the field of explainable AI (XAI) has emerged in research, which aims to develop techniques where the process leading to a model’s output can be understood by humans. This is a challenging task with many hurdles to overcome, which leads to the question of whether interpretability is really necessary from a practical perspective. This question becomes more pressing with the rising popularity of sophisticated machine learning systems, which are not anymore limited to tech titans like Google or Facebook. Small to mid-sized businesses are automating certain procedures within their systems, which will progress into larger-scale automation of decisions that carry substantially more weight, to autonomous control. Technology trends in businesses usually give a good preview of upcoming changes in institutions and governments, where adoption will increasingly follow. With this as an outlook, the discussion of conceptual issues like explainability becomes increasingly urgent.
We will proceed by first looking at three examples of academic approaches that aim to tackle interpretability from different angles. Then we’ll look at the topic conceptually, and discuss whether interpretability is actually necessary in practice . (Spoiler alert: it depends)
Transparent/explainable AI refers to techniques in AI where the process leading to an output can be understood by humans. It stands in contrast to the “black box” implementations, where one cannot explain why the model arrives at a specific decision. Simply put:
The idea of building models that can be understood in human terms is almost as old as artificial intelligence itself. Mycin, a research prototype that could explain which of its hand-coded rules contributed to a diagnosis in a specific case, dates back to the 1970s. This AI-based expert system was engineered at Stanford University to identify bacteria causing certain infections and recommend antibiotics in a dosage adjusted to the individual’s weight. Mycin operated on a knowledge base of around 600 rules, and the power of this early prototype was fairly limited in comparison to the high predictive power modern deep learning systems exhibit in diagnostic contexts. So what approaches have been pursued since then?
A classic example of biologically inspired AI is the convolutional neural network (CNN). The features of CNNs are modeled off the basic organization of our visual system. In 1962, Torsten Wiesel and David Hubel made the discovery that our primary visual cortex is made up of two types of neurons. Simple (S-) cells can be thought of as edge-detectors at a specific retinal location. If the edge appears at a different location in your field of view, they will not respond. Complex (C-) cells have more spatial invariance, meaning they will respond to edges of their preferred orientation within a large receptive field.
