IBM offers explainable AI toolkit, but it’s open to interpretation

Decades before today’s deep learning neural networks compiled imponderable layers of statistics into working machines, researchers were trying to figure out how one explains statistical findings to a human.

IBM this week offered up the latest effort in that long quest to interpret, explain, and justify machine learning, a set of open-source programming resources it calls “AI 360 Explainability.”

It remains to be seen whether yet another tool will solve the conundrum of how people can understand what is going on when artificial intelligence makes a prediction based on data.

The toolkit consists of eight different algorithms released in the course of 2018. The IBM tools are posted on Github as a Python library. The project is laid out in a blog post by IBM Fellow Aleksandra Mojsilovic.

Thursday’s announcement follows on similar efforts by IBM over the course of the past year, such as its open-source delivery in September of “bias detection” tools for machine learning work.

The motivation is clear to anyone. Machine learning is creeping into more and more areas of life, and society wants to know how such programs arrive at predictions that can influence policy and medical diagnoses and the rest.

The now-infamous negative case of misleading A.I. bears repeating. A 2015 study by Microsoft describes a  machine learning model that implied pneumonia patients in hospitals had better prognoses if they also happened to suffer from asthma. However, the above-average prognosis was actually a result of the fact that asthma sufferers received aggressive treatment in the ICU because they were higher risk, not lower. It’s a cautionary tale about how machine learning can make predictions but for the wrong reasons.

The motive is clear, then, but the path to explanations is not clear-cut. The central challenge of so-called explainable A.I., an expanding field in recent years, is deciding what the concept of explanation even means. If one makes explanations too simple, to serve, say, a non-technical user, the explanation may obscure important details about machine learning programs. But a complex, sophisticated discussion of what’s going on in a neural network may be utterly baffling to that non-technical individual.

Another issue is how to balance the need for interpretability with the need for accuracy, since the most powerful neural networks of the deep learning variety have often gained their accuracy as a consequence of becoming less scrutable upon inspection.

IBM’s own researchers have explained the enormous challenge that faces any attempts to explain or justify or interpret machine learning systems, especially when the recipients of said expressions are non-technical clients of the system.

As Michael Hind, a distinguished research staff engineer at IBM, wrote in the Association for Computing Machinery’s journal XRDS this year, it’s not entirely clear what an explanation is, even between humans. And if accuracy is what matters most, most of the time, with respect to a machine learning model, “why are we having higher demands for AI systems” than for human decision-making, he asks.

As observed by research scientist Or Biran with the Connecticut-based A.I. startup Elemental Cognition, the attempts to interpret or explain or justify machine learning have been around for decades, going back to much simpler “expert systems” of years past.