Are All Explainable Models Trustworthy?

Explainable AI or Explainable Data Science is one of the top buzzwords of Data Science at the moment. Models that are explainable are seen…

Explainable AI or Explainable Data Science is one of the top buzzwords of Data Science at the moment. Models that are explainable are seen as the answer to many of recently recognized problems with machine learning, such as bias or data leaks.

A frequently given reason to make models more explainable is that they will then be trusted more readily by users, and sometimes it appears people assume the ideas are almost synonymous. For example, the paper introducing the influential LIME method of explaining black-box models was titled ‘Why Should I Trust You?’, as if having an explanation for how a model came to its decision was a short direct step away from trusting it. Is this really the case, however?

An immediate problem with this equivalence is that trust is given at an emotional level whereas an explanation is a more technical artifact—the assumption behind explaining a model is that there are a certain number of pieces of information which can be provided to ensure the user understands what the model is doing. In contrast, to gain trust means crossing a number of emotional thresholds.

Hence, while it is true that an overly opaque model can be a huge obstacle to gaining a user’s trust, it isn’t the whole story—and there may even be occasions that an opaque model is trustworthy if some other conditions are met.

Firstly, let’s revisit some ways that a model explanation can be useful.

Hence, there are reasons to make a model explainable that don’t immediately correspond to winning trust, although they may be a path to trust themselves. For example, incorporating a local subject matter expert’s opinion into a model may help them to trust it. At the same time, the goals listed above are clearly ends in themselves.

At the core, explainability and trust are different because trust is an emotional issue compared to the fact-based issue of explainability, which simply means that you can identify the effect that individual predictors have had on the model’s output. Because they correspond to different sets of problems, they require different approaches to solve them.

A first step for someone to trust a model will often be to begin to trust the person or organization presenting the model. That, in turn, will mean trying to understand not what the model is doing, but what that person or organization is doing—what do they hope to achieve by implementing this model?

In the ‘Trust Equation’ developed by Maister, Green, and Galfond and promoted in their book ‘The Trusted Advisor’ this corresponds to the self-orientation denominator that can derail the other positive factors.