Towards Human-Centered Explainable AI: the journey so far
“So, the machine has high accuracy and explains its decisions, but we still don’t have engagement with our users?” I asked seeking clarification on a rather perplexing situation. Aware of my prior work in Explainable AI (XAI) around rationale generation, a prominent tech company had just hired me to solve a unique problem. They invested significant resources to build an AI-powered cybersecurity system that aims to help analysts manage firewall configurations, especially “bloat” that happens when people forget to close open ports. Over time, these open ports accumulate and create security vulnerability. Not only did this system have commendable accuracy, it also tried to explain its decision via technical (or algorithmic) transparency. But, there was almost zero to no traction amongst its users. The question was—why?
“Yeah, that’s the confusing part, isn’t it? I think we just need better models…we need to build better rationales [natural language explanations] … guess that’s why we brought you in!” the team’s director chuckled as we continued the meeting.
Even though I was brought in to solve the problem, there was an underlying assumption that the solution to this AI problem was to “build better AI”. This dominant techno-centric assumption stems from a mythology around Explainable AI. For our purposes, we will call this the “algorithm-centered Explainable AI” myth. It goes something like this: If you can just open the black box, everything else will be fine.
In this article, I will challenge this myth and offer an alternative version of XAI, one that is sociotechnically informed and human-centered. I will use my prior work as well as my experience with the aforementioned cybersecurity project to share the journey to the Human-Centered XAI perspective. This human-centered stance emerges from two key observations.
First, explainability is a human factor; it is a shared meaning-making process that occurs between explainer(s) and explainee(s). For our purposes, we will adopt the overarching definition that an explanation is an answer to a why-question. Implicit in Explainable AI is the question: “explainable to whom?”
Understanding the “who” the human in the loop is crucial because it governs what the explanation requirements for a given problem. It also scopes how the data is collected, whatdata can be collected, and the most effective way of describing the whybehind an action. For instance: with self-driving cars, the engineer may have different requirements of explainability than the rider in that car.
Second, AI systems do not exist in a vacuum– they are situated and operate in a rich tapestry of socio-organizational relationships. Given the direct impact AI systems have on human lives, before we even start deploying anything, don’t you think we need to know more about our stakeholders whose lives will be impacted by our systems?
The cost of failure is high—if we fail to understand who the humans are, it’s inconceivable howwe will effectively design meaningful systems.
In fact, as we move from an algorithm-centered perspective to a human-centered one, as we move from AI to XAI and re-center our focus on the human—through Human-centered XAI (HCXAI)—the need to refine our understanding of the “who” increases.
Considering the socially-situated nature of AI systems, our current AI problems are no longer purely technical ones; they are inherently sociotechnical. How can we expect sociotechnical problems to be solved by purely technical means? An algorithm-centered approach can only get us a part of the way. The human-centered approach needs to be sociotechnically informed to address the current challenges. But how might we do that?
My own journey towards a sociotechnically-informed Human-centered XAI has been anything but self-evident. It has been years of work, each “turn” building on the previous one through self-reflection, critique, and refinement.
