If Your Company Uses AI, It Needs an Institutional Review Board

Companies that use AI know that they need to worry about ethics, but when they start, they tend to follow the same broken three-step process: They identify ethics with “fairness,” they focus on bias, and they look to use technical tools and stakeholder outreach to mitigate their risks. Unfortunately, this sets them up for failure. When it comes to AI, focusing on fairness and bias ignores a huge swath of ethical risks; many of these ethical problems defy technical solutions. Instead of trying to reinvent the wheel, companies should look to the medical profession, and adopt internal review boards (IRBs). IRBs, which are composed of diverse team of experts, are well suited to complex ethical questions. When given jurisdiction and power, and brought in early, they’re a powerful tool that can help companies think through hard ethical problems — saving money and brand reputation in the process.

Conversations around AI and ethics may have started as a preoccupation of activists and academics, but now — prompted by the increasing frequency of headlines of biased algorithms, black box models, and privacy violations — boards, C-suites, and data and AI leaders have realized it’s an issue for which they need a strategic approach.

A solution is hiding in plain sight. Other industries have already found ways to deal with complex ethical quandaries quickly, effectively, and in a way that can be easily replicated. Instead of trying to reinvent this process, companies need to adopt and customize one of health care’s greatest inventions: the Institutional Review Board, or IRB.

Most discussions of AI ethics follow the same flawed formula, consisting of three moves, each of which is problematic from the perspective of an organization that wants to mitigate the ethical risks associated with AI.

Here’s how these conversations tend to go.

First, companies move to identify AI ethics with “fairness” in AI, or sometimes more generally, “fairness, equity, and inclusion.” This certainly resonates with the zeitgeist — the rise of BLM, the anti-racist movement, and corporate support for diversity and inclusion measures.

Second, they move from the language of fairness to the language of “bias”: biased “algorithms,” as popular media puts it, or biased “models” as engineers (more accurately) call it. The examples of (allegedly) biased models are well-know, including those from Amazon, Optum Health, and Goldman Sachs.

Finally, they look for ways to address the problem as they’ve defined it. They discuss technical tools (whether open source, or sold by Big Tech or a startup) for bias identification, which standardly compare a model’s outputs against dozens of quantitative metrics or “definitions” of fairness found in the burgeoning academic research area of machine learning (ML) ethics. They may also consider engaging stakeholders, especially those that comprise historically marginalized populations.

While some recent AI ethics discussions go beyond this, many of the most prominent don’t. And the most common set of actions that practitioners actually undertake flows from these three moves: Most companies adopt a risk-mitigation strategy that utilizes one of the aforementioned technical tools, if they’re doing anything at all.

All of this should keep the stewards of brand reputation up at night, because this process has barely scratched the surface of the ethical risks that AI introduces.  To understand why this is, let’s take each of these moves in turn.

The first move sends you off in the wrong direction, because it immediately narrows the scope. Defining “AI ethics” as “fairness in AI” is problematic for the simple reason that fairness issues are just a subset of ethical issues — you’ve just decided to ignore giant swaths of ethical risk. Most obviously there are issues relating to privacy violations (given that most of current AI is ML, which is often powered by people’s data), and unexplainable outputs/black-box algorithms. But there are more.