The Three Pillars of Trusted AI
As AI becomes ubiquitous across dozens of industries, the initial hype of new technology is beginning to be replaced by the challenge of building trustworthy AI systems. We’ve all heard the headlines: Amazon’s AI hiring scandal, IBM Watson’s $62 million failure in oncology, the now-infamous COMPAS recidivism model that discriminated against Black defendants. AI failures are becoming commonplace among large organizations, and they draw justified scrutiny and ire from the public, media, and regulators alike.
AI can radically transform an organization, but just like in human decision making, there are many ways that an AI system can go wrong – inaccuracy, overconfidence, bias, privacy concerns, and dozens of other risks can be encoded into an organization’s AI. And because a single point of failure can have massive repercussions when it comes to automated decision making, one-off solutions and tools don’t solve the broader problem of AI trust. In order for an organization to be able to trust their AI models, they have to approach the problem of trust from a holistic perspective – understanding the high-level picture of how AI can fail across every stage of its development, from data preprocessing to model building and deployment.
Ultimately, there are three main pillars of trusted AI that are necessary to successfully implement trustworthy enterprise AI:
Performance relates to the question: “How well can my model use data to make predictions?” Model accuracy is the most commonly discussed dimension of performance, but trusting your AI’s predictions requires much more than accuracy. Performance also includes criteria like Data Quality, your model’s robustness to dirty or missing data, and the speed with which it can make predictions.
Operations relates to the question: “How reliable is the system that my model is deployed on?” This pillar ensures that you can trust your model in the real world – where data is messy and dynamic, regulations abound, and security is always a concern. Many models that perform perfectly in a sandbox end up breaking once they’re deployed and tested with real data.
Ethics relates to the question: “Does my model align with the ethics and values of my organization?” Or, put another way: “What is the impactof my model on the world?” This is the most important requirement for trusted AI, and also the most overlooked. Ethics includes criteria like bias and fairness, the value generated by the model, and the explainability of its decisions. Ultimately, it doesn’t matter whether your model is accurate and reliable if its impact on the world and on your organization is negative.
It’s important to understand each of these pillars in detail, which I will do over a three-part series. First, let’s dive deeper into the first pillar of AI success – performance – and how it’s one key way to successfully implement trusted AI.
Performance is important throughout the entire AI lifecycle, but it is first evaluated during the data cleaning and model building phase. This is when the model is tested in a sandbox, and the goal is to build a model with the highest possible performance before deploying it out into the real world.
The main criteria tested under performance are:
Data Quality is the foundation of all trustworthy AI: As the old saying goes, “Garbage in, garbage out.” Even the most advanced machine learning model can’t make up for low-quality data.
The first way to ensure Data Quality requires you to track the data’s provenance. Many AI projects require combining data from multiple sources: in-house data warehouses, third-party data, and even open-source datasets, such as census records or even weather reports. It’s critical to understand the different data sources being used by the AI system.
