Testing for bias in your AI software: Why it’s needed, how to do it

When it comes to artificial intelligence (AI) and machine learning (ML) in testing, much of the interest and innovation today revolves around the concept of using these technologies to improve and accelerate the practice of testing. The more interesting problem lies in how you should go about testing the AI/ML applications themselves. In particular, how can you tell whether or not a response is correct?

Part of the answer involves new ways to look at functional testing, but testers face an even bigger problem: cognitive Bias, the possibility that an application returns an incorrect or non-optimal result because of systematic inflection in processing that produces results that are inconsistent with reality.

This is very different from a bug, which you can define as an identifiable and measurable error in a process or result. A bug can typically be fixed with a code change. Bias can be much more insidious and harder to test.

Here's why you need to test for bias in AI, and how to best go about it. 

Bias comes from your data. AI systems are trained with data collected from the problem domain. Even if the data is scientific and objective, it can still be subject to bias.

For example, in 2016 Amazon trained an AI bot to crawl the web to find candidates for IT jobs. To train this bot, the company used the resumes of its existing staff, which was overwhelmingly male. Not surprisingly, what the application "learned" was that males make the best IT employees. Amazon was never able to fix that bias, and withdrew the bot from use.

We’ve also seen examples of commercial facial-recognition systems that badly misclassify dark-skinned subjects, in large part because they are trained overwhelmingly with light-skinned images. People have been arrested and held based on faulty AI-based identifications that police accept without question. Similarly, loan recommendation systems might be trained with data that creates a bias in the system against people living in lower-income or minority neighborhoods.

Identifying bias in an application can be hazardous to your career. Last December, Google fired AI ethics researcher Timnit Gebru, who was known for finding bias in facial analysis systems. While the circumstances of this dismissal remain in question, teams that develop these systems are often faced with conflicting data that can have an impact on the quality of the system.

Bias can also expose an organization to negative publicity, such as when beauty pageant contestants were judged by an algorithm that selected almost all white women as winners. Companies, governments, and other groups that allow bias into their decisions will be perceived as untrustworthy and lose credibility and quite possibly business.

There are three main categories of bias. 

Here, concepts become incorrectly correlated. For example, you may find a correlation between education and income, but that correlation may in fact be spurious. Instead, intelligence, a different measure, may be the actual causative variable.

Here your team doesn't have good data, or the data you have doesn't represent the problem domain well. If that data doesn't accurately and completely represent the problem domain, you are likely to experience bias in your results. 

Alternatively, the problem domain may have subtly changed over time to the point where your historical data no longer represents it.

This happens when an application can become biased through interaction with other biased systems. This can be the case with unsupervised learning systems, where the application continues to learn while being used.