Removing Demographic Data Can Make AI Discrimination Worse
A recent study suggests that denying AI decision makers access to sensitive data actually increases the risks of discriminatory outcome. That’s because the AI draws incomplete inferences from the data or partially substitutes by identifying proxies. Providing sensitive data would eliminate this problem, but it is problematic to do so in certain jurisdictions. The authors present work-arounds that may answer the problem in some countries.
Decisions about who to interview for a job, who to provide medical care to, or who to grant a loan were once made by humans, but ever more frequently are made by machine learning (ML) algorithms, with eight in 10 firms planning to invest in some form of ML in 2023 according to New Vantage. The number one focus of these investments? Driving business growth with data.
While data can come in many forms, when focused on generating business growth a firm is usually interested in individual data, which can belong to customers, employees, potential clients, or almost anyone the organization can legally gather data on. Data is fed into ML algorithms which find patterns in the data or generates predictions — these outcomes are then used to make business decisions — generally about who or what to focus business efforts on.
While investment in ML algorithms continues to grow and drive greater business efficiencies — 30% or more, according to a recent McKinsey report — the use of ML models and individual data does come with some risks, ethical ones to be specific. The World Economic Forum cites unemployment, inequality, human dependency, and security amongst its top risk of using artificial intelligence and ML, but by far the biggest ethical risk in practice is discrimination.
To be sure, unjustified discrimination by firms has always existed. Discrimination of historically disadvantaged groups has led to the formulation of several anti-discrimination laws, including the Fair Housing Act of 1968 and the Equal Credit Opportunity Act of 1974 in the United States, and the European Union Gender Directive. The lending space, in particular, has been a ground for discriminatory treatment, up to the point that discrimination in mortgage lending has been viewed as one of the most controversial civil rights topics.
Historically, in hopes of preventing discriminatory decisions, sensitive data, such as individual race, gender, and age has been excluded from important individual decisions such as loan access, college admission, and hiring. Whether sensitive data has been excluded in line with anti-discrimination laws (such as the exclusion of race and gender data from consumer non-mortgage loan applications in the United States due to the Equal Credit Opportunity Act) or a firm’s risk management practices, the end result is the same; firms rarely have access to, or use sensitive data to make decisions that impact individuals — whether they are using ML or human decision makers.
At first glance this makes sense; exclude individual sensitive data and you cannot discriminate against those groups. Consider how this works when determining who to interview for a job, first with human-based decision making. A human resources expert would remove the names and genders of applicants from resumes before analyzing candidate credentials to try to prevent discrimination in determining who to interview. Now, consider this same data exclusion practice when the decision is made with a ML algorithm; names and genders would be removed from the training data before it is fed into the ML algorithm, which would then use this data to predict some target variable, such as expected job performance, to decide who to interview.
But while this data exclusion practice has reduced discrimination in human-based decision making, it can create discrimination when applied to ML-based decision making, particularly when a significant imbalance between population groups exists.
