Fight AI’s bias without personal info
Bias in AI is a big problem. In particular, AI can compound the effects of existing societal biases: in a recruiting tool, if more men than women are software engineers, AI is likely to use that data to identify job applicants and overscreen for men, creating a vicious circle of bias. Indeed, Amazon recently scrapped its AI recruiting engine project for that reason. Now that AI is increasingly used in high-impact applications, such as criminal justice, hiring, and healthcare, characterizing and mitigating biases is more urgent than ever—we must prevent AI from further disadvantaging already-disadvantaged people.
To what extent can AI be used to address its own problems? Most methodologies that have been proposed to mitigate biases in AI and machine learning systems assume access to sensitive demographic attributes. However, in practice, this information is often unavailable and, in some contexts, may even be illegal to use. How can we mitigate biases if we do not have access to sensitive attributes?
What if we could use fire to fight fire, or in this case, use bias to fight bias?
That’s the approach we came up with for our paper, “What’s in a Name? Reducing Bias in Bios Without Access to Protected Attributes“, to be presented at the 2019 North American Chapter of the Association for Computational Linguistics (NAACL) conference running June 2-7, 2019, in Minneapolis. We’re also pleased to announce that our paper won the Best Thematic Paper award.
In our paper, we propose a method that relies on word embeddings of names to reduce biases without requiring access to sensitive attributes. Our method even tackles intersectional biases, such as biases involving combinations of race and gender. As we showed in previous work, inherent societal biases involving sensitive attributes are encoded in word embeddings. Essentially, word embeddings are mappings of words to vectors, learned from large collections of documents, so they capture any biases represented in those documents. For example, in commonly used word embeddings, Hispanic names are “embedded” closer to ‘taco’ than to ‘hummus,’ as well as reflecting harmful cultural stereotypes.
Specifically, we look at mitigating biases in occupation classification, using names from a large-scale dataset of online biographies: the predicted probability of an individual’s occupation should not depend on their name—nor on any sensitive attributes that may be inferred from it. Crucially, and in contrast to previous work, our method requires access to names only at training time and not at deployment time.
