Combating bias and creating more inclusive AI is unlikely to succeed unless developers include those people who have been historically excluded or ignored.
Advances in automation, machine learning and artificial intelligence (AI) mean we’re on the threshold of discoveries that could change human society irreversibly – for better or worse.
From AI that sorts CVs and shortlists job applicants, to facial recognition algorithms, these technologies are bringing new efficiencies and reducing errors in both public and private sector decision making.
But, increasingly, many of us are becoming aware of the dangers of bias and discrimination inherent in these technologies. A recent MIT study measuring how the technology works on people of different races and gender found that facial recognition AI was less accurate when classifying the faces of people with darker skin.
Meanwhile, Amazon scrapped an AI recruitment tool after the company found that it wasn’t rating candidates for software developer jobs and other technical posts in a gender-neutral way.
Some of these issues and concerns have already been raised by people like Cathy O’Neil, a data scientist who wrote the bestselling Weapons of Math Destruction, and political scientist Virginia Eubanks in Automating Inequality.
But there are several reasons why AI may produce decisions that are tainted by bias or discrimination – and they have serious implications for those who develop and work with AI.
“AI ‘learns’ by referring to the data that humans ‘feed’ it. If certain groups of people are left out of the data set, the automated process won’t capture their characteristics.”
Chief among the concerns about discriminatory AI is the quality and scope of the data used to inform the automated process. AI ‘learns’ by referring to the data that humans ‘feed’ it. If certain groups of people are left out of the data set, the automated process won’t capture their characteristics.
Amazon’s now-abandoned recruitment tool is just one example of this. The data used to train the computer model to select the ‘right’ person for the job was drawn from the resumes of employees that had previously been selected for positions – most of them were men, reflecting male dominance across the tech industry.
So the system was trained to select against women and the types of educational and workplace experiences more common to women.
But these problems with data quality and inclusiveness are also exacerbated by the fact that AI is often a ‘black box’ – meaning the content and how it is used to inform decisions isn’t transparent.
Another reason why AI might produce biased decisions relates to what AI is asked to do.
Programmers and developers sometimes make mistakes or ask the wrong questions.
The University of Melbourne’s Professor Bernadette McSherry, who specialises in mental health law and criminal law, has highlighted some of the difficulties in trying to map symptoms of mental health conditions via social media posts.
