Life-changing decisions are happening in the dark. Machine-learning algorithms now determine decisions from loan applications to cancer diagnoses. In France, they place children in schools. In the US, they determine prison sentences. They can set credit scores and insurance rates, and decide the fate of job candidates and university applicants.
But these programs are often unaccountable. To arrive at their decisions, machine-learning algorithms automatically build complex models based on big data sets, so that even the people using them may not be able to explain why or how a particular conclusion is reached. They’re a black box.
The AI researcher Sandra Wachter is working to drag them into the light. The 32-year-old is a research fellow at both the Oxford Internet Institute and the Alan Turing Institute in London. She trained as a lawyer in her native Austria and attributes her interest in technology to her grandmother, one of the first three women admitted to a technical university in the country. “She shaped my thinking about technology as something that was powerful and interesting, and could be used for good,” she says.
Now Wachter works on the legal and ethical implications of AI, machine learning and robotics. She acts as a link between the makers of technology and the judges and policymakers who will create a legal framework for it. Her work is about “striking a fair balance”, she says, and figuring out how we can reap the benefits of technology without jeopardising our human rights and privacy.
Wachter believes we should have the legal right to know why algorithms come to specific decisions about us. But there’s a clash, as software owners often claim that increasing transparency could risk revealing their intellectual property, or help individuals find loopholes to game the system. Wachter, along with her colleagues Brent Mittelstadt and Chris Russell, has come up with a compromise: counterfactual explanations.
Counterfactuals are statements of how the world would need to be different in order for a different outcome to occur: if you earned £10,000 a year more, you would have got the mortgage; if you had a slightly better degree, you would have got the job.
“If I don’t get a loan, I don’t necessarily care so much how the algorithm works. I actually just want to know why I didn’t get the loan and have some guidance on how to improve,” Wachter says. Counterfactual explanations can provide this by finding the smallest possible change that would have led to the model predicting a different outcome.
Crucially, counterfactual explanations give answers about why a decision was made without revealing the guts of the algorithm. The black box stays sealed.
