Your Data Are Probably Biased And That’s Becoming A Massive Problem

Nobody sets out to be biased, but it’s harder to avoid than you would think. Wikipedia lists over 100 documented biases from authority bias and confirmation bias to the Semmelweis effect, we have an enormous tendency to let things other than the facts to affect our judgments. We all, as much as we hate to admit it, are vulnerable.

Machines, even virtual ones, have biases too. They are designed, necessarily, to favor some kinds of data over others. Unfortunately, we rarely question the judgments of mathematical models and, in many cases, their biases can pervade and distort operational reality, creating unintended consequences that are hard to undo.

Yet the biggest problem with data bias is that we are mostly unaware of it, because we assume that data and analytics are objective. That’s almost never the case. Our machines are, for better or worse, extensions of ourselves and inherit our subjective judgments. As data and analytics increasingly become a core component of our decision making, we need to be far more careful.

Imagine you’re running a business that hires 100 people a year and you want to build a predictive model that would tell you what colleges you should focus our recruiting efforts on. A seemingly reasonable approach would be to examine where you’ve recruited people in the past and how they performed. Then you could focus recruiting from the best performing schools.

On the surface, that would seem to make sense, but if you take a closer look it is inherently flawed. First of all, 100 students spread across perhaps a dozen colleges is far from statistically significant. Second. It’s not hard to see how a one or two standouts or dullards from a particular school would skew the results massively.

A related problem is what statisticians call overfitting, which basically means that because there is an element of bias in every data set, the more specifically we tailor a predictive model to the past the less likely it is to reflect the future. In other words, the more detailed you make your model to fit the data, the worse the predictions are likely to get.

That may seem counterintuitive, and it is, which is why overfitting is so common. People who sell predictive software love to be able to say things like, “our model has been proven to be 99.8% accurate,” even if that is often an indication that their product is actually less reliable than one that is, say, 80% accurate, but far simpler and more robust.

With humans, we are careful to construct learning environments thoughtfully. We design curriculums, carefully selecting materials, instructors and students to try and get the right mix of information and social dynamics. We go to all this trouble because we understand that the environment we create greatly influences the learning experience.

Machines also have a learning environment called a “corpus.” If, for example, you want to teach an algorithm to recognize cats, you expose it to thousands of pictures of cats. Eventually, it figures out how to tell the difference between, say, a cat and a dog. Much like with human beings, it is through learning from these experiences that algorithms become useful.