You’ve Been Warned–Bad Data Models are Capable of Destroying Companies

The field of data science has already delivered incredible results in finding business problems and creating models which generate market insights. But bad or failing models can also deliver erroneous results which can destroy business opportunities and tarnish corporate reputations. According to a recent study that polled 300 U.S. data science executives, an alarming 82% of data executives are concerned about major revenue loss or a hit to brand reputation caused by bad or failing models, highlighting a need for model risk management.

Data science leaders can’t put their heads in the sand when it comes to maintaining data science models–the stakes are too high. A survey by Accenture showed that 75% of executives believe their companies will most likely go out of business if they can’t scale data science successfully within the next five years.

If we start with the assumption that these models are here to stay, and you’d better do it right,  or face the digital graveyard, let’s break down the problem. What’s really at stake? And how can we negate the risks and maximize the benefits?

Data executives say the key dangers of unimproved models include: making wrong decisions and using incorrect KPIs; loss of productivity; security and compliance risks and discrimination and bias in an AI model. Here are some examples that should give you caution.

One of the biggest risks of any data science project is that faulty data will drive unexpected predictions. Data is based on the past. If inequalities existed in the past, it’s easy for a model to reinforce conditions like unequal pay or gender biases. Look at Amazon’s discarded recruiting model. The AI model looked at all the men getting jobs at Amazon, and deranked graduates from women’s colleges, or even someone who mentioned they were the captain of a women’s chess team.

If an AI project is using the wrong signals during machine learning, the end result will suffer. As PWC put it, “It can be too easy for people to let subtle, unconscious biases enter, which AI then automates and perpetuates. That’s why it’s so critical for the data scientists and business leads who develop and instruct AI models to test their programs to identify problems and potential bias.”

Clearly defining business goals and KPIs is a crucial early step when developing a model, and bringing data and business teams together will lead to better results. It may take sales, marketing AND data science teams working together to evolve a data model from predicting which audiences would Like a post to determining which messages will get specific market segments to purchase – the real business goal.

Before the COVID-19 pandemic, the data science team at Instacart had a very successful model for predicting product availability, reaching 93% accuracy. As a consumer, getting the products you’ve ordered is even more important than how soon the order will arrive.

When lockdown orders started, services like Instacart became essential, but hoarding toilet paper and hand sanitizer knocked down model performance to 61% accuracy. Instacart quickly retrained their model using a smaller dataset from the pandemic period, so they could reliably deliver the products customers actually wanted.

Running your data science team so you can respond to market changes and recruit the best team is super crucial.