The Best Data Scientists Are Failures

There are some inbuilt human traits that are huge faults.

Think about anxiety, it does considerably more harm than good, turning the strong into a quivering mess, making logical people irrational, and even increasing the chances of having a heart attack. One of the biggest single causes of anxiety is the potential for failure. It could have been an important algorithm that has a flaw, an analysis that may have used inaccurate data, or even that you left the front door open.

However, it is this inherent anxiety and fear of failure that holds people back, stifles innovation and ultimately costs companies money. This is the case in data science departments more than almost any other part of the business.

When we think about one of the most basic forms of data science - the A/B test - it is the epitome of why the drive to succeed and the fear of failure draws people to make poor decisions. In an A/B test there are two changes that could make a difference and the one that works the best is the one that is used or moved forward in the process. What often happens is that the ‘winner’ becomes the way things are done because it ‘won’ the A/B test. However an A/B test does not show you the best option, it gives you the least bad option. Unless you then continue to test the infinite other variables, you will never know what the ‘best’ option actually is.

This requires a huge amount of failure. You have to come up with the strangest ideas to test, completely new ways of thinking and look for seemingly unconnected correlations. This means that in order to succeed it is imperative to fail considerably more than you succeed, because it is impossible to know if you have the best solution. If you think you do, you then need to constantly test against it to try and prove yourself right.

Thomas Edison, one of the most famous scientists of all time famously said ‘I have not failed. I’ve just found 10,000 ways that won’t work’ and this is the approach that all data scientists need to take.