How Hard Is It to Be a Real Data Scientist?
How many mathematicians study Linguistics? How many mathematicians study Healthcare? So why are we any good at solving problems in these fields?
The art of being a Mathematician comes from the ability to abstract a problem in a manner that makes it solvable. In Linguistics, we can treat each “phone” as a discrete variable and create a model that determines the joint distribution between each phone. In Healthcare, we can build a model that picks up latent features in X-rays that discern a disease.
To be a successful machine learning researcher you have to really be willing to put the time and effort into fully immersing yourself in the domain knowledge. Many of the successful game-changers in the field have broken ground in fields that they had experience in. Deepminds founder Demis Hassabis ran a games company before returning to UCL to study Neuroscience, ultimately leading to his developments in Reinforcement Learning and leading to his advances in games like Atari and Go.
Not all of us are as fortunate as Demis in having a background in a field that we’re trying to revolutionise. Often we’ll be at work and a project comes up that we have to try figure out: and next week we may have another task. Project switching has its pro’s and con’s, but ultimately you suffer on the level of depth you go to.
It definitely helps if you know a little bit about your niche before you apply some ML, but for what it’s worth, I sympathise with your struggles.
Noise is second to none in statistics, machine learning and data science. Honestly, it’s everywhere. From dirty data, to rogue data points, to literature built on weak foundations, to models capturing latent bias: noise is literally everywhere.
Machine Learning models generally perform by minimising the squared sum of errors (or some form of misclassification measure) but when you’re researching a new topic or getting feedback from a colleague, noise can be pretty hard to define — the last thing you want to do is be chasing down the rabbit hole.
There are a few ways to get around it:
Experiment more, speak to people more, try more things and eventually you’ll begin to recognise and ‘smell’ noise. You’ll avert it, and progress quicker.
As an example: many algorithms have a high accuracy rating because the dependant variable happens so infrequently. E.g. a model which predicts how many people in London get struck by lightening on a daily basis will almost certainly be 99.9999% correct without any training.
