Are data scientists going out of style?
“We are looking for a mixture between a statistician, scientist, machine learning expert and engineer: …The ideal candidate understands human behavior and knows what to look for in the data.”
That is a real excerpt from a recent job listing looking for a data scientist. From startups to Fortune 500 companies, business are on the hunt for that rare individual that likely has a Ph.D. in mathematics, is versed enough in Python or R to write production-ready code, has an intuitive grasp on decoding their coworkers’ behavior, and likely also needs to have experience in whatever industry the business is in.
Sound like a tall order? Perhaps it’s these lofty expectations that have been fueling the corporate hiring reality that data scientists are about as hard to find as unicorns.
In recent years, the position of data scientist has become one of the most desired and sought after positions for enterprises, but finding someone with this skill set is a challenge. The number of open positions is ballooning. In fact, Forbes predictsthat by 2020 the number of job listings for data science and analytics jobs will continue to grow from roughly 364,000 now to 2.72 million, with most of these positions concentrated in finance and insurance, professional services, and IT.
But actually finding these people often proves tough. Multiple surveys over the past few years have suggested there is a data scientist shortage. These job positions stay open an average of five days longer than the typical market average of 45. When companies do land one of these needle-in-the-haystack hires and they prove their merit, they are often poached by bigger companies that can offer even bigger salaries. This has driven the average tenure of these positions down from 2.5 to two years.
This leaves businesses in a bit of a conundrum. They supposedly need data scientists, who are being presented as the greatest thing since sliced bread, but finding and keeping these employees is a tenuous mission at best. But there could be another way out of this situation: What if organizations don’t need data scientists as much as they think they do?
It may be hard to imagine, but thinking through the details makes it clear that the current mantra that businesses must have a data scientist needs retooling. What if instead of having a single person at the helm of these many — and disparate — duties is mandatory, there are actually other options that could help organizations properly pull off their data science goals without latching them to hiring one perfect data scientist?
The following are a few areas enterprises should consider to rethink the role of the data scientist today.
In the rush to not miss out on the big data hype of the early 2010s, companies stood up a lot of different tools that were supposed to help make sense out of the endless onslaught of data points they were collecting. The outcome of this for many companies was a messy architecture that needed to be rejiggered and jury-rigged to manage whatever was the most pressing analytics issue of the moment.
Unfortunately, this means that the proper infrastructure isn’t in place for today’s data scientists. Thus, these professionals end up spending a lot of their time performing maintenance instead of doing data science. They end up fixing this infrastructure and its accompanying engineering problems, a task below their pay grade, and are forced to focus on data quality and data management — definitely not as sexy as that job description made things sound.
The reality is, finding an alternative is necessary for many businesses to succeed. Organizations, strapped to find someone that can handle the messy data infrastructure they’ve amassed, can’t expect that person to want to stick around, performing work that’s menial versus the original task set out before them.
