We have gobs of data, nearly limitless cloud compute, and ever-improving machine learning algorithms, so what on earth is holding companies back from succeeding with Big Data? “Talent, talent, talent,” says Dr. Kirk Borne. “The limiting factor is talent.”
To be sure, Borne has done more than most when it comes to fostering data science talent. Fourteen years ago, before his recent stint at Booz Allen Hamilton or his new gig at DataPrime, Borne helped create the nation’s first data science degree program at George Mason University.
That proved to be a pivotal point for data science in academia, and today, there are thousands of Bachelors, Masters, and PhD-level data science programs all over the country, not to mention an untold number of bootcamps and certificate programs.
With all that effort put into minting new data scientists, the world should be awash with the unicorns now. Yet the data science gap still persists. According to Borne, it all comes down to insatiable demand.
“There’s almost exponential growth in the talent pool,” Borne tells Datanami. “But unfortunately, so to speak, for the business world, the number of job opportunities that businesses are creating is also exponentially growing, but at a faster pace than the talent production. The difference between two exponentials is effectively still an exponential, and so there’s still this rapidly growing talent gap.”
We’ve come a long way from the early days of Big Data, Borne says.
“The technologies we’re using today are not what we used eight years ago,” he says. “Remember, Hadoop was all the rage, and everyone had to learn Hadoop and hardly no one even mentions that word in a sentence anymore.” (Well, almost no one!)
The Hadoop experiment was painful for some, to be sure. It’s not exactly clear whether we had to go through it (a good case can be made that we did). In any case, the important thing now is that big data technology is much better and more usable today than it was 10 years ago or even five years ago, and that’s a huge benefit to organizations that want to work with big data.
“[Technology] is not a hindrance anymore. It’s the enabler,” Borne says. “What’s happening now is we’re in this platform revolution phase, where with cloud, you can basically have almost infinite scalability. You don’t have to buy your own supercomputer–you just rent it for minutes or hours or days you need it, and then you give it back.”
Today, organizations have a vast array of compelling big data tools and AI technologies to choose from, most of which is running in the cloud. The three bigs–AWS, Microsoft Azure, and Google Cloud–not to mention upstarts like Snowflake and Databricks and the hundreds of other companies in this vibrant ecosystem, are all participating in the rise of a “function as a service,” which has dramatically opened up access to big data tech.
“For example, you need to build a recommender engine, or you need a chat bot?” Borne says. “You just basically call this function that someone else has already built. Like, why build it yourself?”
The accumulation of pre-built functions and ready-to-use data science platforms on the cloud is opening up all kinds of new business opportunities. Two kids in a garage spinning up huge jobs to crunch data with SQL analytics, or train a machine learning model with the freshest data, can now control it by API call from a single console. They’re now competing with multi-billion-dollar multinationals. It’s lowered the bar–and raised the stakes for everybody.
“The platform revolution has enabled plug and play of all kinds of different applications and tools and services, Borne says. “You just put them together the right way to serve a business community and you’re off to the races, basically.
Alas, amid the compelling riches of cloud-based big data tech, the limiting factor is that persistent talent gap.
Signs of the talent gap show up all over the place.
