Data Engineer A Perfect Combination Of Data Analyst And Data Scientist
Data engineer may be a comparatively new position that is a hybrid of types between a data analyst and a data scientist. Whereas information scientists are reception making and standardization refined machine learning models and alternative kinds of analysis, data engineers shine at manipulating huge amounts of data and guaranteeing the complete huge data code stack will scale to support large workloads.
“A data engineer is the all-purpose everyman of a big data analytics operation, working between downstream analysts on the one hand, and upstream data scientists on the other. They will often come from programming backgrounds and are experts in big data frameworks, such as Hadoop. They’re called on to ensure that data pipelines are scalable, repeatable, and secure, and can serve multiple constituents in the enterprise.”
Trifecta co-founder and CTO Sean Kandel noted an uptick in demand for data engineers two years ago. “I’m definitely seeing that title pop up more, and seeing more postings for it,” he told us back then.
Now, that little surge in demand seems to be blossoming into a full-blown data engineering shortage. According to a new report released by Stitch and Galvanize, there are only 6,500 self-reported data engineers across the whole country according to an analysis of their LinkedIn profiles, but more than 6,600 job openings for data engineers in the San Francisco Bay Area alone.
According to the study, the number of data engineers has doubled from 2013 to 2015, with the biggest concentration of data engineers in the information technology and services industries. The top five skills needed by data engineers are SQL, Java, Python, Hadoop, and Linux.
Stitch’s analysis (conducted in SQL, Python, and Jupyter, naturally) found that while there is currently twice the number of data scientists as data engineers, the number of data engineers is growing much faster than any other position.
That gap spells trouble for digital companies hoping to hire big data engineering talent, including high-flying Silicon Valley firms like Uber, Airbnb, and Spotify. But it smells like an opportunity for folks like Galvanize CEO Jim Deters, who says his company is working to satisfy both sides of the house through boot camp style programs for both data scientists and data engineers.
“There’s vast demand on each side,” Deters tells Datanami during a recent interview. “For those that are simply getting down to dabble their toes into understanding they seem to be a computer code company and a data company, they could not be as refined with wherever they are going. they are graduating from totally different business intelligence tools and using R.
