This article was written by John Hammink. John Hammink is an American engineer, musician, artist and linguist, with his own entry in Wikipedia.
With the explosion of “Big Data” over the last few years, the need for people who know how to build and manage data-pipelines has grown. Unfortunately, supply has not kept up with demand and there seems to be a shortage of engineers focused on the ingestion and management of data at scale. Part of the problem is the lack of education focused on this growing field.
Currently, there seems to be no official curriculum or certification to become a Data Engineer or Data Architect (that we know of). While that’s bad news for companies that need qualified engineers, that’s great news for you! Why? Because it’s basic supply and demand and data engineers and data architects are cleaning up! In fact, we did a little research and found that the average salary for a data engineer is around $95.5k. While the salaries for data architects average around $112k nationally, the main path to this strategic, coveted position (and salary) means cutting your teeth as a data engineer and working your way up or making a lateral job move.
What does it take to start getting a piece of this action? Don’t worry: we did research on that, too! We did a survey of a pool of ‘Data Engineer’ and ‘Data Architect’ job ads on LinkedIn and GlassDoor and ranked the top 20 sought after job skills. They are:
Percentage of times these skills showed up in data-related job descriptions
Keep in mind that most job ads (and recruiters) are behind the curve on invoking new technology, because recruiters and HR typically get their information second hand. Plus, cutting-edge technology typically enjoys a period of minimal adoption while it is being tried out.
If you’re starting from the beginning, and set an end goal to become a data architect, the first step is to learn the skills of the data engineer. While this article won’t – and can’t – completely connect the dots for you, our aim is to get you thinking about this actively and starting to look for yourself in the right direction.
So what are the roles in a data organization?
Data Engineers are the worker bees; they are the ones actually implementing the plan and working with the technology.
Managers (both Development and Project): Development managers may or may not do some of the technical work, but they help to manage the engineers. Project managers help handle the logistical details and time-lines to keep the project moving according to plan.
Data Architects are the visionaries. They lead the innovation and technical strategy of the product and architecture. Highly experienced and technical, they grow from an engineer position. Very experienced and valuable, they are rare ducks since they’ve essentially been working in this field since its beginning.
