The flood of data-related roles unleashed in recent years has the potential to cause untold confusion, particularly for senior executives inexperienced in the field. It is, however, necessary that organizations looking to join the upper echelons of data maturity stop trying to hire some 'renaissance data scientist' to collect, clean, and analyze the data all by themselves. It is tremendously rare to find any one person who embodies all the qualities companies are looking for, with demand for such so-called unicorns currently far outstripping supply. Organizations should be looking to employ a full complement of capabilities to manage the entire process, with specialists to collect data and business users empowered to translate it into insights that can improve the bottom line.
One data role growing in importance is the data engineer, which again made LinkedIn's list of most promising jobs in 2018. With a median base salary of $107,000 and a 35% year-on-year increase in the number of job openings to over 1,400, it's not hard to see why it's become such an inviting career path. More than this though, it's also no longer perceived to be a water carrier role, the Watson to the Data Scientist's Sherlock, doing the grunt work while they sweep up the accolades. Data engineers are critical to a successful data journey, and companies are increasingly realizing they need one in place as early as possible.
The definition of what it is exactly a data engineer may vary from company to company, but the core remains essentially the same. They are there to optimize the performance of their Organization’s big data ecosystem to put data scientists in the best possible position to do modeling. They prepare an infrastructure capable of ensuring the pipeline is maintained by designing, building, and integrating the tools, infrastructure, frameworks, and services needed to properly collect and ingest batch and stream-oriented data. They clean the raw data, which will invariably contain errors, and check it for unformatted and system-specific codes to make it usable. They may also run ETL (Extract, Transform, and Load) on top of big datasets, although they are not typically expected to know any machine learning or analytics for big data. They are also there to act as advocates for new and better data, and to develop a holistic approach for the rest of the Organization that ensures data assets are protected and accessible.
