Why Data Engineering Is Not Just About Data Science
Data engineering, as a distinct field, whose practitioners have a cohesive group identity as data engineers, is fairly new. So new, in fact, that there are many people who don’t seem to understand exactly what data engineering is and what it is not, and where the border exists between data engineering, data science and software engineering.
“When you look at some job descriptions, a lot of times you’ll see that they want a data engineer, but when you read the details, the company is actually looking for someone who specializes in machine learning, or someone who has a background in data science or someone who’s an analyst or a visualization engineer,” explained Robbie Smith, senior data engineer at Guild Education. “In a lot of job descriptions, data engineering is conflated with other data-related professions.”
Perhaps ironically, data engineering, as a profession, has more in common with software engineering than with data science. Most data engineers started as software engineers, and there’s a fairly broad overlap in skills sets used for data engineering and software engineering.
“I see data engineering as a subcategory of software engineering,” explained Luke Feeney, co-founder and chief operating officer at TerminusDB. “In most shops, we see that the data engineers are writing Python scripts to get their data from point A to point B. These are people who are coders.”
This sentiment was echoed by Smith, who described his own career trajectory as starting out as a software engineer before specializing in data engineering when offered the chance to do so at a new job.
In fact, the largest misconception about data engineering is that it is closely related to data science. The two disciplines are related, but in the same way goats are related to grass, not the way goats are related to sheep. Data engineers build the pipelines that data scientists depend on, but the two professions are very different. Whereas there is a giant overlap in skill sets between software engineering and data engineering, the skillsets and career path of a data engineer and a data scientist are quite different.
Data engineers are responsible for building a beautiful data pipeline that works every time, that has revision control and is very structured and orderly. Data scientists are trying to make sense of that data — to understand why anyone should be moving around in the first place and to use the data for business reasons.They generally have Ph.D.s in statistics and approach their work like scientists — they want to run experiments, not write code.
Andrew Stevenson, chief technology officer at lenses.io, thinks organizations should value data engineers who can do more than create sleek pipelines, but admits that what many organizations see them as. “I used to see great data engineers who best understood business requirements being muscled out in an organization because they didn’t adopt the latest open-source, bleeding-edge technologies,” he said. “Many of these big data projects failed.
