How automation is changing data science and machine learning
Almost any article you read about how automation will affect our future can be classified into one of two narratives. The first one is that it will definitely lead to a better future, as it always had since the industrial revolution. Of course, some people will lose their jobs, but as history shows, new jobs will be created. And not just new jobs, but better jobs. The other narrative is that this time is different. The robots are becoming more and more intelligent and capable. And the number of jobs and industries they’ll destroy will far exceed the number of jobs they create. Of course, it’s impossible to tell which of the two narratives will become a reality. What we can tell is that these narratives share similar inception: more and more parts of our jobs and lives are being automated.
Take for example the process of driving. For many years now, we have been taking small parts of the driving process and automating them. For a better driving experience, we built cruise control. For route planning, we developed the GPS. Now, we are able to tackle more complex problems like lane merging and emergency braking. In the next couple of years, we will most certainly have fully autonomous cars driving on the roads. Just last week Waymo, Alphabet’s self-driving subsidiary, officially received the very first California permit to test their vehicles in the state without a human behind the wheel.
All these developments illustrate an important lesson: automation happens in steps. And this process can be observed in many jobs. The only major variable is the timeframe at which it happens. If you’re a doctor, it’s going to take a while until major parts of your job are automated. If you’re an entry-level accountant, the timeframe is much smaller. Just as with driving, gradual automation is happening in data science as well. Although a complex topic, I’ll be focusing on two important developments that are changing the data science landscape:
Before I start talking about data science platforms, let me give you a short introduction into what data science is. In its essence, it is a field that uses tools taken from computer science, statistics, and machine learning to extract insights from data. In other words, if you have some data and you want to make some decision or predictions based on it, you use data science. However, extracting information from big data sets can be expensive. To implement any type of big-data project, a company must build a data infrastructure first. Think of it as different pieces of technology that can run all the tools a data scientist needs. The issue is that for many years building such an infrastructure was like building a car just from parts. Possible, but you needed people with highly specialized skills, and it took a lot of money and a time. Fortunately, this is changing. What we have seen in the past few years is the appearance of platforms that automate this process. Take for example various cloud-based platforms that make it much easier to develop and maintain big-data infrastructures, from my own team to others in the market like Amazon Web Services (AWS), Google, Microsoft, and Anaconda.
Automated big data platforms are only part of the story. Although they enable us to set up and maintain data infrastructures more easily, somebody still needs to write lines of code to clean the data and experiment with machine learning models. This process can be quite time consuming and needlessly complex. To understand what I mean, let me walk you through the data science workflow.
Usually, every data science project consists of three parts: data processing, modeling, and deployment. When it comes to how time is spent in each of these parts, we have this rule of thumb: 50-60 percent is spent on just processing the data, and the rest is spent on modeling and deployment. As you can imagine, this is a very inefficient use of a data scientist’s time.
