Automated Machine Learning — A Paradigm Shift That Accelerates Data Scientist Productivity

There is a growing community around creating tools that automate many machine learning tasks, as well as other tasks that are part of the machine learning workflow. The paradigm that encapsulates this idea is often referred to as automated machine learning.

At Airbnb, we are always searching for ways to improve our data science workflow. A fair amount of our data science projects involve machine learning, and many parts of this workflow are repetitive. These repetitive tasks include, but are not limited to:

Exploratory Data Analysis: Visualizing data before embarking on a modeling exercise is a crucial step in machine learning. Automating tasks such as plotting all your variables against the target variable being predicted as well as computing summary statistics can save lots of time.

Feature Transformations: There are many choices in how you can encode categorical variables, impute missing values, encode sequences and text, etc. Many of these feature transformations are canonical such that they can be reliably applied to many problems.

Algorithm Selection & Hyper-parameter Tuning: There are a dizzying number of algorithms to choose from and related hyper-parameters that can be tuned. These tasks are very amenable to automation.

Model Diagnostics: Learning curves, partial dependence plots, feature importances, ROC and other diagnostics are extremely useful to generate automatically.

  There is a growing community around creating tools that automate the tasks outlined above, as well as other tasks that are part of the machine learning workflow. The paradigm that encapsulates this idea is often referred to as automated machine learning, which I will abbreviate as “AML” for the rest of this post.

There is no universally agreed upon scope of AML, however the folks who routinely organize the AML workshop at the annual ICML conference define a reasonable scope on their website, which includes automating all of the repetitive tasks defined above.

The scope of AML is ambitious, however, is it really effective? The answer is it depends on how you use it. Our view is that it is difficult to perform wholesale replacement of a data scientist with an AML framework, because most machine learning problems require domain knowledge and human judgement to set up correctly.

Also, we have found AML tools to be most useful for regression and classification problems involving tabular datasets, however the state of this area is quickly advancing. In summary, we believe that in certain cases AML can vastly increase a data scientist’s productivity, often by an order of magnitude.

We have leveraged AML at Airbnb in the following ways:

  There is a wide array of commercial and open source tools that address the AML paradigm. We have experimented with the following tools:

  At Airbnb, we use machine learning to build customer lifetime value models (LTV) for guests and hosts. These models allow us to improve our decision making and interactions with our community at very granular levels (down to the user, level if we like).

LTV models are setup as a standard regression problem for guests, where the target variable is the spend of each guest over a time horizon.