Automate your feature engineering

In machine learning, a feature is another word for an attribute or input, or an independent variable.
Feature engineering is a process of preparing inputs for machine learning models. The goal of feature engineering is to to improve classification accuracy by considering the limitations of the learning algorithms. Preparation involves working with raw data through various feature transformations, extractions, feature selection and handcrafting new features. This process sometimes involves bringing in additional information from other sources.
Below are a few feature engineering examples that are commonly used by practitioners:
In predictive modeling tasks, data scientists consistently report that they spend most of their time on feature engineering.
Feature engineering requires spending a lot of time with data. This includes examining descriptive statistics (such as number of levels of nominal variables, missingness, skewness of interval variables, pairwise correlations and so on) and simple graphs of input variables. The process involves thinking about structures in the data, the underlying form of the problem, and how best to expose these features to predictive modeling algorithms.
The success of this tedious human-driven process depends heavily on the domain and statistical expertise of the data scientist.
The most successful features are often the ones that are hand crafted very thoughtfully by exploiting deep knowledge of the domain considered or a good understanding of the datasets. For example, consider a housing data set like Kaggle’s competition data for the Zillow Prize.
By using the zip code and another continuous feature such as total square foot of houses, you can create a new feature for the median total square foot within each zip code. You can even create another feature for the difference between the median and the actual total square foot. This new variable can be very predictive because within each neighborhood it helps to identify the houses that are different in terms of total square feet.
The feature engineering process requires a good understanding of the size and the quality of the data in hand, the performance metric that’s being optimized, and the machine learning algorithm that is used. For example, if boosted trees are used, then transformations (that do not affect the rank order) tend to make a slight difference since only rank order matters for splits in trees.
In addition, when creating new features, you always need to guard against information leakage by using cross-validation very carefully. For example, if you use the target values in new feature creation directly or indirectly in the training set, you need to make sure that you don’t use target values in the same way for the validation set. Otherwise, your prediction accuracy would be too optimistic due to information leakage.
Although feature engineering is one of the most time-consuming processes and requires many skill sets, if done right, it’s also the most rewarding part of data science pipelines.
Good features not only boost your predictive accuracy but also give you the flexibility to train simpler models. This, in turn, enables you to understand and interpret your model better. By using powerful features, you can choose to fit a more interpretable regression model with fewer predictors as opposed to training a complex black-box model (which is hard to interpret) without sacrificing your prediction accuracy.


