Choosing a Machine Learning Model
The part art, part science of picking the perfect machine learning model.
The number of shiny models out there can be overwhelming, which means a lot of times people fall back on a few they trust the most and use them on all new problems. This can lead to sub-optimal results.
Today we’re going to learn how to quickly and efficiently narrow down the space of available models to find those that are most likely to perform best on your problem type. We’ll also see how we can keep track of our models’ performances using Weights and Biases and compare them.
You can find the accompanying code here.
Unlike Lord of the Rings, in machine learning there is no one ring (model) to rule them all. Different classes of models are good at modeling the underlying patterns of different types of datasets. For instance, decision trees work well in cases where your data has a complex shape:
Whereas linear models work best where the dataset is linearly separable:
Before we begin, let’s dive a little deeper into the disparity between model selection in the real world vs for competitive data science.
As William Vorhies said in his blog post “The Kaggle competitions are like formula racing for data science. Winners edge out competitors at the fourth decimal place and like Formula 1 race cars, not many of us would mistake them for daily drivers. The amount of time devoted and the sometimes extreme techniques wouldn’t be appropriate in a data science production environment.”
Kaggle models are indeed like racing cars, they’re not built for everyday use. Real-world production models are more like a Lexus — reliable but not flashy.
Kaggle competitions and the real world optimize for very different things, with some key differences being:
The real world allows you to define your problem and choose the metric that encapsulates the success of your model. This allows you to optimize for a more complex utility function than just a singular metric, where Kaggle competitions come with a single pre-defined metric and don’t let you define the problem efficiently.
In the real world, we care about inference and training speeds, resource and deployment constraints and other performance metrics, whereas in Kaggle competitions the only thing we care about is the one evaluation metric. Imagine we have a model with 0.98 accuracy that is very resource and time intensive, and another with 0.95 accuracy that is much faster and less compute intensive. In the real world, for a lot of domains, we might prefer the 0.95 accuracy model because maybe we care more about the time to inference. In Kaggle competitions, it doesn’t matter how long it takes to train the model or how many GPUs it requires, higher accuracy is always better.
Similarly, in the real world, we prefer simpler models that are easier to explain to stakeholders, whereas in Kaggle we pay no heed to model complexity. Model interpretability is important because it allows us to take concrete actions to solve the underlying problem. For example, in the real world looking at our model and being able to see a correlation between a feature (e.g. potholes on a street), and the problem (e.g. likelihood of car accident on the street), is more helpful than increasing the prediction accuracy by 0.005%.
Finally, in Kaggle competitions, our dataset is collected and wrangled for us. Anyone who’s done data science knows that is almost never the case in real life. But being able to collect and structure our data also gives us more control over the data science process.
All this incentivizes a massive amount of time spent tuning our hyperparameters to extract the last drops of performance from our model and, at times, convoluted feature engineer methodologies. While Kaggle competitions are an excellent way to learn data science and feature engineering, they don’t address real world concerns like model explainability, problem definition, or deployment constraints.
