A cheat sheet to the best practices for data preparation for machine learning

These best practices are designed to help your data preparation initiatives in machine learning.

Machine learning, or ML, is growing in importance for enterprises that want to use their data to improve their customer experience, develop better products and more. But before an enterprise can make good use of machine learning technology, it needs to ensure it has good data to feed into artificial intelligence and ML models.

Data preparation involves cleaning, transforming and structuring data to make it ready for further processing and analysis. Data doesn’t typically reach enterprises in a standardized format and thus needs to be prepared for enterprise use.

Before data scientists can run machine learning models to tease out insights, they’re first going to need to transform the data—reformatting it or perhaps correcting it—so it’s in a consistent format that serves their needs. In fact, as much as 80% of a data scientist’s time is spent on data preparation. Given how costly it can be to recruit and retrain data science talent, this is an indication of just how important data preparation is to data science.

ML models will always require specific data formats in order to function properly. Data preparation can fix missing or incomplete information, ensuring the models can be applied to good data.

Some of the data an enterprise collects in its data lake or elsewhere  is structured—like customer names, addresses and product preferences—while most is almost certainly unstructured—like geo-spatial, product reviews, mobile activity and tweet data. Either way, this raw data is effectively useless to the company’s data science team until it’s formatted in standardized, consistent ways.

Talend, a company that provides tools to help enterprises manage data integrity, has suggested a few key benefits of data preparation, which include the ability to fix errors quickly by “catch[ing] errors before processing” and the reduction of data management costs that can balloon when you try to apply bad data to otherwise good machine learning (ML) models.

For a broad overview, you can check out these top five tips for data preparation; these more general tips mostly apply to ML data preparation as well. However, there are some particular nuances for ML data preparation that are worth exploring.

You likely know in advance what you want your ML model to predict, so it pays to prepare accordingly. If you have a good sense of the outcome you’re hoping to achieve, you can better define the kinds of data you’ll want to collect and how you want to clean it up.

This also allows you to better respond to missing or incomplete data. A common approach to missing data is null value replacement. For example, if you’re an airline with passenger data, you might elect to drop a null value into the field that tracks meal preferences.

But depending on your application, null value replacement might be a terrible approach. From our previous example, the airline shouldn’t insert a null value for missing passenger nationality data, as this could create serious problems with their travel experience. Knowing which data is critical and how you’ll deal with incomplete records is essential.