7 Steps to Mastering Data Preparation with Python

Follow these 7 steps for mastering data preparation, covering the concepts, the individual tasks, as well as different approaches to tackling the entire process from within the Python ecosystem.

Data preparation, cleaning, pre-processing, cleansing, wrangling. Whatever term you choose, they refer to a roughly related set of pre-modeling data activities in the machine learning, data mining, and data science communities.

For example, the all-knowing Wikipedia defines data cleansing as:

Data wrangling, for comparison, is defined by Wikipedia as:

While I would first point out that I am not thrilled with the term “data sink,” I would go on to say that it is “identifying incomplete, incorrect, inaccurate or irrelevant parts of the data and then replacing, modifying, or deleting the dirty or coarse data” in the context of “mapping data from one ‘raw’ form into another…” all the way up to “training a statistical model” which I like to think of data preparation as encompassing, or “everything from data sourcing right up to, but not including, model building.” That is the vague-yet-oddly-precise definition we’ll move forward with.

Keep in mind, however, that this article covers one particular set of data preparation techniques, and additional, or completely different, techniques may be used in a given circumstance, based on requirements. You should find that the prescription held herein is one which is both orthodox and general in approach.

  First, let’s stress what everyone else has already told you: it could be argued that this data preparation phase is not a preliminary step prior to a machine learning task, but actually an integral component (or even a majority) of what a typical machine learning task would encompass. For our purposes, however, we will separate the data preparation from the modeling as its own regimen.

As Python is the ecosystem, much of what we will cover will be Pandas related. For the uninitiated, Pandas is a data manipulation and analysis library, is one of the cornerstones of the Python scientific programming stack, and is a great fit for many of the tasks associated with data preparation.

Data preparation can be seen in the CRISP-DM model shown above (though it can be reasonably argued that “data understanding” falls within our definition as well). We can also equate our data preparation with the framework of the KDD Process — specifically the first 3 major steps — which are selection, preprocessing, and transformation. We can break these down into finer granularity, but at a macro level, these steps of the KDD Process encompass what data wrangling is.

While readers should be able to follow this guide with few additional resources, for those interested in a more holistic take on Pandas, helpful information can be found in the following:

For some feedback on the data preparation process from 3 insiders — Sebastian Raschka, Clare Bernard, and Joe Boutros — read this interview before moving on:

  Exploratory data analysis (EDA) is an integral aspect of any greater data analysis, data science, or machine learning project. Understanding data before working with it isn’t just a pretty good idea, it is a priority if you plan on accomplishing anything of consequence.