A Guide to Data Preparation with Python
The Data Preparation of any data project is arguably the most important part. If you have garbage going in, you’ll have garbage going out.
I’ll show you some Data Preparation examples for each of the following:
Prior to an analysis, you always to remove unnecessary data.
Let’s say you don’t need to know the Tax or the Branch of a transaction.
…..you can also use
Now you can confirm that df columns are removed by calling df.columns
You can see that “Branch” and “Tax 5%” columns are now removed from the data frame.
Sometimes the number of columns you need to drop outnumber the number of columns you need.
You can create a new data frame object containing the specific columns you need.
Here’s are the columns of the new data frame called df_sales:
One of the quickest ways to find the number of null values is to use the .info() method. It shows you the columns, the number of columns, the data type, number of rows, and the number non-null objects.
If you want to find the number of missing values for a particular column you can use the following code.
The Melbourne Housing dataset contains 13,580 data points. In statistics, we believe that 30 random data points for 3 columns is a good sample size.
A quick solution if you have a large dataset, you can drop rows containing null values.
If you have a small dataset, you can impute values using Panda’s fillna method which contains a few options. I would be cautious of just filling values, depending on the columns and scenario it can sometimes make sense.
One quick imputation technique is to use fillna(). Fillna() contains methods like Backfill or bfill method which uses the previous row’s value. Front fill or pad which uses a succeeding row’s value.
Let’s say you want to use the mean or a median of a column, you can specify this.
For Categorical columns, you can decide to completely remove a column if too many values are missing. Sometimes if you just need to replace null values you can use “Unknown” or another fill word.
When working duplicates you can use Panda’s .duplicated() method. Let’s look at how many duplicates are in the Melbourne Housing dataset.
If your dataset, happens to contain duplicates, you can use the following code to remove them. Because this dataset contains no duplicates, it want change anything.
