The Data Science Process
What does a day in the data science life look like? Here is a very helpful framework that is both a way to understand what data scientists do, and a cheat sheet to break down any data science problem.
At Springboard, our data students often ask us questions like “what does a Data Scientist do?”. Or “what does a day in the data science life look like?”
These questions are tricky. The answer can vary by role and company.
So we asked Raj Bandyopadhyay, Springboard’s Director of Data Science Education, if he had a better answer.
Turns out, Raj employs an incredibly helpful framework that is both a way to understand what data scientists do, and a cheat sheet to break down any data science problem.
Raj calls it “the Data Science Process”, which he outlines in detail in a short 5-day email course. Here’s a summary of his insights.
The first thing you have to do before you solve a problem is to define exactly what it is. You need to be able to translate data questions into something actionable.
You’ll often get ambiguous inputs from the people who have problems. You’ll have to develop the intuition to turn scarce inputs into actionable outputs–and to ask the questions that nobody else is asking.
Say you’re solving a problem for the VP Sales of your company. You should start by understanding their goals and the underlying why behind their data questions. Before you can start thinking of solutions, you’ll want to work with them to clearly define the problem.
A great way to do this is to ask the right questions.
You should then figure out what the sales process looks like, and who the customers are. You need as much context as possible for your numbers to become insights.
You should ask questions like the following:
In response to your questions, the VP Sales might reveal that they want to understand why certain segments of customers have bought less than expected. Their end goal might be to determine whether to continue to invest in these segments, or de-prioritize them. You’ll want to tailor your analysis to that problem, and unearth insights that can support either conclusion.
It’s important that at the end of this stage, you have all of the information and context you need to solve this problem.
Step 2: Collect the raw data needed for your problem
Once you’ve defined the problem, you’ll need data to give you the insights needed to turn the problem around with a solution. This part of the process involves thinking through what data you’ll need and finding ways to get that data, whether it’s querying internal databases, or purchasing external datasets.
You might find out that your company stores all of their sales data in a CRM or a customer relationship management software platform.You can export the CRM data in a CSV file for further analysis.
Now that you have all of the raw data, you’ll need to process it before you can do any analysis. Oftentimes, data can be quite messy, especially if it hasn’t been well-maintained. You’ll see errors that will corrupt your analysis: values set to null though they really are zero, duplicate values, and missing values. It’s up to you to go through and check your data to make sure you’ll get accurate insights.
You’ll want to check for the following common errors:
You’ll need to look through aggregates of your file rows and columns and sample some test values to see if your values make sense. If you detect something that doesn’t make sense, you’ll need to remove that data or replace it with a default value.
