Why Is Data Science Different than Software Development? It Starts with Data…Lots o’ DATA!!
Data science development is very different from software development, and getting the two to mesh is sometimes like trying to cobble together Tinker Toys with Lincoln Logs. One data science expert once described the differences as:
Software development is “Measure twice; cut once,” while Data Science is “Cut, cut, cut!”
The methodologies and processes that support successful software development do not work for data science projects according to one simple observation: software development knows, with 100% assurance, the expected outcomes, while data science – through data exploration and hypothesis testing, failing and learning – discoversthose outcomes.
First introduced in the blog “What’s The Difference Between BI Analyst and Data Scientist?”, the Data Science Engagement methodology in Figure 1 supports the rapid exploration, rapid testing, and continuous learning Data Science “Scientific Method[1]”.
As highlighted in Figure 1, the Data Science Development differs from Software Development in the following areas:
Let’s review each of these in more detail.
Big Data is the fuel for data science, as the more granular, diverse and complete the data sets, the more accurate the Artificial Intelligence, Deep Learning and Machine Learning predictive models.
Gartner analyst Doug Laney coined the “3 V’s of Big Data” concept in a 2001. The 3 V’s (volume, variety and velocity) are the defining dimensions of big data where volume refers to the amount of data, variety refers to the number of types of data, and velocity refers to the speed of data processing. And the continued growth of big data grows unabated, courtesy of the advent of wearables (Apple Watch, FitBit), Internet of Things (sensors, connected devices), photos, images, video, audio and voice.
But it’s not the volume of data that is valuable; it’s the granularity of the data. The value in Big Data resides in the insights, behaviors, tendencies, inclinations, associations and relationships at the individual level, whether individual humans (student, patient, doctor, technician, engineer) or individual devices (turbine, compressor, chiller, pump, switch, motor). Whether human Analytic or Behavioral Models, or device Digital Twins, the monetization value and actionability of Big Data is found at the level of the individual entity.
Step 1of the Data Scientist Development Methodology (Figure 1) starts with the Data Scientist identifying the prediction or hypothesis that they want to test. This is a result of collaborating with the business stakeholders to understand the key sources of business differentiation and then brainstorming data and variables that might yield better predictors of performance.
The Data Science Development methodology requires significant collaboration with the business stakeholders throughout the development process to:
To facilitate the business stakeholder-data science collaboration, we have created the Hypothesis Development Canvas – a template that we will populate prior to executing a data science engagement to ensure that we thoroughly understand what we are trying to accomplish, the business value, how we are going to measure progress and success, what are the impediments and potential risks associated with the hypothesis (see Figure 2).
The Hypotheses Development Canvas includes the following:
A Vision Workshop accelerates the collaboration between the business stakeholder and the data science team to identify the hypothesis requirements that underpin data science engagement success.
The key to being a successful data scientist is similar to being a good detective – ask lots of questions, create lots of propositions, and do lots of exploration and investigation.
