Why Extreme Programming can be an asset or a liability for data scientists

Is Extreme Programming (aka XP) a good match for data science and big data analytics?

It seems like a leading question, and if you know me well, you might be right. I absolutely believe the values and practices provided by the Extreme Programming movement of the early 2000s deliver the best opportunity for data science teams to build high-quality analytic solutions with big data. But it’s not easy.

My journey with Extreme Programming and agile concepts started in 2004. I was consulting at Hitachi Data Systems, and we had just completed a grueling three-month build of a custom reporting system that tracked the schedule and results of its interoperability testing. We were up against a hard deadline, and we needed to get a lot of moving parts working together (data warehouse, transactional database, web interface, transformation mappings, etc.). We delivered on time, but we nearly killed ourselves in the process—long, frustrating days of continuous development and testing. We knew another build and release was coming next, but there was no way we would go through that experience again. One of our teammates suggested Extreme Programming, and that’s the direction we went. Looking back, that was one of the most significant inflection points of my life.

Over the years, I’ve developed a passion for agile database development. And when data science hit the scene, things got even more interesting.

With all its benefits, Extreme Programming has never been easy to implement, and when you throw persistent data into the mix, it gets even harder. When you add big data, sophisticated algorithms, machine learning, and data visualization, it becomes a formidable feat.

All that said, Extreme Programming practices can work on a data science team, but you must be vigilant about doing them right.

But why even bother with Extreme Programming? Wouldn’t it be much easier to just set up a traditional software development lifecycle with a structured and phased movement through requirements, design, build, test, and deployment?

The answer is yes—it would be easier to execute this way. However, that was the thinking back in the mid 1900s, and there are compelling reasons why Extreme Programming and the other agile methodologies were born. The bottom line is: If you follow a more traditional development methodology, you’ll likely miss the optimal solution from the end user’s perspective. This will manifest in a number of ways: misalignment on business requirements, compromise on features, and production level bugs, just to name a few.