How to get your big data team to implement functional code every two weeks

Typically, when I challenge data science teams to build functional code in two-week iterations, they think I’m crazy. When most teams embark on a project to create a new data solution, the best they can imagine accomplishing in the first two weeks is maybe a high-level understanding of the requirement. So, the notion of having functional code that end users can use after a period of two weeks is unthinkable. How can you possibly combine advanced mathematics and/or artificial intelligence, a high-volume persistence system, data visualization, and a user interface into a usable system in just two weeks?

Well, as I like to say, “it’s only impossible until it’s not.”

To be fair, there are qualifications to this assertion that are worth noting.

First, you must be prepared to launch into a project with two-week iterations. If you’re not already set up for success, you’ll need to invest some time into an Iteration Zero.

Second, although you’ll deliver functional code that’s useful to end users, the scope of the deliverable (especially the first deliverable) is very limited. Although the users certainly have the option of stopping after just two weeks of development, that’s not the intent.

Finally, you must have an ecosystem of rules and practices, like those found in Extreme Programming that insulate the two-week iteration from all the risks that are inherent in this style of development.

All that said, even with the right environment in place, I still find that data professionals have a hard time conceiving of a two-week iteration. So, I developed a structured method to help data science teams get started with this initially difficult practice—it’s simply called the 4-3-2 method.

The 4-3-2 method consists of: four days of testing, followed by three days of development, followed by two days of refactoring. It fits nicely into a two-week iteration that starts with a one-day iteration-planning meeting with the end user(s). I understand that this method, on the surface, makes the idea of a successful two-week iteration look even more impractical than when we started—however, just stick with me, and give it a try.

At the iteration-planning meeting, although it’s good to have the perspective from multiple end users, I find it’s best if you have one gold user who can speak for all of them; this will prevent valuable time lost in your iteration-planning meeting while end users argue with each other about the requirement.

There are a few important outcomes from this meeting, most importantly the scope for the iteration. The scope must be very small: a simple report, data transformation, or mathematical function that will make their lives a little easier. There will be a lot of reports and functions that the users want, but pick the one that users can most benefit from right now (or in two weeks) and that the data science team knows it can build in two weeks. Then pick one to three more functions and rank them in priority order; your team will definitely hit the top priority, and will probably hit one or more of the others.