The 12 Rules of DataOps to Avert a DataOops
One of my first consulting assignments with my current employer began several years ago. I was part of a team advising a large organization in how to design and implement an enterprise analytics solution group for the organization’s full end-to-end business activities. Those business activities included operations, manufacturing, supply chain and logistics, and all the steps through pricing, sales and customer support, as well as analytics for internal enterprise functions. Our design sprint team consisted of data scientists (including me), data engineers, a cloud expert, a DevOps expert, business domain experts, and change management leaders.
Our cloud expert responded by saying that our approach would essentially be DevOps for Data. I jumped into the discussion and said, “oh, you mean DataOps!” Everyone in the room had a collective “Aha!” moment and agreed that was the right way to describe it.
And that’s when I became convinced that DataOps is key to successful projects through all DIO phases – it is the methodology that helps deliver on the promise of systems thinking: “the focus is not just on building the systems right, but also on building the right systems.”
My interest in DataOps led eventually to me co-authoring and publishing the small e-book “Ten Signs of Data Science Maturity”, which included this one: “using Agile and leveraging DataOps — DevOps for data product development.”
Since then (and even before then), there have been many definitions, interpretations, and publications that address the DataOps concept. It is more than a data operations or technology concept. It is even more than my original statement “DataOps is DevOps for data.” It is a way of doing the business of data: data understanding, insights discovery, analytic production, and actionable intelligence. It is a full-organization effort since it is now imperative that every facet of the business requires continuous “always on” trusted data at the point of need – the world is now digital; the world is now data. Latency in data products or data analytics delivery is not acceptable.
Entire conferences and handbooks have focused on the diverse meanings of DataOps. Consequently, it is beyond the scope of this brief article to summarize and review all of those. So, instead, I list here my 12 rules (“The 12 Commandments of DataOps”) to help organizations avoid a DataOops in their data analytics initiatives.
It is not only about failed data systems or failed data product implementations. DataOops is exemplified by one or more of the three F’s that affect data-related projects. Those F’s are: Fragility, Friction, and FUD (Fear, Uncertainty, Doubt).
Fragility occurs when a built system is easily “broken” when some component is changed. These changes may include requirements drift, data drift, model drift, or concept drift. Requirements drift is a challenge in any development project, but the latter three are more apropos to data analytics and data product development activities. A system should be sufficiently agile and modular such that changes can be made with as little impact to the overall system design and operations as possible, thus keeping the project off the pathway to failure.
Friction occurs when there is resistance to change or to success somewhere in the project lifecycle or management chain. This can be overcome with small victories (MVP minimum viable products, or MLP minimum lovable products) and with instilling (i.e., encouraging and rewarding) a culture of experimentation across the organization. When people are encouraged to experiment, where small failures are acceptable (if there are lessons learned and subsequent improvements), then friction can be minimized, failure can be alleviated, and innovation can flourish.
FUD occurs when there is too much hype and “management speak” in the discussions.
