DataOps: Applying DevOps to Data
The term DataOps is a contraction of ‘Data Operations‘ and a play on the idea of DevOps. It seems to have been coined in a 2015 blog post by Tamr co-founder and CEO Andy Palmer. In this blog post, I’ll dive into what DataOps means today, and how enterprises can adopt its practice to create reliable, always-on dataflows that unlock the value of their data.
In his 2015 post, Palmer argued that the democratization of analytics and the implementation of “built-for-purpose” database engines created the need for DataOps. In addition to the two dynamics Palmer identified, a third has emerged: the need for analysis at the “speed of need,” which, depending on the use, can be real-time, near-real-time or with some acceptable latency. Data must be made available broadly, via a more diverse set of data stores and analytic methods, and as quickly as required by the consuming user or application.
What’s driving these three dynamics is the strategic imperative that enterprises wield their data as a competitive weapon by making it available and consumable across numerous points of use, in short, that their data enables pervasive intelligence. The centralized discipline of SQL-driven business intelligence has been subsumed into a decentralized world of advanced analytics and machine learning. Pervasive intelligence lets “a thousand flowers bloom” in order to maximize business benefits from a company’s data, whether it be speeding product innovation, lowering costs through operational excellence, or reducing corporate risk.
There is an ailment that infects data delivery that has arisen due to the sheer complexity of the emerging data ecosystem – data drift. Data drift is defined as unending, unexpected, and unpredictable changes to data structure and semantics. Data drift can break data pipelines and bring data-driven applications to a halt, or even worse, insidiously invalidate analysis by polluting the data.
Data drift is caused by the explosion of complexity in the data supply chain. It is a consequence of unexpected changes to data sources, especially external data feeds or loosely governed sources such as systems logs. It is induced by changes to the infrastructure processing the data, such as a move to the cloud. Lastly, data drift is caused by changing business requirements, for instance, the addition of latitude and longitude to an address table. Data drift happens without notice, and the consequences impact data stores and downstream applications and consumers.
Legacy data integration assumed a privileged cadre of analysts, and a single analytical ‘source of truth’ with a tightly controlled relational schema. Data integration was analogous to the traditional ‘Waterfall’ approach to software engineering – taking an inventory of operational systems, building the all-encompassing data warehouse, and then basking in the glow of its perfection. The more audacious requirements of pervasive intelligence combined with the peril of data drift break this traditional approach to data movement. The proper response to a new, agile “DataOps” discipline to deal with this more fluid environment.
In 2001, a group of software developers established the Agile Manifesto, a declaration of the tenets of Agile Software Development:
The traditional Waterfall model of software development had a team of systems analysts producing a requirements document, from which a team of software architects produced a system architecture, from which a team of developers wrote the application, which a QA team would then test. Each step could take weeks, months or even years, with no working software available until the conclusion of system testing. If the process did result in a working application, it was, more often than not, late or even obsolete before it went live.
The Agile approach, in contrast, encourages multidisciplinary teams focused on an iterative development model.
