DataOps: The New DevOps of Analytics
According to Gartner’s report, Innovation Insight for DataOps, 27 December 2018, “DataOps is a collaborative data management practice focused on improving the communication, integration, and automation of data flows across an organization.” A relatively new approach, DataOps represents a change in culture that focuses on improving collaboration and accelerating service delivery by adopting lean or iterative practices. Unlike its close cousin DevOps, which focuses on operations and development teams, DataOps is geared towards the data developers, data analysts or data scientists. It also focuses on data operations that are streaming data pipelines down to data consumers such as intelligent systems, advanced analytic models or people.
While the promise of DataOps seems strong, it’s important to understand how the two concepts are the same and how they are different. For example, DataOps isn’t just DevOps applied to data analytics. While the two methodologies have a common theme of establishing new, streamlined collaboration, DevOps responds to organizational challenges in developing and continuously deploying applications. DataOps, on the other hand, responds to similar challenges but around the collaborative development of data flows, and the continuous use of data across the organization.
Exploring the Similarity of Both Approaches
DevOps as a methodology or approach came about to help with providing faster application development process as traditionally the development and delivery of apps and/or products used a waterfall methodology. Developers would gather the requirements first, then start the design, development, test, and quality assurance and finally push the product into production by handing it over to their IT ops counterparts. This methodology worked fine for multi-year development cycles but today most shops use agile development methodology and very quick release cadences. So how are DataOps and DevOps similar? Basically, they each support three common traits:
Just as we have made cloud the delivery mechanism of business applications, we are relying on it more and more for facilitating the DataOps movement. Long gone are the days of relying on a traditional ETL and Enterprise Data Warehouse architecture put together using a monolithic approach. Those design principles were good solutions decades ago, where we used to structure the data ahead of time, extract and load data into a single location for reporting and go through major change management with new requirements. Today, data changes so rapidly that we cannot rely on such a pre-determined, methodical approach.
Just like how one of the most important metrics for a product or application delivery in DevOps is time to market, the same holds true in the data world. The faster you develop and deliver insights to all relevant data consumers, the better your competitive advantage gain will be. As a result, it is important that all data platforms that are accessed by, and made available to, your organization can meet these requirements.
Another key commonality between DataOps and DevOps stems from their ability to promote collaboration among all parties involved. For instance, one of the key principles of DevOps is to get developers and operations on the same page to build and design applications. What used to require a big hand over from developers to IT ops, is now more fluid. In the past, the developer ensured that code is designed property and is tested, and IT ops ensured that it runs properly on various operating systems and can scale. Thanks in large part to containers, which consist of an entire runtime environment, the game is now completely changed! For example, an engineer can provision the environment for their code to run using containers, which makes the actual handover a non-issue, because the container includes everything necessary to run the desired software.
