data has emerged as an imperative foundational asset for all organizations. data fuels significant initiatives such as digital transformation and the adoption of analytics, machine learning, and AI. Organizations that are able to tame, manage, and unlock their data assets stand to benefit in myriad ways, including improvements to decision-making and operational efficiency, better fraud prediction and prevention, better risk management and control, and more. In addition, data products and services can often lead to new or additional revenue.
As companies increasingly depend on data to power essential products and services, they are investing in tools and processes to manage essential operations and services. In this post, we describe these tools as well as the community of practitioners using them. One sign of the growing maturity of these tools and practices is that a community of engineers and developers are beginning to coalesce around the term “DataOps” (data operations).
Our conversations with members of this nascent community revealed a few key activities associated with DataOps: automation, monitoring, and incident response. In brief, DataOps is composed of tools and processes for monitoring and automating tasks and software that raise the efficiency of operations in support of all data products and services. DataOps tools and processes allow organizations to deliver data products and services quickly, reliably, and efficiently.
More than a decade after the rise of big data management systems, the amount of data that companies need to collect, manage, and unlock keeps growing. Both data volume and the number of data sources have exploded. The emergence of cloud computing, SaaS, mobile computing, and sensors have made operational tasks pertaining to data assets much more challenging. The types of data companies are collecting have also expanded. Machine learning tools have made it possible for companies to unlock unstructured data by incorporating new techniques from computer vision, language models, and speech technologies.
Companies are under increasing competitive pressure to use data and machine learning to modernize their operations and decision-making to gain a competitive advantage in their markets. This means adopting tools that expand the pool of workers—beyond developers, engineers, and data scientists—who use data on a regular basis. Frontline workers, analysts, managers, and executives all need to incorporate data in their decision-making and operations. To raise the productivity of workers who use data, companies will need to adopt tools, such as feature stores and data catalogs, that facilitate collaboration, discovery, and reuse.
Not only do more workers and services depend on data, these new users expect a certain amount of reliability and freshness—near real-time updates in certain scenarios—in their data assets. As more people come to rely on and use data, companies need to adopt technologies and processes that ensure critical data pipelines and infrastructure are actively being monitored and managed. Failures are inevitable in a world of complex systems. The best companies have tools and processes in place that minimize their mean time to recovery from failures.
These challenges are occurring at a time when regulators and users are increasingly concerned with issues related to data privacy and security. Landmark privacy regulations in many jurisdictions have forced companies to improve their tools, not only for data security and privacy, but also for data retention and governance. Data teams are also increasingly under pressure to account for important concerns that fall under the umbrella of Responsible AI (aside from security and privacy, Responsible AI includes such issues as fairness and transparency). DataOps provides a formal set of processes and tools that can help detect, prevent, and mitigate many of the issues that arise as a consequence of Responsible AI.
