How to Manage a DataOps Team
Using a DataOps approach to your big data project — modeled on similar methods used in DevOps teams — could unlock real value for your firm.
Big data should bring big changes in how you work as well as the tools you use if you want to take full advantage of emerging technologies and innovative architectures. DataOps – a style of work that extends the flexibility of DevOps to the world of large-scale data and data-intensive applications – can make a big difference. It’s more than just a buzzword. To make DataOps work, you have to know how to organize and manage a DataOps team. Let’s look at what DataOps is, why it’s worth your consideration and how to make the necessary changes in your cultural organization to put this style of work into action.
We’ll start with value. There’s huge potential value in large-scale data, but just collecting it and storing it isn’t going to get you much value from it. To get the benefits of big data, you have to connect the results of data-intensive applications to actions that address practical business goals – and you need to be able to do this “at the speed of business”. Modern approaches such as streaming, real-time or near real-time data processing, microservice architectures, and machine learning/AI optimization of certain decisions all offer new ways for data-intensive applications and the actions based on them to be a better fit for the way business happens.
These approaches can also give agility and flexibility that lets you respond in a timely manner when conditions change. But it’s difficult to take full advantage of modern agile approaches and emerging technologies if you have a rigid, monolithic organizational culture. You will need to make a change.
This key, but sometimes overlooked, aspect of successful development and production deployment is beginning to get attention. A 2017 survey by New Vantage Partners of F1000 firms and industry leaders indicated that one of the biggest challenges they face is to change their business culture appropriately to deal with big data. Based on what I hear from people in the field, I think that DataOps is a big part of that change.
What exactly is meant by “DataOps”? This term can mean somewhat different things to different people, but fundamentally, roles such as data engineering and data science are coupled with operations and software development to form a DataOps team. This doesn’t usually require hiring additional people; often it’s just a matter of re-organization of people into teams with the right mix of skills. You might embed people with data-heavy data skills into an existing DevOps team. When done properly, the result is not only a faster time-to-value and better use of people’s efforts, but also a change in the rhythm of human decisions throughout the lifecycle of an application. Work goes forward efficiently towards a focused goal, but the team also has the ability to pivot and make adjustments in response to new situations. But the thing that defines the DataOps team – indeed the thing that drives its successful execution – is having a shared data-focused goal connected to real business value.
A DataOps team brings together diversity in experience and skills, which is a good thing. But managing this diverse group may feel like herding cats unless you get genuine buy-in to a shared goal. A strength of a well-designed DataOps team, flexibility with focus, is possible in part because the team cuts across skill guilds and is not slowed down or stalled by a cumbersome series of departmental-based decisions at each step of the pipeline. The team has the skills needed to build and deploy the desired application, from planning to production.
