Reducing Friction with DataOps
In most cases, businesses are intolerant of inefficiency of any kind, especially if it takes the form of people, process, and/or technological bottlenecks. Businesses expect their day-to-day operations, their hierarchies and organizational structures, their supply chains, their IT operations, etc. to run as smoothly and efficiently – to be as frictionless – as possible. These are reasonable and laudable goals.
On the other hand, we can easily imagine scenarios in which friction functions as a kind of necessary, essential, and – in the proper context – advantageous constraint for certain purposes.
At a basic level, friction is what makes Usain Bolt go. It’s what enables Simone Biles to stick her spectacular landings. Most of us can easily imagine how friction operates as a positive factor in the design of mechanical parts, too: tires, brakes, transmissions, flywheels, belt-driven pulleys, etc., all depend on friction to work. In the same way, friction is a key force in fluid mechanics: it’s friction, for example, that helps govern the flow of liquid – water, oil, etc. – in a closed conduit, such as a pipe.
Like it or not, friction fulfills an important, positive function in any system.
This is as good a segue as any to the crux of this post, which has to do with data management and data governance. The point I want to make is two-fold. First, friction is actually a useful concept in both disciplines. Governance, for example, is the inevitable product of friction between two or more opposing forces or entities – e.g., divergent values, priorities, purposes, etc. – that come into conflict with one another. In a sense, governance isfriction. Albeit friction of a necessary and essential type.
Second, friction is a positive force – a feature, not a bug – because it makes certain things hard to change. If this point seems counter-intuitive, think, for a moment, about your high-value investments. Consider your management information systems (MIS) infrastructure, for example. Some companies pour millions (or tens of millions) of dollars into designing and maintaining their MIS infrastructures. As much as companies like to complain about the state of their data mart, data cube and ETL assets, the fact remains that managers and directors, along with senior and C-level executives, depend on operational reports, dashboards, KPIs, scorecards, etc. to support day-to-day business decision making. As a result, IT doesn’t introduce new data sources, develop new KPIs, devise new business rules, or deploy new dashboards without thoroughly testing and trouble-shooting them. This is one reason it takes so long to provision new data or analytics in conventional data warehouse architecture. Another reason is that – until recently – IT just didn’t have other options. Now it does. I’ll say more about this below.
If it seems as if IT’s priorities are at odds with the self-service ethic, it’s because they are. This is less of a problem than most of us realize, however! Think about it: we’re used to framing the relationship between data management/data governance and self-service as antagonistic: i.e., a zero-sum collision of top-down authority with bottom-up insurgency. Thanks to technological and economic disruption, it’s possible – and more helpful – to see it in terms of a both-and, not a zero-sum, relationship.
First, some obligatory background. In the main, self-service tools were a response to users’ frustration with unnecessary and inessential forms of IT friction, such as overly restrictive data management and data governance controls. Self-service forced IT to reassess both disciplines: what was necessary and essential in data management and data governance – e.g., reusable, governed data transformation routines that produce consistent, well-managed data – is less important in other contexts.
