Reimagining data governance in the age of AI and machine learning
The growth rates of emerging technologies in artificial intelligence, machine learning and the Internet of Things have drastically exceeded the current speed and ability of businesses to govern and protect their data and information assets.
In many cases, business leaders think that this is an either/or decision: they can either innovate rapidly using emerging technologies, or govern and control using existing governance models. Unfortunately, the larger the gap between the advancements in technology and the underlying data that is governed and protected, the greater the risk and potential loss for the business.
Is there a way for the two to co-exist? Can capitalizing on AI, ML and other emerging technologies be done in a manner that ensures governance?
Traditional data governance provides a rigorous framework designed to establish data standards, business rules, data protection policies and a set of organizational roles and responsibilities such as data stewardship. Like conventional methods, new architectures should also serve the principles of trust, data quality and overall protection of data, but they need to be reimagined for the current context organizations find themselves in.
Onboarding Data Despite the Lack of Predefined Rules
With agility at its core, the future landscape of any data governance architecture should not business definitions to be provided a priori.
Take IoT data for instance. IoT devices are both data gatherers and generators. Wearable devices, sensors and smart electronics collect data by the millisecond and stream that data into a cloud of infinite possibilities. With proper design, IoT data can become the foundation of disruptive modes of customer engagement, new product and service offerings, business models and ultimately to drive pervasive digital transformation initiatives.
The fact is that onboarding of IoT devices or the ingestion of data from these uncertified data sources is extremely difficult in an environment governed by conventional data validation requirements. In these early stages of the data lifecycle, conformity with predefined rules or standards should not be a data governance objective.
Instead, governance should enable quick and efficient incorporation of new data and provide tools that facilitate exploration, pattern detection, and discovery of risk-causing anomalies that eventually will shape governance rules and policies.
Starting with the Data Before Automating Governance
Unlike the old world, where governance came first followed by business applications and analytic models, the new world requires the process to be reversed. Today it’s best to start with a killer application of data, define its success metrics, validate and test it in the market and, finally, create a governance framework around it.
This new world of “apps first, governance second,” has emphasized the need for automated governance, where the hallmarks of traditional data governance, such as data lifecycle management, data lineage and data quality remain important, but happen as a natural byproduct of the data management and analytic cycle without much human oversight.
