Data lineage: Making artificial intelligence smarter

Imagine you work in an office building in the bustling center of a large city. On your lunch break, you go for a walk to get some exercise and clear your head. Half an hour later, you realize that you haven’t been paying attention to your surroundings and don’t know where you are – but you need to get back to the office quickly. You pull out your smartphone and use a few trusty GPS-enabled apps to see your exact location, the path you took to get there and the fastest route back to the office. You even get some recommendations for quick lunch stops along the way. That’s a good analogy for data lineage, which details the journey data took to get from where it started to where it is now. These days, data lineage is particularly important in the context of artificial intelligence (AI). But before we delve into that, let’s look at a few definitions.

Data lineage defined
As it traces data’s path from its origins to the current location, data lineage shows many important details. These include technical, business and operational metadata – information that describes the following items:
Origins. Data lineage shows where and when data was created or captured, and where it is stored and maintained. This applies to both internal and external data sources.

Characteristics. What the data means in business and technical terms is known as its characteristics. Business metadata provides a glossary of human language descriptions of data that business users understand. Technical metadata provides the language that data models, applications and their proprietary interfaces use to describe the data and its structure.

Relationships. This shows how the data is related, both within itself (e.g., hierarchies) and to other data – including key-based relationships, associations, dependencies, copies or derivatives.

Movements. Movement is all about where the data has been. In today’s hybrid data ecosystems, data moves around a lot in multiplatform environments, from source to staging and sandboxes, to data warehouses and data lakes , and into analytics tools and reports that provide business intelligence. This point-to-point data flow – or data integration from source to current reference point to all destinations beyond – must be fully mapped to reflect a true sense of direction regarding data’s movement.

Processes. It’s important to know what processes the data passed through that may have influenced its values, formatting or filtering, such as data quality , modeling, preparation and integration.
Transformations. This refers to how data was altered during its journey. This includes translations, transformations, data quality rules, data quality test results and reference data values.

Users. This relates to who or what uses the data. Which people and tools have access to the data and for what reasons? When and how often is the data consumed by these users?

Data lineage provides a complete audit trail for data, which is increasingly important for compliance with regulations such as the EU GDPR. Data lineage enables you to trace data quality issues and other errors back to their root cause and perform impact analysis on proposed changes. As it links data in disparate systems at a logical level by showing how metadata is connected, data lineage helps identify business rule discrepancies and data incompleteness. Data lineage also helps data stewards react to issues before they become a problem, define strategies for data quality improvement and promote effective reuse of existing information.

Data Management for Artificial Intelligence
When it comes to artificial intelligence, the old adage “garbage in, garbage out” applies more than ever. Establish a data management strategy for the future – one that accounts for the vital role lineage plays in understanding data and helping AI reach its full potential.