Data lineage: What it is and why it’s important
Databases are good at inserting, updating, querying, and deleting data and representing the data’s current state. Developers rely on data consistency so APIs can perform the correct transactions and applications can retrieve accurate records. Other consumers of data include data scientists developing machine learning models and citizen data scientists creating data visualizations.
Query a SQL or NoSQL database for what the data looked like two days ago and you might have to rely on database snapshots or proprietary features to get this view. Snapshots and backups may be good enough for developers or data scientists to compare older data sets, but they are not adequate tools for tracking how the data changed.
There are many good reasons to know more about how people and systems modify data. It’s important to have the capabilities to answer questions such as:
Data lineage is comprised of methodologies and tools that expose data’s life cycle and help answer questions around who, when, where, why, and how data changes. It’s a discipline within metadata management and is often a featured capability of data catalogs that allow data consumers to understand the context of data they are utilizing for decision-making and other business purposes.
One way to explain data lineage is that it’s the GPS of data that provides “turn-by-turn directions and a visual overview of the completely mapped route.” Others view data lineage as a core datagovops practice, where data lineage, testing, and sandboxes are data governance’s technical practices and automation opportunities.
Capturing and understanding data lineage is important for several reasons:
Compliance requirements: Many organizations must implement data lineage to stay on the good side of government regulators. Data lineage in risk management and reporting is required for capital market trading firms to support BCBS 239 and MiFID II regulations. For large banks, automating extracting lineage from source systems can save significant IT time and reduce risks. In pharmaceutical clinical trials, the ADaM standard requires traceability between analysis and source data. Other regulations, including General Data Protection Regulation (GDPR), Personal Informational Protection and Electronic Documents Act (PIPEDA), and California Consumer Privacy Act (CCPA), also require more organizations to implement data governance and data lineage capabilities, especially to track private and sensitive data.
A data-driven culture: Organizations developing citizen data science programs, establishing key performance indicator dashboards, managing a hybrid BI (business intelligence) environment, and taking other steps to become data-driven organizations can easily trip up on data lineage challenges. When the financial data in a dashboard changes significantly, it’s a safe bet that executives want to know what caused the change. Citizen data science and other self-service BI programs are hard to get off the ground if subject matter experts don’t trust the data. Data lineage tools help them better understand data sources, flows, and rules around data they are querying, reporting on, or building into data visualizations.
Transparency: Organizations developing products, services, and workflows seek to improve data quality, create master data hubs, or invest in master data management. These approaches typically include data lineage as a capability to provide transparency on business rules and changes.
