Documenting Data Lineage: Practical Implementation Steps
Implementing a data lineage initiative is a complex, and necessary project. An established data management framework and collaboration between data management professionals and stakeholders are a prerequisite for successful implementation of data lineage. There is a set of steps all successful data lineage projects use, and each step is essential.
A company should have serious reasons to start documenting data lineage. Some of these reasons might be:
If one or more of these reasons is crucial for meeting business goals, then a company is ready to start discussing data lineage documentation.
Neither data management nor data lineage should be implemented just for the sake of it, since they require a lot of resources, human as well as financial, and will consume a lot of time. Without the dedication and active support from senior management, such initiatives have no future. Two key groups of benefits might convince a management of a company to support such an initiative. These are:
These monetary benefits will be a result of reducing the cost of many manual operations with data, optimized application landscape etc.
Once senior management has approved the data lineage initiative, the next step is to think about the scope of the initiative.
For each business driver a company has chosen, the corresponding data sets can be found. This is the first filter to use that will reduce the scope of the initiative. For example, GDPR focuses on personal data. If a company just started a data quality initiative, chances are high that the first data set to be reviewed will be customer data.
The second filter is identification of critical data elements (CDE) within these data sets. CDEs are data elements that have the biggest impact on the organization’s performance and customer experience. Usually, these are the key KPIs used to manage the company.
The techniques to identify these CDEs (KPIs) are simple. First step is to choose the most critical management reports and their KPIs, since each report will contain one or more CDEs. The difficulties start with identification of data elements that are needed to calculate these KPIs, and resolution of the primary CDE sources. This step is where the story with data lineage documentation begins. Once the scope of the data lineage initiative has been agreed upon, the scope of data lineage implementation can be defined.
Data lineage can be scoped using the concepts of ‘horizontal and vertical data lineage’.
The whole scope of data lineage starts with the original data sources and ends at the point of final usage.
