Data Lineage: The History of your Data
A common scenario that data analysts in general encounter is what I like to describe as “data denialism”. Often, and especially while consulting, an analyst will find that the data tells a different story than what the customer holds to be true. It is also often the case that, when presenting this finding, the customer will outright deny the evidence, asserting that either the data or the analysis must be wrong. For example, it may be that a retailer focused on the low-end market is getting most of its sales from high-end customers, and such a fact upends months -maybe even years- of marketing planning and strategy. (This may, or may not, be based on one of my previous consulting experiences)
It is of course part of the analyst’s job to present and discuss such controversial findings carefully and in a way that they can be understood an accepted, or tell a story that is compelling enough to be believable. Of course, too, some discussion about findings is definitely healthy and desirable. But even if the customer is convinced that the analyst did their job right, there’s still the matter of the data itself, for how can the customer be assured that the data is correct? After the myriad transformations, schema modifications, unifications and predictive tasks, how can even the analyst be sure that everything went right?
What the analyst needs to do in this case is to have some form of data lineage system, that is, a way of keeping track of the data’s origins and transformations. This can help not only in justifying controversial statements, but also in debugging and regenerating lost information. If the analyst also has a way of representing this visually, or in a simple summary, he can easily convey his confidence in the analysis to the customer.
At the highest level of abstraction, lineage can be represented as a graph, with task executions (instances or runs of a task) and datasets as the nodes, and the edges representing the connections between them. For example, a collection task could generate several datasets, which could then be processed by many different scripts, resulting in a single, denormalized dataset that is used for a regression task, which finally produces predictions for a key variable (e.g. sales). In a data product pipeline, such predictions may then be used as input for a control task or presented to a decision maker through a webapp.
An important advantage of maintaining and presenting the lineage, is that these predictions (or summaries, cluster groups, market baskets or recommendations in general) are naturally understood to be the ultimate consequence of the full pipeline, not of a detached machine learning task. It is a natural way to present every single part of the pipeline as necessary and meaningful, as opposed to giving most importance to the final, more sophisticated task (A common mistake made by analysts trying to sell their job).
The lineage, then, is a useful tool for the analyst to provide transparecy and confidence to their customer, but also serves as a kind of process documentation (extremely useful for expanding existing work) and reproducibility. All of these extremely important concepts in data analysis in particular, and science in general.
Here is a simple lineage example, manually built and visualized using the excellent Neo4J browser UI.
A keen observer will note that a graph like this is not enough to determine exactly which datasets were involved in the creation of a specific output. It may be that a single task execution produces two different datasets, and if there is no explicit declaration of which inputs were involved in the creation of which output, there is no way to unambiguously connect two datasets.
A way to solve the above issue would be to add explicit documentation about inputs and outputs, for example by implementing the Open Provenance Model into the lineage system, which specifies explicit, abstract definitions for interactions between datasets and tasks.
