Does the Quality of Data Matter?
Software plays an instrumental role in both the production and the correction of quality data.In the first instance, software is used to automate the profiling, cleansing, conditioning and validation of derived data sets. In the second instance, and within limits, software is used to correct for deficiencies in the quality of a derived data set. This contributes to the recognition that software tools can be used to “fix” many types of data quality problems. It likewise contributes to a prejudice: Namely, that data quality does not matter—or, more precisely, that the quality of data does not matterquite as much.
This article explores the problem of data quality from the perspective of the expert user: the data scientists, self-service analytic discoverers, ML/AI engineers, software engineers, etc. who rely on data to do their jobs. It considers the uses and abuses of software tools to correct for the data quality issues that these users are likely to encounter as they acquire, condition and use data in their work.
What Do We Mean by the Quality of Data?
The data quality requirements of data scientists and other experts differ from those of conventional consumers. In creating a derived data set, experts take pains to ensure that their data is statistically and scientifically relevant–that it comprises a sufficiently large (and representative) sample.
In his recent bookData: A Guide to Humans, my friendPhil Harveydescribes five core aspects of what he calls “modern” data quality. Harvey distinguishes between the priorities of traditional data quality–which tend to focus on sales, product, customer, etc. data generated by core OLTP systems–and the more rigorous data quality regime that he believes is appropriate to experimental or scientific use.
: The “what” of the data set, its constitutive entities and relations. Quality problems include missing values, misspellings, inconsistent spellings and missing relations. : The “when” of the data set; agreement and consistency in its temporal dimension. Common quality problems include granularity (for example, data points are recorded at per-second and per-minute intervals for the same entity) and granularity (for example, data points are recorded at per-month and per-minute intervals for the same entity). : The “how” of quality. Do the data points in a data set make sense? Does something about them seem off? Are the individual values of data points consistent with prior samples? The range of data points? Does the of the change recorded in the data set make sense? : The “why” of quality. This “why” has to do not only with the uses to which the data set will be put, but with its status as an accurate representation of the relevant domain. If the data is representative, is it likely to embody certain known biases? The ethical aspect also has to do with how or under what conditions the data set was created. It has to do with the sensitivity of the data (that is, with its “taxonomy,” to use our term). It asks the questions: this data be used in this way for this work? And, if so, what safeguards should we employ prior to using it? : This has to do with the practical application of the data set. What actionable outcomes are likely to devolve from how the data set gets used? A probability calculus must take account of the dominant corporate culture: How comfortable is this culture with ambiguity? Does the culture (or its most influential decision makers) privilege deterministic answers? What does a “yes” answer mean to a decision maker in this culture?
Each of the data quality aspects that Harvey describes is in play when machine learning (ML) engineers assemble a data set for their work. The better the data that a model is trained with, and the better the data it runs against in production, the more predictable its behavior and the more useful its output.
The first three aspects (taxonomy, availability, and deviation) have to do with the content and relevance of the data set itself.
