Five Steps to Better Data Quality
If software is eating the world, then it’s data that serves as the calories. Just as there are good calories and bad calories, not all data is equal. As more and more companies use data to drive their decision-making processes, getting good quality data is a crucial ingredient.
You might think data science is a recent thing. But actually, it’s been around since the 1980s, in the form of concepts such as data warehousing. If you scour the internet, looking for advice on getting better data quality, you’ll read about long-standing concepts such as metrics of accuracy and timeliness. In addition, you’ll find more recent tactical advice on how to sanitize data using Python or R. How do we better navigate through this dizzying array of information?
Well, I’m here to help. Here are the five steps you can take today to get better data quality.
Whether it’s to help management make better decisions faster or to help ground level staff be more responsive, your data has to be useful. Therefore, you have to define what “useful data” looks like.
These are the most common metrics we use to define useful data:
You can then use these metrics as a checklist to help guide your data quality control:
As a result, you have a wide range of options to define what useful data looks like.
And sometimes less is more. For example, Pinterest realized their new visual AI feature was more useful when they deliberately went for less precision:
That’s the lesson Pinterest learned from avocados. Exact matches are the specialty of Google search, which has been optimized to respond to specific questions—like, “How do you grill fish?”—with the perfect link. Pinterest users tend to pose vaguer queries: They might search for “seafood dinner ideas” several times a week. For them, a non-exact match is not an error. It’s inspiration. — Fast Co. Design article
For users, this imperfect matching is a feature rather than a bug. So, you don’t need to have perfect data to have good results. Aim for usefulness, not perfection.
Profiling means you analyze the information in order to clarify the structure, content, relationships, and derivation rules of the data. This is a crucial step. Users tend to have an understanding, on an intuitive level, of how data is interrelated. Unfortunately, machines currently still need precise instructions. So you need to profile the data at hand and make it work for your users via data analysis software.
Start by clarifying how different data points are related to one another. How do you want to group and structure them? What rules do you want to apply to the data to derive for display purposes? These are the typical steps you apply in data profiling.
But don’t think this is a one-and-done step. Usually, you perform more in-depth profiling before you build your models. After that, you may still continue to perform detailed profiling. That’s because continuous detailed profiling helps determine the appropriate data for extraction and the appropriate filters to apply to your data set.
Sometimes, even after the data loading stage, you may want to continue to perform profiling. This helps ensure that you correctly sanitize the data and transform it to comply with your requirements.
Think of it like this:
If you’re doing it right, the amount of profiling work should decrease over time for your project
This chart displays how the nature of profiling work changes as the project progresses.
Setting policies about data standardization is another crucial step for data quality. Standards help improve communications.
Good communication means two different parties can understand one another quickly and completely, with minimal confusion. That’s also true for communicating your data to your audience.
There are two kinds of standardization: external and internal. External standards (as in outside your organization) are appropriate for commonly used data types.
