Data Quality Problems: Understanding Data Quality and Errors Arise

3 min read
Curated from blog.syncsort.com →

Data quality is important to business. That you know. But do you understand what it takes to provide data quality In yesterday’s blog, we looked at one real-life example of a data quality failure. Today we’ll review how data quality problems can arise.

Briefly defined, data quality refers to the ability of a data set to serve whichever need a company hopes to use it for. That need could be sending marketing materials to customers. It could be studying the market to plan a new product feature. It could be maintaining a database of customer data for help with product support services, or any number of other goals.

No matter what the exact use case for your data, data quality is important because without it, the data can’t fulfill its intended purpose. Errors within a database of addresses would prevent you from using the data to reach customers effectively. A database of phone numbers that doesn’t always include area codes for each entry falls short of providing the information you need to put the data to use in many situations.

Now that we’ve outlined what data quality means and provided a few examples of what it looks like in the real world, let’s delve a bit deeper into the types of problems that lead to data quality shortcomings.

Here are six common ways in which data quality errors can creep into your organization’s data operations, even if you generally adhere to best practices when it comes to managing and analyzing your data:

Humans are prone to making errors, and even a small data set that includes data entered manually by humans is likely to contain mistakes. Typos, data entered in the wrong field, missed entries and so on are virtually inevitable.

Machines can make mistakes when entering data, too. In cases where organizations must digitize large amounts of data quickly, they often rely on Optical Character Recognition, or OCR, technology to do so. OCR technology scans images and extracts text from them automatically. It can be very useful when, for example, you want to take thousands of addresses that are printed on paper and enter them into a digital database so you can analyze them using Hadoop. The problem with OCR is that it is almost always imperfect.

If you’re OCR’ing thousands of lines of text, you’re almost certainly going to have some characters or words that are misinterpreted – zeroes that are interpreted as eights, for example, or proper nouns that are read as common words because the OCR tool fails to distinguish properly between capital and lowercase letters.  The same sorts of issues arise with other types of automated machine entry of data, such as text-to-speech

3. Lack of Complete Information When compiling a data set, you frequently run into the problem of not having all information available for every entry.

Continue Reading

Enjoyed this summary? Read the complete article at the source:

Continue at blog.syncsort.com →

Yves Mulkers

Yves Mulkers is the founder of 7wData and a widely followed voice in the data and AI community. He curates the 7wData and AI Beat newsletters, reaching hundreds of thousands of data and AI professionals, and writes on data strategy, analytics, AI, and the evolving data ecosystem.