Machine learning for data cleaning and unification
The biggest problem data scientist face today is dirty data. When it comes to real world data, inaccurate and incomplete data are the norm rather than the exception. The root of the problem is at the source where data being recorded does not follow standard schemas or breaks integrity constraints. The result is that dirty data gets delivered downstream to systems like data marts where it is very difficult to clean and unify, thus making it unreliable to utilize for analytics.
Today data scientists often end up spending 60% of their time cleaning and unifying dirty data before they can apply any analytics or machine learning. Data cleaning is essentially the task of removing errors and anomalies or replacing observed values with true values from data to get more value in analytics. There are the traditional types of data cleaning like imputing missing data and data transformations and there also more complex data unification problems like deduplication and repairing integrity constraint violations. All of these are inter-related, and it is important to understand what they are.
Schema mapping looks at multiple structured data and figures out whether they are talking about the same thing in the same way. In the example below, does “building #” and “building code” both represent building number?
Record linkage is where multiple mentions of the same real-world entity appear across the data. The different formatting styles for each source leads to records that look different but in fact all refer to the same entity. In the example below all four table records are referring to the same medical lab.
Missing data refers to values that are missing from a dataset. Missing value imputation is the process of replacing missing data with substituted values. In practice, the problem is more complicated because missing data is not represented by Nulls but instead by garbage, like in the example below.
Integrity constraints ensure that data follow functional dependencies and business rules that dictate what values can legally exist together. Deducing constraints from data can be very difficult, especially since most data relations are non-obvious. In the example below, Jane Smith is a building manager for both the Medical Lab and Management building which breaks the business rule. For the rule to hold, either Jane Smith is not the management for one of the two buildings or the medical lab is actually a management building, or the management building is a medical lab.
Above we’ve seen a few of several data quality challenges. The problem is that most data scientists are employing rule-based tools and ETL scripts that handle each of the data quality issues in isolation. Whereas the fact is that most data, like the figure below, usually have all if not most data quality problems and they interact in complex ways. The problem is not just of tools being unable to handle interaction between data quality issues. The solutions don’t even scale well on large data sets due to high levels of computation and require multiple passes before enough corrections have been made.
Considering the issues with current solutions, the scientific community is advocating for machine learning solutions for data cleaning which consider all types of data quality issues in a holistic way and scale to large datasets.
Entity resolution is a good example of data unification task where machine learning is useful. The three primary tasks involved in entity resolution are deduplication, record linkage, and canonicalization. At the core of deduplication, we want to eliminate duplicate copies of repeated data. With record linkage we aim to identify records that reference the same entity across different sources.
