How data quality improves artificial intelligence

Whether it’s neural networks, natural language processing, machine learning or computer vision, artificial intelligence (AI) is increasingly used to improve enterprise solutions. All AI applications are data-driven and therefore dependent on high-quality data. In this post, we’ll examine examples of how data quality improves AI.

Let’s begin by looking at an AI application that has been around for decades, which most of us still use several times daily – an Internet search engine. Its goal is to help you find websites, images, videos and other online content relevant to your search query. A search engine is driven by, and dependent on, high-quality data in three ways.

First, a search engine is driven by an index of keywords and other metadata that its algorithms use to filter and weight search results. This is how it returns a ranked list of links with the most relevant results listed first. Just like any machine learning for AI, the quality of the search engine’s training data is essential. Its training data is the initial search index along with a set of expected search results for the most common and relatively unambiguous search queries. This enables the analytical model the search engine is building to get off to the right start by establishing the correct correlations between the search index and different search queries.

Up-to-date and frequently refreshed data is necessary for improving any AI application. Web crawlers help search engines obtain this type of data. Web crawlers go from website to website to collect data for updating and optimizing the search index. This is where it encounters an interesting data quality challenge – search engine optimization (SEO).

From an SEO perspective, both individuals and organizations try to game the system to get their websites to rank higher in search results. I remember the early days of the Internet when the footer of fraudulent and malware-ridden websites contained long lists of common keywords. That was an attempt to rank higher and draw more visitors to the page. In this example, search algorithms had to be updated – not because of bad data but because of good data put to bad uses.