Data Pre-Processing: A Crucial Element of Analytics
The goal of the Internet of Things (IoT) is to acquire data from various embedded systems and impart analytical processes on that data to improve performance, efficiency, and business outcomes. The ability to create analytics that process massive amounts of business and engineering data is enabling designers in many industries to develop intelligent products and services. Designers can use analytics to describe and predict a system’s behavior, and further combine analytics with embedded control systems to automate actions and decisions.
Whether cloud-based or embedded, the first step in developing analytics is to access the wealth of available data to explore patterns and develop deeper insights. This abundance of data is explored through data pre-processing, a crucial, yet often understated step in the creation of analytics-driven embedded systems.
Datasets can be large in size, come from many different sources and represent many different attributes. Therefore, the software tools used for exploratory analysis and analytics development should be capable of accessing all the data sources and formats that comprise the dataset. As real world data tends to be incomplete, noisy and inconsistent, data preparation is a very important procedure for both data warehousing and data mining.
Data preparation includes data cleaning, data integration, data transformation and data reduction. Data cleaning steps can be used to fill in missing values, smooth the noisy data, identify outliers and correct data inconsistencies. Data integration combines data from multiple sources to form a coherent data store. Data transformation routines segregate data into appropriate forms for mining. Data reduction can be used to obtain a reduced representation of the data while minimizing the loss of information content.
When exploring this wealth of information data pre-processing cleans and prepares the data before predictive models are developed. Predictions from incorrect data can be difficult to debug, or worse, can lead to inaccurate or misleading results that impact system performance and reliability. The goal here is to find the most predictive features of the data and filter it so it will enhance the predictive power of the analytics model. Some common techniques include feature selection to reduce high-dimension data, feature extraction and transformation for dimensionality reduction, and domain analysis such as signal, image, and video processing.
The information gathered from data pre-processing is then taken and implemented across a number of analytics-driven embedded systems. An example of this is the innovation in using Big Data and analytics to make cars smarter.
