Five Challenges of Analyzing Internet of Things (IoT) Data
The analysis of Internet of Things (IoT) data is quickly becoming a mainstream activity. I’ve written about the Analytics of Things (AoT) before (some examples here, here, and here). For this blog, I’m going to focus on a few unique challenges that you’ll most likely encounter as you move to take IoT data into the AoT realm.
With many historical data sources, such as transactional data, it was often quite an effort to gather the source data required for analysis. It was necessary to identify what information was available, how it was formatted, and also to reconcile data from different sources that often contained similar information, but had inconsistencies in how it was provided. Ironically, this is one area where IoT sensor data can seem deceptively simple compared to many other sources.
Most sensors spit out data in a simple format. There is a timestamp, a measure identifier (temperature, pressure, etc.), and then a value. For example, at 4:59pm the temperature is 95 degrees. The good news is that this makes ingesting raw sensor data fairly straight forward in terms of the coding logic required. So, you can fairly quickly go from a raw feed to a dataset or table that’s ready for exploration. The catch is that after the ingestion of the raw data there are some challenges you’ll face before you can analyze your data, as we’ll explore next. Don’t let the simplicity of ingestion fool you.
When it comes to data, such as a retail transaction, we typically care about every single record. With IoT data, it is necessary to determine the cadence that actually makes sense for your specific problem. For example, a temperature sensor may spit out a reading every millisecond. However, in most cases, receiving data at that cadence is overkill. That overkill has a price due to the cost of storing the extra data and the cost and complexity of analyzing masses of data that aren’t valuable.
As a result, it is necessary to determine what cadence actually has value for the problem you’re tackling. If you’re monitoring a car engine, readings once per second might be more than enough. It could be that readings every 5 or 10 or 60 seconds would be plenty. The point is that you have to assess each metric and determine what you need through some experimentation. Then, filter the data down to the proper level. Otherwise, you’ll be overwhelmed with data and meaningful patterns will be that much harder to identify.
At the heart of many IoT analytics is the need to identify complex patterns or trends that occur over time. Classic time series and forecasting models are oriented toward identifying a trend and then extending it forward. When analyzing IoT data, in contrast, we are often interested in deviations from normal rather than projecting the expected.
After identifying what is normal we must do work to find abnormal patterns that are of importance. However, there are multiple ways that abnormal patterns might evolve. Sudden increases in temperature would naturally draw interest. But, what about the impacts of a very small rise in temperature that either persists for an extended period or that comes and goes with increasing frequency? There is much complexity in the identification of these time-based patterns.
This challenge builds on the last two.
