Building IoT Data Pipelines with Python and Talend Data Streams
Benoît Barranco is a Customer Success Architect at Talend. He brings years of experience working with customers and partners across multiple industries, helping them in their solution architectures to become data-driven businesses. In recent years, Benoit has focused on Analytics and Big Data, Data science and Internet of Things working at IBM and PPG Industries. Outside of the office, he likes spending time with his family (wife and twins), playing rugby, Kitesurfing, skiing, snowboarding and making electronics projects.
I was incredibly excited when Talend finally released a new product called Talend Data Streams that you can access for on free on the AWS marketplace. From day one, I have been testing out a few use cases.
Following my previous post on Data Streams, today, I want to walk you through a simple use case of anomaly detection pipeline for IoT data. I’m going to show you how to connect to your Kafka queue from Talend Data Streams, collect data from an IoT device, transform the data, create a streaming window and build Z-Score Model with a Python processor to detect anomalies in humidity readings.
If you want to build your pipelines along with me, here’s what you’ll need:
In my previous post, I’ve described how to install, run and create a topic in Kafka. Feel free to reference that post if you need more detail on how to useKafka here. In this post, I’ll focus on Talend Data Streams itself.
I will be using the same device as my previous post, but this time we will analyze humidity values. As I mentioned before, you could use any device simulator.
Anomaly detection is a technique used to identify unusual patterns that do not conform to expected behavior, called outliers. It has many applications in business, from intrusion detection (identifying strange patterns in network traffic that could signal a hack) to system health monitoring (spotting a malignant tumor in an MRI scan), and from fraud detection in credit card transactions to fault detection in operating environments.
One of the most popular and simple method for outlier detection in the IoT world is Z-Score.
You can forget about importing NumPy, SciPy or Scikit learn libraries since Data Streams can only access to Python 2.7 standard libraries for now PythonDoc2.7 This is not a problem that we can’t fix. Let’s start with Statistics lesson and calculate the Z-Score manually.
The z-score or standard score of an observation is a metric that indicates how many standard deviations a data point is from the sample’s mean, assuming a Gaussian (normal) distribution. This makes z-score a parametric method.
After making the appropriate transformations to the selected feature space of the dataset (in our case creating a sliding window), the z-score of any data point can be calculated with the following expression:
x is the sensor value, m is the mean of the sensor values within the window and the s Standard deviation.
So now what is the Standard Deviation? The simple definition according to MathIsFun is that a standard deviation is a measure of how spread out numbers are. It can be calculated with the following expression:
xi is the series of sensor value within the defined windows and N the number of values within the window.
Coming back to the Z-Score with a simple example, the goal is to find how many standard deviations a value is from the mean.
In this example, the value 1,7 which is 2 standard deviations away from the mean of 1,4, so 1,7 has a z-score of 2. Similarly 1,85 has a z-score of 3. So to convert a value to a Z-Score first subtract the mean, then divide by the standard deviation
In IoT use cases we usually consider that an anomaly is detected when the sensor value is outside the -2s and +2 s range.Z-score is a simple yet powerful method to get rid of outliers in data if you are dealing with parametric distributions in a low dimensional feature space.
I’ll assume for this part that you have a Data Streams AMI up and running.
