Serverless Data Lake: Storing and Analysing Streaming Data using AWS

By this time, we have downloaded and preprocessed the dataset. Since files contain millions of records, you can partition and divide your files into multiple years. I’m using the Python Kinesis Aggregation module. Here are some of the benefits of using KPL:

I’m using Callback-based Aggregation, for this, you must register a callback via the method. As you add individual records to the object, you will receive a callback (on a separate thread) whenever a new fully-packed aggregated record is available.

Kinesis Data Analytics applications continuously read and process streaming data in real-time. You write the application code using SQL to process the incoming streaming data and produce output. Then, Kinesis Data Analytics writes the output to a configured destination. For this, let’s login to the AWS Console, and head over to the Kinesis service.

Step 1: Go to Kinesis Data Analytics and create a new Application.

Step 3: In with Lambda section, enable that and create a new lambda function. Since Kinesis Data Analytics accepts only csv and json records, so we need to convert our KPL formatted records into csv so that Kinesis Data Analytics can process it. For each batch of records, the application receives, the service manages how each batch gets passed to your Lambda function. You can learn about Lambda Preprocessing . All records that are passed to your Lambda function, must be returned with recordId, result and data properties, otherwise, the application rejects them. For each record, you parse multiple csv records, process and join them, and send them back to the application with the same RecordId. Here is Lambda Function’s Code:

Step 4: Choose an IAM role which gives the application the right permissions to do the following:

Step 5: Start your application and discover schema. You can filter the columns and edit the schema to your choice.

Step 6: We will be using a TOP_K_ITEMS_TUMBLING function which returns the most frequently occurring values in the specified in-application stream column over a tumbling window. We will be displaying the trending locations of crimes over a tumbling window of 60 seconds. Save and run your SQL. Run your producer and you can see the real-time query results in the window.

Step 7: Configure Kinesis Firehose as your application output stream. We will be using a buffer period of 5MB and 300 seconds. You can configure the following Amazon S3 object key used for delivering data from Kinesis Data Firehose.

Here is the final configuration of our output stream.

You can run your producer, once either of the condition is met (5 MB or 300 seconds), a new partition will be created on S3.