Enterprise applications are evolved over the last few decades and sharpen itself with the evolution of IT infrastructure and user needs. Now it is...
Enterprise applications are evolved over the last few decades and sharpen itself with the evolution of IT infrastructure and user needs. Now it is the time for its next evolution, there are a lot of companies evaluating and revisiting their enterprise architecture and applications to move into the cloud. As the initial step, they have started converting their services and apps into micro-services. Since stream processing to be one of the key components in the enterprise architecture; we foresee stream processors will also follow the same trend. Emerging cloud-native stream processing software requires characteristics such as being lightweight, loosely coupled and adhering to the agile dev-ops process. However, most of the traditional stream processors are heavy and depends on bulky monolithic technologies which makes them harder moving to the cloud.
In this blog, I will discuss on Siddhi Stream Processor (https://siddhi.io) which is a 100% open-source cloud-native stream processing agent that allows to implement stream processing applications and run them natively on cloud (i.e, Kubernetes). Siddhi also comes with a set of connectors which can be used to integrate with external systems such as NATS, Kafka, RDBMS, MongoDB, and Prometheus. Let’s understand more about Siddhi and its capabilities below.
Siddhi core is a Complex Event Processing and Stream Processing engine which took birth in 2011. In the Sinhala language [1], ‘Siddhi’ means ‘Events’. Overtime Siddhi engine is integrated with various other open-source and proprietary projects such as Apache Eagle [2], Apache Flink [3], UBER [4] and Punch Platform [5], and many others. Siddhi engine also plays the primary role in various WSO2 products such as WSO2 Stream Processor, WSO2 Data Analytics Server, WSO2 API Management (Traffic Manager component), and WSO2 Identity and Access Management (Adaptive Authentication component). With all the experiences and knowledge gained over the years on building Stream Processing applications, Siddhi team is now working in building distributed Cloud Native Stream Processor powered by the Siddhi engine.
Streaming applications help to express the streaming requirement/need. As per Forrester [6], Streaming applications can be used to achieve below things.
We could identify various patterns in stream processing based on its usages in numerous domains; these streaming processing patterns help to solve the commonly known stream processing problems and to provide some hints for the users on where to apply what stream processing pattern.
As shown in the above figure, Streaming patterns can be applied from the event collection phase itself. then it can spread across into event cleansing, event transformation, event enrichment, summarization, rule processing, event correlation, trend analysis, machine learning, data pipelining, event publishing, on-demand processing, etc. Each of these patterns requires specific streaming constructs such as windows, patterns, IO connectors and ML algorithms to achieve the requirement. Siddhi natively supports all of these streaming constructs which required for the user to achieve their requirements [7].
There are various ways followed over the past to write streaming applications such as code it by yourself, use a stream processing tool and streaming SQL based stream processor. Streaming SQL is the most used and handy way to write Streaming applications; it gives you the natural feeling of dealing with the events. Please check below example,
Consider the above example streaming application which is related to a healthcare use case where Glucose reading events are received through HTTP source and mapped into a stream called ‘GlucoseReadingStream’.
