Real-Time Activity Tracking With Kafka
When LinkedIn started growing its member base, the site’s functionalities got more complex day by day. In 2010, they decided to invest in redesigning the infrastructure to facilitate the blooming need of scaling their multiple data pipelines without much hassle. As a result, Kafka — a single, distributed pub-sub — was born to handle real-time data streams in each pipeline. In the very next year, Kafka went open-source under Apache and has been used in huge production scales ever since.
Kafka is a fast, scalable, durable, fault-tolerant pub-sub messaging system. It’s written in Scala and Java and uses Apache Zookeeper for reliable distributed coordination. Kafka provides four core APIs: Producer, Consumer, Streams, and Connector.
Kafka is composed of a few building blocks. The topic is the feed that represents a stream of records in Kafka. Each topic has one or more partitions that are physical separations of ordered and immutable sequence of records within a topic. Kafka uses the concept of a commit log to append each record, assigned with a sequential integer, the offset in a partition. Producers publish records to topics, while Consumers can subscribe to topics and consume the records. These messaging systems that maintain the published data are called Brokers. One or more such brokers compose a Cluster.
Over big data, fast data is becoming more of a household name lately, as companies are struggling to process real-time data streams. Since Kafka is capable of handling real-time data feeds with high throughput, low latency, and guaranteed reliability, more than a third of the Fortune 500 companiesnow use Kafka in production.
In its earlier stages, Kafka was used by LinkedIn only for online and offline real-time event consumption, traditional messaging use cases, gathering system health metrics, activity tracking, and feeding the data streams into their Hadoop grid. However, today, Kafka is a critical part of LinkedIn’s central data pipeline, handling over 1.4 trillion messages a day, as a whole ecosystem has been built around it.
Last year in Kafka Summit, impressive facts were revealed about the Four Comma Club. Netflix was one of the companies processing over one trillion messages a day using Kafka, apart from LinkedIn and Microsoft. This boasts the high scalability that this distributed messaging system facilitates. In their Keystone pipeline, Netflix uses Kafka for complex event processing and real-time monitoring.
Uber uses Apache Kafka in its core infrastructure for online and near real-time event processing. The list goes on and on, from traditional messaging, application monitoring, and activity tracking to usages in recommendation and decision engines, custom preferences and personalization, fraud detection, complex event processing, and ingesting data into Spark and Hadoop.
Handling enormous volumes of real-time data streams generated by systems like IoT has erupted pivotal challenges for enterprise giants. Precursory technologies and tools were not equipped to tackle the problems caused due to the scale and speed of these systems. This gave rise to a growing need for real-time analytics rather than traditional big data analytics. Apache Kafka is a fast, scalable, durable, fault-tolerant pub-sub data streaming platform, and hence is endowed to address many of these business problems.
In this article, we are going to discuss integrating Apache Kafka with UltraESB-X using UltraStudio, for real-time messaging. If you want to know more about UltraESB-X, this post can provide a good starting point. UltraStudio provides a graphical IDE to build, test, and deploy integration projects without any fuss.
Hogwarts is in deep waters lately. After Dolores Umbridge took over the headmastership, not even a pixie could flutter without her knowing. She made sure that school’s webmaster, Prof. Quirinus Quirrell, is also dancing under her “imperius.
