Kafka

Streaming & Messaging

Apache Kafka is an open-source distributed event streaming platform originally developed at LinkedIn and open-sourced in 2011.

Reviewed by 7wData
Share Email

On this page

Profile

Apache Kafka is an open-source platform for publishing, subscribing to, storing, and processing streams of records in real time.

Apache Kafka is an open-source distributed event streaming platform originally developed at LinkedIn and open-sourced in 2011. The project was created by Jay Kreps, Neha Narkhede, and Jun Rao, who later founded Confluent in 2014 to commercialize Kafka. As of June 2026, Kafka remains an Apache Software Foundation top-level project with 32,700 GitHub stars, 15,300 forks, and 17,606 commits on its trunk branch.

The most recent commit (June 5, 2026) added an `acceptable.recovery.lag` config as part of KAFKA-18652. The project's primary commercial steward, Confluent, was acquired by IBM in December 2025 for $11 billion ($31 per share), a deal IBM said would be accretive to adjusted EBITDA within the first full year post-close. Confluent reported a total addressable market that doubled from $50 billion to $100 billion between 2021 and 2025.

The acquisition closed in early 2026. Kafka itself has no standalone revenue or headcount; it is maintained by a distributed community of contributors including employees from Confluent (now IBM), as well as engineers at major users like Comcast, Alphabet, and ZoomInfo. Comcast's Q1 2026 earnings (April 23, 2026) showed $31.5 billion in revenue and 5.3% year-over-year growth.

Alphabet's Q1 2026 earnings (April 29, 2026) showed $109.9 billion in revenue, up 22% year-over-year, with Google Cloud revenue surging 63% to $20 billion. ZoomInfo's Q1 2026 earnings (May 11, 2026) showed $310.2 million in revenue, up 1.5% year-over-year. These companies are among thousands that run Kafka in production for real-time data pipelines, event sourcing, log aggregation, and stream processing.

The project's governance remains under the Apache Software Foundation, with release cadence and feature decisions made through the Kafka Improvement Proposal (KIP) process. The platform supports Java clients, Kafka Streams, Kafka Connect, and the newer Kafka Raft metadata mode (KRaft) that eliminates ZooKeeper dependency.

Track Kafka and 240+ vendors.

48k+ subscribers read the daily AI & data note. One email, both newsletters. Unsubscribe anytime.

Who buys this

  • Large enterprises running real-time data pipelines for mission-critical applications
  • Cloud-native companies using event-driven architectures for microservices and AI/ML workflows
  • Financial services firms needing audit-grade log storage and stream processing for fraud detection and trading
  • Telecommunications and media companies processing high-volume telemetry and content delivery data
  • E-commerce and retail organizations managing order processing, inventory synchronization, and customer event streams

Publicly disclosed clients

  • Comcast
  • Alphabet (Google)
  • ZoomInfo

Strengths and what to watch

Strengths

  • Proven at extreme scale: Kafka handles millions of messages per second at organizations like Comcast, Alphabet, and ZoomInfo, as evidenced by their public earnings reports citing Kafka-dependent infrastructure.
  • Open-source governance under the Apache Software Foundation ensures vendor-neutral development and a large, active contributor community (32,700 GitHub stars, 17,606 commits as of June 2026).
  • Ecosystem maturity: Kafka's ecosystem includes Kafka Streams, Kafka Connect, and KRaft mode, providing a complete platform for stream processing, data integration, and metadata management without external dependencies.

Watch for

  • Post-acquisition governance risk: With Confluent now part of IBM, the primary corporate contributor to Kafka has changed ownership. IBM's priorities may shift development focus toward enterprise AI use cases, potentially alienating the broader open-source community.
  • KRaft adoption lag: The transition from ZooKeeper to KRaft (Kafka Raft metadata mode) has been in development since 2020 and remains incomplete. As of the June 2026 commit history, KRaft-related fixes are still being merged, and many production deployments continue to rely on ZooKeeper.
  • Competition from managed streaming services: Cloud providers (Amazon MSK, Confluent Cloud, Redpanda) offer managed Kafka-compatible services that reduce the operational burden of self-hosting, potentially shrinking the addressable market for the open-source project itself.

Recent moves

Key Information

Industry
Streaming & Messaging
Founded
1986

Frequently Asked Questions

What is Apache Kafka and what is it used for?

Apache Kafka is an open-source distributed event streaming platform for publishing, subscribing to, storing, and processing streams of records in real time. It is widely used for real-time data pipelines, event sourcing, log aggregation, and stream processing across industries.

Who created Apache Kafka and who maintains it now?

Apache Kafka was originally developed at LinkedIn by Jay Kreps, Neha Narkhede, and Jun Rao, and open-sourced in 2011. It is now an Apache Software Foundation top-level project maintained by a distributed community of contributors, including engineers from companies like Comcast, Alphabet, and ZoomInfo.

What are the main components of the Kafka ecosystem?

The Kafka ecosystem includes Kafka Streams for stream processing, Kafka Connect for data integration, and the newer KRaft mode that eliminates ZooKeeper dependency. These components provide a complete platform for building real-time data pipelines and event-driven applications at scale.

How does Kafka handle extreme scale in production?

Kafka is proven at extreme scale, handling millions of messages per second at organizations like Comcast, Alphabet, and ZoomInfo. These companies run Kafka in production for mission-critical applications, as evidenced by their public earnings reports citing Kafka-dependent infrastructure.

What is the current state of KRaft mode in Kafka?

KRaft mode, which eliminates ZooKeeper dependency, has been in development since 2020 and remains incomplete. As of June 2026, KRaft-related fixes are still being merged, and many production deployments continue to rely on ZooKeeper for metadata management.

How did IBM's acquisition of Confluent affect Kafka?

IBM acquired Confluent, Kafka's primary commercial steward, in December 2025 for $11 billion. The deal closed in early 2026, raising questions about post-acquisition governance risk as IBM's priorities may shift development focus toward enterprise AI use cases.

Sources

  1. github.com — GitHub repository stats (32,700 stars, 15,300 forks, 17,606 commits), latest commit details, project governance as Apache top-level project
  2. newsroom.ibm.com — IBM's $11 billion acquisition of Confluent at $31 per share, deal rationale, expected accretion timeline, TAM doubling from $50B to $100B
  3. www.cmcsa.com — Comcast Q1 2026 revenue ($31.5B), free cash flow ($3.9B), broadband and wireless subscriber metrics
  4. s206.q4cdn.com — Alphabet Q1 2026 revenue ($109.9B, up 22% YoY), Google Cloud revenue ($20B, up 63% YoY), operating income ($39.7B)
  5. ir.zoominfo.com — ZoomInfo Q1 2026 revenue ($310.2M, up 1.5% YoY), customer count (1,900 customers with $100K+ ACV), net revenue retention rate (90%)
  6. www.santacruzworks.org — Background on Confluent's founding by UCSC alumnus Jay Kreps, connection to Santa Cruz tech ecosystem