Samza

Apache Samza is a distributed stream processing framework originally developed at LinkedIn and contributed to the Apache Software Foundation in 2013, becoming a top-level project on January 22, 2015. It is designed for organizations that need to build stateful, real-time applications processing data from multiple sources, particularly Apache Kafka, with extremely low latencies and high throughput. Samza is best suited for engineers who already run Kafka and want a framework that integrates natively with Kafka's partitioning model, offering guaranteed at-least-once delivery and in-order processing within each partition. It supports flexible deployment on YARN, Kubernetes, or as a standalone library, making it adaptable to various infrastructure preferences. The framework is battle-tested at scale, with adopters including LinkedIn, Uber, TripAdvisor, Slack, eBay, Redfin, and Optimizely, and is used for use cases ranging from real-time analytics to monitoring and alerting.

Samza's architecture is built around a pluggable design that integrates with multiple sources and sinks, including Kafka, HDFS, AWS Kinesis, Azure Event Hubs, key-value stores, and ElasticSearch. It offers a rich set of APIs for building applications: a low-level API for fine-grained control, a Streams DSL for common patterns, Samza SQL for SQL-based stream processing, and support for Apache Beam. A notable capability is the ability to run the same code for both batch and streaming data, simplifying code maintenance. Samza scales horizontally to handle several terabytes of state using incremental checkpoints and host-affinity, which reduces recovery time by keeping state local to the container. In a benchmark published by LinkedIn, Samza processed 1.2 million messages per second on a single node, demonstrating its high-throughput performance. The framework processes messages in the order they appear in each partition, which is critical for keyed data like database update streams.

In the stream processing market, Samza competes directly with Apache Flink, Apache Storm, Kafka Streams, and Kafka Connect. Compared to Storm, Samza offers simpler at-least-once delivery without the overhead of tuple ancestry tracking, and it processes messages in order within a partition, making it better for keyed data. Unlike Flink, which has gained broader community momentum, Samza has seen declining community activity and fewer recent releases. Kafka Streams, being a lightweight library rather than a framework, competes for similar Kafka-centric use cases but with a smaller footprint. Samza's integration with YARN and standalone modes gives it deployment flexibility that Kafka Streams lacks, but its complexity and learning curve are higher. The framework's founder has acknowledged that Samza was proposed for wrong use cases and was "born at the wrong time," reflecting its struggle to maintain relevance against more popular alternatives.

The honest trade-offs with Samza are significant. Community activity has declined, with the latest release (1.8.0) coming in January 2023, and fewer contributions compared to Flink or Kafka Streams. The application code can be complex, especially when using the low-level API, and the framework's design lessons suggest it may have been over-engineered for some use cases. Samza only offers at-least-once delivery currently, with exactly-once semantics planned but not yet implemented, which limits its use for applications requiring strict consistency. Its tight coupling with Kafka, while a strength for Kafka users, makes it less suitable for environments using other message brokers. Additionally, the learning curve is steeper than Kafka Streams, and the documentation, while thorough, can be overwhelming for newcomers. For teams already invested in Kafka and needing high-throughput, stateful stream processing with flexible deployment, Samza remains a viable option, but the declining ecosystem and competition from Flink and Kafka Streams make it a riskier long-term choice.

Open-source under Apache License 2.0; enterprise support available from third-party vendors but no official pricing tiers from the Apache project.