Amazon MSK

Amazon Managed Streaming for Apache Kafka (MSK) is AWS’s fully managed Kafka service, handling cluster operations, infrastructure scaling, and broker recovery without requiring deep expertise in Kafka administration.

Reviewed by 7wData
API Available

On this page

Publisher review

Amazon Managed Streaming for Apache Kafka (MSK) is AWS's fully managed Kafka service, handling cluster operations, infrastructure scaling, and broker recovery without requiring deep expertise in Kafka administration. It offers three deployment models: Provisioned (traditional broker-based), Serverless (auto-scaling on partition demand), and Express (3x throughput, 20x faster scaling, 90% faster recovery). The service is tightly integrated with AWS—IAM for authentication, CloudWatch for observability, S3 for tiered storage, Lambda for event triggers, and VPC for network isolation.

These integrations eliminate the need for separate tooling layers. MSK is well-suited for teams deeply invested in AWS who prioritize operational simplicity over platform flexibility. However, the underlying Kafka architecture couples compute, storage, and recovery on local broker volumes, creating structural constraints.

Scaling requires manual broker additions and partition rebalancing; retention economics favor shorter windows because tiered storage to S3 still charges broker-local storage until data ages; cross-AZ replication generates substantial data transfer costs. Organizations with dynamic workloads, strict data locality requirements, or multi-cloud mandates often find MSK's preset configurations and limited autoscaling force over-provisioning and operational workarounds that eventually outweigh the managed-service convenience.

Get the AI & data signal, daily.

41k+ read Ins7ghts, 25k+ read AI Beat. Ins7ghts and AI Beat: two free emails every morning on AI and data. Unsubscribe anytime.

How it works

  1. Express Brokers

    Purpose-built broker type delivering 3x higher throughput and 20x faster scaling than Standard brokers, with 90% lower recovery time and M7g Graviton3 CPU for 24% better price-performance than M5 instances.

  2. Tiered Storage to S3

    Automatically offloads older broker-local data to Amazon S3, reducing broker storage costs for long-retention workloads while keeping data queryable without reading from expensive local volumes.

  3. Native IAM & CloudWatch

    Integrates AWS Identity & Access Management for authentication and Apache Kafka authorization without a separate auth layer; CloudWatch metrics, alarms, and logs for operational monitoring.

  4. Serverless Mode

    Consumption-based pricing with automatic scaling: pay per partition-hour plus data ingestion/egress, no manual cluster sizing needed, scales to zero.

  5. Multi-AZ High Availability

    Automatic detection and recovery from broker and infrastructure failures across availability zones; replicated topics survive single-AZ outages.

  6. Managed Kafka Connect

    Pre-built, fully managed source and sink connectors for common integrations (S3, DynamoDB, Kinesis, RDS, Lambda) without operating Connect clusters.

  7. VPC Peering & Private Connectivity

    MSK clusters run inside customer VPCs; no internet exposure required. Multi-VPC private connectivity available for cross-VPC client access without data egress charges.

Strengths and trade-offs

Strengths

  • Seamless AWS integration (IAM, CloudWatch, S3, Lambda) eliminates separate tooling and auth layers for AWS-native architectures.
  • Express brokers deliver dramatic performance improvements (3x throughput, 20x scaling speed) for partition-heavy workloads with lower operational overhead.
  • Low-overhead management: automatic broker recovery, capacity planning tools, and simplified initial setup for teams familiar with AWS.

Trade-offs

  • Broker-local storage architecture forces over-provisioning: teams must reserve 50% headroom for failures and traffic spikes; more than half of brokers sit idle most of the time.
  • Limited autoscaling and manual scaling friction: cluster expansion requires partition reassignment spreadsheets and operational freezes; no single-mode fits all workload types.
  • High cross-AZ costs and retention bottlenecks: replication between AZs generates substantial data transfer charges; tiered storage still charges for broker-local storage until data ages; complex cost management for multi-region setups.

Pricing context

MSK Provisioned charges hourly broker instance fees (billed per second), storage in GB-months, and data transfer. Example: three kafka.m7g.large Standard brokers in US East with 1TB for 15 days and 2TB for 16 days costs approximately $607/month before data transfer. Express brokers cost more but deliver higher throughput and faster scaling.

MSK Serverless charges hourly cluster fees, per-partition-hour fees, and data-in/data-out consumption; example: 5 topics with 20 partitions each, 100GB daily writes, 200GB daily reads costs approximately $1,300/month. MSK Connect charges per compute unit. All models incur standard AWS data transfer fees for ingress/egress outside AWS; cross-AZ replication is free, but outbound traffic to customers or other regions carries full AWS data transfer costs. No minimum fees or upfront commitments.

Getting started with Amazon MSK

  1. Sign up for AWS account

    Create an AWS account if you don't have one. Navigate to the AWS Management Console, search for MSK, and open the Amazon MSK dashboard. Ensure your IAM user has permissions to create and manage MSK clusters.

  2. Create an MSK cluster

    In the MSK dashboard, click Create cluster. Choose a deployment model (Provisioned, Serverless, or Express). Configure broker instance type, number of brokers, storage per broker, and VPC settings. Set authentication method (IAM or TLS). Click Create.

  3. Configure client connectivity

    After cluster creation, note the bootstrap broker string from the cluster details. Configure your client application's security groups to allow inbound traffic on port 9092 (or 9094 for IAM). If using IAM, attach an IAM policy granting kafka-cluster permissions to your client role.

  4. Produce and consume test messages

    Install the Apache Kafka client tools on an EC2 instance in the same VPC. Use kafka-console-producer.sh to send messages to a topic, and kafka-console-consumer.sh to read them. Verify end-to-end connectivity and message flow.

  5. Set up monitoring and alerts

    In the MSK console, enable CloudWatch metrics for your cluster. Create CloudWatch alarms for key metrics like CPU utilization, disk usage, and request latency. Optionally, configure AWS Lambda triggers for automated responses to cluster events.

Frequently Asked Questions

What is Amazon MSK and how does it work?

Amazon MSK is AWS's fully managed service for Apache Kafka that handles cluster operations, infrastructure scaling, and broker recovery. It offers three deployment models: Provisioned, Serverless, and Express, and integrates with AWS services like IAM, CloudWatch, and S3.

What are the different deployment models for Amazon MSK?

Amazon MSK offers Provisioned (traditional broker-based), Serverless (auto-scaling on partition demand), and Express (3x throughput, 20x faster scaling, 90% faster recovery). Each model has different pricing and performance characteristics to suit various workload needs.

How does Amazon MSK pricing work with examples?

MSK Provisioned charges hourly broker fees, storage in GB-months, and data transfer. For example, three kafka.m7g.large Standard brokers with varying storage costs about $607/month. Serverless charges hourly cluster fees and per-partition-hour fees plus data consumption. Express brokers cost more but offer higher throughput.

What are the main limitations of Amazon MSK?

Amazon MSK's broker-local storage architecture forces over-provisioning, requiring 50% headroom. Scaling is manual with partition rebalancing friction. Cross-AZ replication generates substantial data transfer costs, and tiered storage still charges for broker-local storage until data ages.

How does Amazon MSK compare to Confluent Cloud?

Amazon MSK offers seamless AWS integration but has preset configurations and limited autoscaling. Confluent Cloud provides more multi-cloud support, a richer connector ecosystem, and greater configuration flexibility. MSK may require operational workarounds for dynamic workloads.

What AWS integrations does Amazon MSK support?

Amazon MSK integrates with IAM for authentication, CloudWatch for monitoring, S3 for tiered storage, Lambda for event triggers, and VPC for network isolation. These integrations eliminate the need for separate tooling layers in AWS-native architectures.

Alternatives

How Amazon MSK compares

Direct head-to-head against 2 competitors. Picked by 7wData.

This tool

Amazon MSK

Pricing
MSK Provisioned charges hourly broker instance fees (billed per second), storage in GB-months, and data transfer. Example: three kafka.m7g.large Standard brokers in US East with 1TB for 15 days and 2TB for 16 days costs approximately $607/month before data transfer. Express brokers cost more but deliver higher throughput and faster scaling. MSK Serverless charges hourly cluster fees, per-partition-hour fees, and data-in/data-out consumption; example: 5 topics with 20 partitions each, 100GB daily writes, 200GB daily reads costs approximately $1,300/month. MSK Connect charges per compute unit. All models incur standard AWS data transfer fees for ingress/egress outside AWS; cross-AZ replication is free, but outbound traffic to customers or other regions carries full AWS data transfer costs. No minimum fees or upfront commitments.
Target
Amazon Managed Streaming for Apache Kafka (MSK) is AWS's fully managed Kafka service, handling cluster operations, infrastructure scaling, and broker recovery without requiring deep expertise
Strength
Seamless AWS integration (IAM, CloudWatch, S3, Lambda) eliminates separate tooling and auth layers for AWS-native architectures.
Watch for
Broker-local storage architecture forces over-provisioning: teams must reserve 50% headroom for failures and traffic spikes; more than half of brokers sit idle most of the time.

Confluent Cloud

Pricing
Pay-as-you-go per broker, storage, and data transfer. Claims 40-70% TCO savings vs self-managed Kafka.
Target
Teams needing a fully managed Kafka platform with advanced features like ksqlDB and Schema Registry.
Deployment
AWS, GCP, Azure
Strength
Cloud-native Kora engine with autoscaling and 99.99% SLA, 10x better than MSK.
Watch for
Pricing can escalate with data transfer and advanced features; vendor lock-in risk.

Amazon Kinesis Data Streams

Pricing
Pay per shard-hour and data ingested. Predictable, usage-based pricing.
Target
Teams prioritizing simplicity and deep AWS integration for real-time streaming.
Deployment
AWS only
Strength
Simpler setup than MSK with native AWS Lambda and S3 integration.
Watch for
Limited to AWS ecosystem; smaller tool ecosystem than Kafka; shard limits require scaling management.

User reviews

No user reviews yet. Be the first to write one.

Sources

Reporting on this tool draws on these publicly available sources.

  1. aws.amazon.com — Core capabilities, deployment models, Express brokers, and AWS integration features
  2. docs.aws.amazon.com — Broker types, instance sizes (Standard vs Express), M7g Graviton3 support, and performance recommendations
  3. aws.amazon.com — Pricing models for Provisioned, Serverless, and Connect; example costs; storage and data transfer charges
  4. www.automq.com — Structural limitations: broker-local storage coupling, scaling friction, cross-AZ costs, retention economics, replication complexity
  5. www.peerspot.com — Customer review feedback: stability and scalability, AWS integration, low maintenance, documentation gaps, limited flexibility and autoscaling
  6. www.peerspot.com — Head-to-head comparison with Confluent Cloud: cost, features, multi-cloud support, connector ecosystem, configuration flexibility
  7. www.automq.com — Scaling limitations, partition rebalancing costs, overprovision requirements, performance bottlenecks