Kinesis vs MSK (Managed Kafka) — Overview
AWS offers two managed streaming platforms: Amazon Kinesis Data Streams (AWS-native, serverless option) and Amazon MSK (managed Apache Kafka). Both handle real-time data streaming but differ in complexity, ecosystem, pricing, and operational model. This comparison is relevant for SAP-C02 (architecture selection) and MLA-C01 (data ingestion for ML).
Feature Comparison
| Feature | Kinesis Data Streams | Amazon MSK |
|---|---|---|
| Protocol | AWS SDK (PutRecord API) | Apache Kafka protocol (any Kafka client) |
| Scaling unit | Shards (1MB/s in, 2MB/s out per shard) | Partitions + brokers (scale independently) |
| Scaling mode | On-demand (auto) or Provisioned (manual) | Add brokers/partitions (MSK Serverless auto-scales) |
| Retention | 24 hours default, up to 365 days | Unlimited (tiered storage to S3) |
| Consumer model | KCL, Lambda, Enhanced Fan-Out | Consumer groups (unlimited consumers per topic) |
| Multiple consumers | Enhanced Fan-Out (dedicated 2MB/s per consumer) | Consumer groups (each reads all partitions independently) |
| Connectors | Limited (Firehose for delivery, Lambda for processing) | Kafka Connect (100+ source/sink connectors) |
| Stream processing | Managed Flink, Lambda | Kafka Streams, Managed Flink, ksqlDB |
| Ordering | Per shard (partition key based) | Per partition (message key based) |
| Exactly-once | At-least-once (dedup at consumer) | Exactly-once semantics (EOS) with transactions |
| Encryption | KMS at rest, TLS in transit | KMS at rest, TLS in transit, SASL/SCRAM auth |
| Serverless option | On-demand mode (fully serverless) | MSK Serverless (auto-scaling, pay per data) |
| Infrastructure | Fully managed (no servers/clusters) | Managed brokers (you choose instance types) or Serverless |
| Portability | AWS-only (proprietary API) | Kafka-compatible (move to/from any Kafka deployment) |
Decision Guide
| Choose Kinesis When | Choose MSK When |
|---|---|
| You want serverless, fully managed with zero ops | You already have Kafka expertise and existing Kafka consumers/producers |
| Lambda is your primary consumer | You need 100+ connectors via Kafka Connect ecosystem |
| AWS-native integration is priority (Firehose, Analytics) | You need exactly-once semantics (transactions) |
| Low-moderate throughput (<10 MB/s) | High throughput (100+ MB/s, many partitions) |
| Quick setup, no Kafka knowledge on team | Multi-cloud/hybrid portability (same Kafka code runs anywhere) |
| Short retention sufficient (≤365 days) | Unlimited retention needed (tiered storage to S3) |
Architecture Patterns
Pattern 1: Kinesis — Real-time Analytics Pipeline
- Flow: IoT devices → Kinesis Data Streams → Managed Flink (windowed aggregation) → DynamoDB/Redshift
- Why Kinesis: Serverless (on-demand mode), Lambda integration for lightweight processing, Firehose for direct S3/Redshift delivery
Pattern 2: MSK — Event Backbone for Microservices
- Flow: 20 microservices produce → MSK topics → Consumer groups per team → Each team processes independently
- Why MSK: Kafka Connect pulls from databases (CDC), consumer groups allow unlimited independent readers, Kafka Streams for stateful processing
Pattern 3: MSK → Kinesis Firehose (Best of Both)
- Flow: MSK (high-throughput streaming backbone) → Firehose (managed delivery to S3/Redshift/OpenSearch)
- Why: Kafka ecosystem for producers/consumers + Firehose’s auto-batching, compression, and format conversion for lake/warehouse delivery
Pricing Comparison
| Model | Kinesis | MSK |
|---|---|---|
| Provisioned | $0.015/shard-hour + $0.014/million PUT | Broker instance-hour ($0.21-$2.16/hr depending on type) + storage |
| Serverless | $0.04/GB ingested + $0.04/GB retrieved | $0.10/hr per partition + $0.10/GB data in/out |
| Cheapest at low volume | ✅ (on-demand scales to zero cost) | More expensive at low throughput (minimum broker cost) |
Rule of thumb: Kinesis is cheaper below ~5 MB/s. MSK becomes more cost-effective above ~50 MB/s with provisioned brokers + Graviton instances.
Exam Tips
| Exam | Key Points |
|---|---|
| SAP-C02 | “Existing Kafka workload” or “Kafka expertise” → MSK. “Serverless streaming” or “Lambda consumer” → Kinesis. “Multiple independent consumers” → MSK consumer groups OR Kinesis Enhanced Fan-Out. “Migrate from self-managed Kafka” → MSK (same code). |
| MLA-C01 | Kinesis for ML feature streaming (real-time feature ingestion to Feature Store). MSK for high-volume training data pipelines. Both feed Managed Flink for stream processing before ML. |
AWS Certification Exam Practice Questions
Question 1:
A company is migrating from self-managed Apache Kafka on EC2 to a managed service. They have 30+ microservices using Kafka producers/consumers with custom Kafka Connect connectors. They want to minimize code changes. Which service should they use?
- Amazon Kinesis Data Streams (rewrite producers to use AWS SDK)
- Amazon MSK (drop-in Kafka replacement, same client code)
- Amazon SQS FIFO (provides ordering like Kafka)
- Amazon EventBridge (replace Kafka topics with event bus rules)
Show Answer
Answer: B — MSK is managed Apache Kafka. Existing Kafka producers, consumers, and Connect connectors work with zero code changes — just point to new MSK bootstrap servers. Kinesis would require rewriting all 30+ services to use the AWS SDK. The “minimize code changes” requirement makes MSK the clear answer.
Question 2:
A startup needs real-time clickstream processing. They want the simplest, most serverless approach: data arrives → process in real-time → deliver to S3 in Parquet format. No Kafka expertise on the team. Which combination is simplest?
- MSK Serverless → Kafka Streams → S3 sink connector
- Kinesis Data Streams (on-demand) → Lambda → S3
- Kinesis Data Streams → Kinesis Data Firehose → S3 (Parquet conversion built-in)
- MSK → Managed Flink → S3
Show Answer
Answer: C — Kinesis Data Streams (on-demand = serverless, no shard management) → Firehose (auto-batches, converts to Parquet, delivers to S3). Zero servers, zero Kafka knowledge, built-in Parquet conversion. The entire pipeline is fully managed. MSK requires Kafka expertise the team doesn’t have.
Question 3:
A financial platform needs exactly-once processing guarantees for payment events. Events must be processed exactly one time, even if producers retry. Which streaming option provides this natively?
- Kinesis Data Streams with deduplication in Lambda consumer
- Amazon MSK with Kafka transactions (exactly-once semantics)
- SQS FIFO queue with deduplication ID
- EventBridge with archive ensuring no duplicates
Show Answer
Answer: B — Apache Kafka (MSK) provides native exactly-once semantics (EOS) through idempotent producers + transactional consumers. The Kafka protocol guarantees each record is written and consumed exactly once without application-level deduplication. Kinesis is at-least-once (consumer must handle deduplication). SQS FIFO has 5-minute dedup but isn’t a streaming platform.
Question 4:
A company streams 500 MB/s of log data from 1000 servers. They need 30 days retention and 5 independent consumer applications reading the same data. Which is MOST cost-effective?
- Kinesis with 500 shards + Enhanced Fan-Out for 5 consumers
- MSK Provisioned with Graviton brokers + tiered storage + 5 consumer groups
- Kinesis on-demand mode with 5 Lambda consumers
- 5 separate Kinesis streams (one per consumer)
Show Answer
Answer: B — At 500 MB/s, MSK with provisioned Graviton brokers is significantly more cost-effective than Kinesis (which would need 500+ shards at $0.015/hr each = $2,700/month just for shards). MSK consumer groups allow unlimited independent consumers at no additional per-consumer cost. Tiered storage provides 30-day retention cheaply by moving older data to S3-backed storage.
Question 5:
A team uses Kinesis Data Streams but finds that one consumer (analytics) is falling behind because the shared shard read limit (2MB/s) is consumed by another faster consumer (alerting). How should they fix this without increasing shards?
- Increase the consumer’s batch size
- Register the analytics consumer as an Enhanced Fan-Out (EFO) consumer
- Move the analytics consumer to a separate Kinesis stream
- Switch to MSK for consumer group isolation
Show Answer
Answer: B — Enhanced Fan-Out (EFO) provides each registered consumer a dedicated 2MB/s throughput per shard, independent of other consumers. The alerting consumer and analytics consumer no longer share read bandwidth. EFO uses push delivery (SubscribeToShard) instead of polling, also reducing latency to ~70ms.
Related Architecture Patterns
- SNS vs SQS vs EventBridge
- Event-Driven Serverless
- Data Lake & Analytics
- All Architecture Patterns – Hub
Related Posts
- Kinesis Streams vs Firehose vs Managed Flink
- SNS vs SQS vs EventBridge
- Event-Driven Serverless Architecture
- Data Lake & Analytics Architecture
References
- Best Practices for Kafka on AWS — AWS Blog
- Amazon MSK Serverless — AWS Blog
- Kinesis Enhanced Fan-Out — AWS Docs
- MSK Tiered Storage — AWS Blog
Frequently Asked Questions
Can I use Kinesis with existing Kafka consumers?
Not directly — Kinesis uses a proprietary API (AWS SDK / KCL), not the Kafka protocol. If you have existing Kafka consumers, MSK allows them to work with zero changes. To use Kinesis, you’d need to rewrite consumers using the KCL or Lambda event source mapping.
What is MSK Serverless vs MSK Provisioned?
MSK Serverless auto-scales partitions and throughput — no broker management, pay per data. MSK Provisioned gives you dedicated broker instances (you choose types, scaling is manual). Serverless is simpler; Provisioned is cheaper at high sustained throughput and gives more configuration control.
When should I use Managed Flink instead of Lambda for stream processing?
Use Lambda for simple, stateless per-record transformations (filter, enrich, route). Use Managed Flink for complex stateful processing: windowed aggregations, joins across streams, pattern detection, CEP (Complex Event Processing). Flink maintains state across records; Lambda processes each batch independently.