Managed Streaming for Apache Kafka – MSK

Managed Streaming for Apache Kafka – MSK is an AWS streaming data service that manages Apache Kafka infrastructure and operations.

Apache Kafka
- is an open-source, high-performance, fault-tolerant, and scalable streaming data store platform for building real-time streaming data pipelines and applications.
- stores streaming data in a fault-tolerant way, providing a buffer between producers and consumers.
- stores events as a continuous series of records and preserves the order in which the records were produced.
- runs as a cluster and stores data records in topics, which are partitioned and replicated across one or more brokers that can be spread across multiple AZs for high availability.
- allows many data producers and multiple consumers that can process data from Kafka topics on a first-in-first-out basis, preserving the order data were produced.
makes it easy for developers and DevOps managers to run Kafka applications and Kafka Connect connectors on AWS, without the need to become experts in operating Kafka.
operates, maintains, and scales Kafka clusters, provides enterprise-grade security features out of the box, and has built-in AWS integrations that accelerate development of streaming data applications.

always runs within a VPC managed by the MSK and is available to your own selected VPC, subnet, and security group when the cluster is setup.
IP addresses from the VPC are attached to the MSK resources through elastic network interfaces (ENIs), and all network traffic stays within the AWS network and is not accessible to the internet by default.
supports dual-stack connectivity (IPv4 and IPv6) for both MSK Provisioned and MSK Serverless clusters, enabling clients to connect using either protocol.

integrates with CloudWatch for monitoring, metrics, and logging.
offers two cluster types:
- MSK Provisioned – you select, manage, and manually scale cluster capacity (broker type, number, and storage).
- MSK Serverless – capacity is automatically provisioned and scaled without managing compute or storage.

supports Topic Management APIs (CreateTopic, UpdateTopic, DeleteTopic) allowing topic management via AWS CLI, SDKs, CloudFormation, and the console without requiring Kafka admin clients.

MSK Provisioned – Broker Types

MSK Provisioned offers two broker types: Standard brokers and Express brokers.

Standard Brokers

Standard brokers are the traditional MSK broker type using Apache Kafka’s native storage engine.

You must provision and manage EBS storage volumes attached to each broker.
Support all Apache Kafka versions available on MSK.
Available in 2-AZ and 3-AZ configurations.
Support Graviton3-based M7g instances, providing up to 29% more throughput and up to 24% lower costs compared to previous generation instances.

Express Brokers

Express brokers are a new broker type (GA at re:Invent 2024) designed for higher throughput, faster scaling, and simplified operations.

Key benefits include:
- No storage management – elastic, virtually unlimited, pay-as-you-go storage with no provisioning or sizing needed.
- Higher throughput – up to 3x more throughput per broker (e.g., 500 MBps ingress on m7g.16xlarge Express vs. ~154 MBps on equivalent Standard broker).
- Faster scaling – scale clusters and move partitions up to 20x faster than Standard brokers.
- Faster recovery – 90% faster recovery from broker failures compared to standard Apache Kafka brokers.
- No maintenance windows – Amazon MSK automatically updates cluster hardware on an ongoing basis.
- Pre-configured best practices – includes guardrails on critical Kafka configurations, throughput quotas, and capacity reservations for background operations.
- Intelligent Rebalancing – supports 180x faster partition rebalancing operations.
Express brokers are only available in 3-AZ configurations.
Support Apache Kafka versions 3.6, 3.8, and 3.9.
Support KRaft mode (ZooKeeper-free) from Apache Kafka version 3.9 onwards.
Work with Apache Kafka APIs but don’t yet fully support KStreams API.

Data is distributed across three AZs by default with replication factor of 3 and minimum in-sync replicas of 2.

MSK Serverless

MSK Serverless is a cluster type that helps run Kafka clusters without having to manage compute and storage capacity.
automatically provisions and scales capacity while managing the partitions in your topic, so you can stream data without thinking about right-sizing or scaling clusters.

offers throughput-based pricing, so you pay only for the data volume you stream and retain.
fully manages partitions, including monitoring and moving them to even load across a cluster.
creates 2 replicas for each partition and places them in different AZs. Additionally, MSK Serverless automatically detects and recovers failed backend resources to maintain high availability.

encrypts all traffic in transit and all data at rest using Key Management Service (KMS).
allows clients to connect over a private connection using AWS PrivateLink without exposing the traffic to the public internet.
offers IAM Access Control for both client authentication and authorization to Kafka resources such as topics.

supports dual-stack connectivity (IPv4 and IPv6).
is fully compatible with Apache Kafka, allowing use of any compatible client applications to produce and consume data.

MSK Tiered Storage

Tiered Storage offers a virtually unlimited, low-cost remote storage tier in addition to the high-performance local storage tier on MSK Provisioned clusters.
makes it cost-effective to retain data for longer durations (days, weeks, or months) in Apache Kafka without increasing local broker storage.

data is first stored in the performance-optimized primary (local) storage tier and is automatically tiered into the low-cost remote storage tier based on retention policies.
consumers can transparently read data from both tiers using the same Kafka APIs without application changes.
can save 50% or more on storage costs compared to provisioning all data on local broker storage.

can be enabled with a few clicks for new or existing clusters.
improves cluster resiliency by reducing local storage pressure and simplifying broker replacement.
supported on Apache Kafka version 3.6.0 and later.
Note: Express brokers include built-in elastic, virtually unlimited storage and do not require separate tiered storage configuration.

MSK Replicator

MSK Replicator is a fully managed feature that enables reliable replication of streaming data across Amazon MSK clusters in different or the same AWS Regions.
eliminates the need to write custom code, manage infrastructure, or set up cross-region networking for replication.
supports both active-active and active-passive cluster topologies for high availability and disaster recovery.

Key use cases:
- Business continuity and disaster recovery – replicate data to a secondary region; if the primary region is degraded, redirect applications to the secondary region.
- Multi-Region applications – serve global clients with reduced latency by having data copies in multiple geographies.
- Data aggregation – aggregate data from multiple clusters for centralized analytics.
- Partner data sharing – share streaming data with partners across accounts/regions.
- Migration – migrate from Standard brokers to Express brokers using MSK Replicator.

provides automatic asynchronous replication with topic data, consumer group offsets, and topic configurations.
both source and target clusters must be in the same AWS account.
available in most AWS Regions and expanding continuously.

MSK Connect

MSK Connect is a feature that makes it easy to stream data to and from Apache Kafka clusters using Kafka Connect connectors.

uses Kafka Connect versions 2.7.1 or 3.7.x open-source frameworks.
deploys fully managed connectors that move data into or pull data from popular data stores like Amazon S3, Amazon OpenSearch Service, and databases.
provisions the required resources, continuously monitors connector health and delivery state, patches underlying hardware, and auto-scales connectors to match throughput changes.
eliminates the need to provision and maintain Kafka Connect cluster infrastructure.

supports updating connector configurations without needing to delete and recreate connectors (Jan 2025).
supports worker configuration management – delete worker configurations, tag resources, and manage worker configurations and custom plugins using AWS CloudFormation for CI/CD automation (Mar 2024).
supports cross-account access to MSK clusters using IAM authentication.
supports dual-stack networking (IPv4 + IPv6) for connectors.
offers two capacity modes:
- Provisioned – you specify the number of workers.
- Auto-scaled – MSK Connect automatically adjusts the number of workers based on workload.

Multi-VPC Private Connectivity

Multi-VPC private connectivity is a managed solution that simplifies networking for multi-VPC and cross-account connectivity to MSK clusters.
powered by AWS PrivateLink, keeping all traffic within the AWS network.

allows Apache Kafka clients in different VPCs or different AWS accounts to privately connect to an MSK cluster without complex networking setup (VPC peering, transit gateway, etc.).
supports all authentication mechanisms: IAM Access Control, SASL/SCRAM, and TLS certificate authentication (Sep 2024 expansion).
the cluster owner can enable/disable private connectivity for one or more auth schemes.

the cluster must be in ACTIVE state to configure multi-VPC connectivity.

MSK Security

MSK uses EBS server-side encryption and KMS keys to encrypt storage volumes.
Clusters have encryption in transit enabled via TLS for inter-broker communication. For provisioned clusters, you can opt out of using encryption in transit when a cluster is created.

MSK clusters running Kafka version 2.5.1 or greater support TLS in-transit encryption between Kafka brokers and ZooKeeper nodes.
For provisioned clusters, you have three options:
- IAM Access Control for both AuthN/Z (recommended),
- TLS certificate authentication (CA) for AuthN and access control lists for AuthZ
- SASL/SCRAM for AuthN and access control lists for AuthZ.

MSK recommends using IAM Access Control as it defaults to least privilege access and is the most secure option.
For serverless clusters, IAM Access Control is used for both authentication and authorization.
Multi-VPC private connectivity (PrivateLink) enables secure cross-VPC and cross-account access while keeping traffic on the AWS network.

MSK Monitoring and Operations

MSK integrates with Amazon CloudWatch for cluster-level and broker-level metrics.
supports three monitoring levels: DEFAULT, PER_BROKER, and PER_TOPIC_PER_BROKER (increasing levels of granularity).
supports sending broker logs to CloudWatch Logs, Amazon S3, and Amazon Data Firehose.

supports Open Monitoring with Prometheus-compatible metrics endpoint for integration with third-party monitoring tools (Grafana, Datadog, etc.).
AI Agent Skills (Jun 2026) – Amazon MSK now offers AI Agent Skills to help developers operate MSK efficiently and accelerate migrations to MSK.
Topic Management APIs (Feb 2026) – CreateTopic, UpdateTopic, and DeleteTopic APIs enable topic management via AWS CLI, SDKs, AWS CloudFormation, and the MSK console without setting up Kafka admin clients.

AWS Certification Exam Practice Questions

Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).

AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.

AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated

Open to further feedback, discussion and correction.

A company needs a managed Apache Kafka service to ingest real-time clickstream data from its website. The data will be consumed by multiple downstream analytics applications. The solution must be highly available, support encryption at rest and in transit, and require minimal operational overhead. Which AWS service should the solutions architect recommend?
1. Amazon Kinesis Data Streams
2. Amazon MSK Provisioned with Standard brokers
3. Amazon MSK Serverless
4. Self-managed Apache Kafka on Amazon EC2
Answer: c. MSK Serverless provides a fully managed Kafka experience with automatic scaling, built-in encryption at rest and in transit, and minimal operational overhead.
A streaming data application requires consistently high throughput of 1.5 GBps ingress across 3 brokers and needs to scale rapidly during daily peak traffic periods. The team wants to minimize storage management overhead. Which MSK configuration best meets these requirements?
1. MSK Provisioned with Standard brokers and auto-scaling storage
2. MSK Provisioned with Express brokers
3. MSK Serverless
4. MSK Provisioned with Standard brokers and tiered storage
Answer: b. Express brokers provide up to 3x more throughput per broker (500 MBps per m7g.16xlarge), scale up to 20x faster, include elastic pay-as-you-go storage, and eliminate storage management overhead.
A company operates MSK clusters in us-east-1 and needs to implement a disaster recovery strategy with a secondary cluster in eu-west-1. The solution must automatically replicate topic data, consumer group offsets, and topic configurations with minimal custom code. Which approach should the company use?
1. Deploy MirrorMaker2 on Amazon EC2 instances across regions
2. Use Amazon MSK Replicator
3. Use AWS Database Migration Service (DMS)
4. Configure S3 Cross-Region Replication with MSK tiered storage
Answer: b. MSK Replicator provides fully managed, automatic asynchronous replication of topic data, consumer group offsets, and topic configurations across MSK clusters in different regions without custom code or infrastructure management.

A development team needs to stream data from an MSK cluster to Amazon OpenSearch Service for near-real-time search and analytics. The team wants a managed solution that auto-scales with throughput changes and does not require managing Kafka Connect infrastructure. Which solution meets these requirements?
1. Deploy Kafka Connect on Amazon ECS with auto-scaling
2. Use Amazon MSK Connect with an OpenSearch sink connector
3. Use Amazon Data Firehose with MSK as source
4. Write a custom AWS Lambda consumer function
Answer: b. MSK Connect provides fully managed Kafka Connect connectors with auto-scaling capability, eliminating the need to provision or manage Kafka Connect cluster infrastructure.
An organization has Apache Kafka clients in multiple VPCs across different AWS accounts that need to connect to a centralized MSK cluster. The traffic must stay within the AWS network and not traverse the public internet. Which MSK feature addresses this requirement?
1. MSK public access with TLS encryption
2. VPC peering between all accounts
3. Multi-VPC private connectivity (powered by AWS PrivateLink)
4. AWS Transit Gateway with MSK
Answer: c. Multi-VPC private connectivity uses AWS PrivateLink to provide managed cross-VPC and cross-account connectivity, keeping all traffic within the AWS network without complex networking setup.
A company wants to retain 90 days of streaming data in their MSK cluster for compliance purposes but wants to minimize storage costs. The majority of consumers only read data from the last 24 hours. Which MSK feature should they enable?
1. MSK Express brokers with elastic storage
2. MSK Tiered Storage
3. Amazon S3 archival via MSK Connect
4. Increase EBS volume size on Standard brokers
Answer: b. MSK Tiered Storage provides a low-cost remote storage tier for older data while keeping recent data on high-performance local storage, reducing storage costs by 50% or more while maintaining transparent consumer access.

Which authentication methods are supported by Amazon MSK Provisioned clusters? (Choose THREE)
1. IAM Access Control
2. SASL/SCRAM
3. TLS certificate authentication (mutual TLS)
4. Amazon Cognito User Pools
5. LDAP/Active Directory directly
Answer: a, b, c. MSK Provisioned supports IAM Access Control (recommended), SASL/SCRAM, and TLS certificate authentication (CA) for client authentication.
A data engineering team is migrating from Standard brokers to Express brokers in Amazon MSK to improve performance. Which statements about Express brokers are correct? (Choose THREE)
1. Express brokers are available in 2-AZ and 3-AZ configurations
2. Express brokers eliminate storage provisioning and management
3. Express brokers recover 90% faster from failures
4. Express brokers have no maintenance windows
5. Express brokers support all Apache Kafka versions available on MSK
Answer: b, c, d. Express brokers provide elastic storage (no provisioning), 90% faster recovery, and no maintenance windows. They are only available in 3-AZ configurations and support Kafka versions 3.6, 3.8, and 3.9.