allows real-time processing of streaming big data and the ability to read and replay records to multiple Amazon Kinesis Applications.
Amazon Kinesis Client Library (KCL) delivers all records for a given partition key to the same record processor, making it easier to build multiple applications that read from the same Amazon Kinesis stream (for example, to perform counting, aggregation, and filtering).
designed for high-volume, real-time data ingestion and processing with multiple concurrent consumers reading the same data.
offers a reliable, highly-scalable hosted queue for storing messages as they travel between applications or microservices.
It moves data between distributed application components and helps decouple these components.
provides common middleware constructs such as dead-letter queues, poison-pill management, and dead-letter queue redrive (including FIFO DLQ redrive).
provides a generic web services API and can be accessed by any programming language that the AWS SDK supports.
supports both standard and FIFO queues
supports Fair Queues (launched Jul 2025) for standard queues to mitigate noisy neighbor impact in multi-tenant systems.
Scaling
Kinesis Data Streams offers three capacity modes:
Provisioned mode – requires manual shard management and scaling.
On-demand Standard mode – automatically scales throughput (up to 200 MB/s write), eliminating the need for manual shard provisioning.
On-demand Advantage mode (launched Nov 2025) – provides warm throughput for instant scaling to handle traffic surges up to 10 GB/s, with 60%+ lower pricing compared to On-demand Standard for high-volume workloads.
SQS is fully managed, highly scalable and requires no administrative overhead and little configuration. It scales transparently to handle any volume of messages.
SQS FIFO queues support High Throughput mode with up to 70,000 transactions per second per API action (without batching), and up to 700,000 messages per second with batching in select regions.
Ordering
Kinesis provides ordering of records within a shard (by partition key), as well as the ability to read and/or replay records in the same order to multiple Kinesis Applications
SQS Standard Queue provides best-effort ordering but does not guarantee strict data ordering and provides at least once delivery of messages
SQS FIFO Queue guarantees strict data ordering within the message group
Data Retention Period
Kinesis Data Streams stores the data for up to 24 hours, by default, and can be extended to 365 days (with extended retention up to 7 days, and long-term retention from 7 to 365 days)
SQS stores the message for up to 4 days, by default, and can be configured from 1 minute to 14 days but clears the message once deleted by the consumer
Delivery Semantics
Kinesis and SQS Standard Queue both guarantee at least one delivery of the message.
SQS FIFO Queue guarantees Exactly once delivery (exactly-once processing via deduplication)
Parallel Clients
Kinesis supports multiple consumers reading from the same stream simultaneously
With shared throughput (GetRecords), all consumers share the 2 MB/s per shard read capacity
With Enhanced Fan-Out, each consumer gets a dedicated 2 MB/s per shard throughput via SubscribeToShard (push-based)
On-demand Advantage mode supports up to 50 enhanced fan-out consumers per stream (vs 20 on On-demand Standard or Provisioned)
SQS allows the messages to be delivered to only one consumer at a time and requires multiple queues to deliver messages to multiple consumers
SQS Fair Queues (Jul 2025) dynamically reorder message delivery to ensure fair processing across tenants when one tenant becomes a noisy neighbor
Message/Record Size
Kinesis supports a maximum record size of 1 MB per data record
SQS supports a maximum message payload size of 1 MiB (increased from 256 KiB in August 2025). For larger payloads, the Amazon SQS Extended Client Library can be used to store the payload in S3.
Throughput
Kinesis Data Streams
Provisioned: 1 MB/s write and 2 MB/s read per shard
On-demand Standard: automatically scales, default 4 MB/s write, can burst to 200 MB/s
On-demand Advantage: supports instant scaling with warm throughput up to 10 GB/s
SQS
Standard queues: nearly unlimited throughput (no per-queue limits)
FIFO queues: 300 TPS default, up to 70,000 TPS with High Throughput mode enabled
Integration with AWS Lambda
Kinesis integrates with Lambda via event source mapping with parallelization factor, tumbling windows, and failure handling (bisect on error, max retry)
SQS integrates with Lambda via event source mapping with support for batch windows and Provisioned Mode (Nov 2025) that provides 3x faster scaling (up to 1,000 concurrent executions per minute) and 16x higher concurrency (up to 20,000 concurrent executions)
Use Cases
Kinesis use cases requirements
Ordering of records.
Ability to consume records in the same order a few hours later (data replay)
Ability for multiple applications to consume the same stream concurrently
Routing related records to the same record processor (as in streaming MapReduce)
Real-time analytics, log and event data aggregation, IoT telemetry ingestion
SQS uses cases requirements
Messaging semantics like message-level ack/fail and visibility timeout
Leveraging SQS’s ability to scale transparently
Dynamically increasing concurrency/throughput at read time
Decoupling microservices and serverless event-driven architectures
Kinesis Data Streams vs SQS Comparison Table
Feature
Kinesis Data Streams
SQS
Primary Use Case
Real-time streaming data processing
Message queuing and decoupling
Ordering
Per-shard (partition key)
Best-effort (Standard) / Strict per message group (FIFO)
Delivery
At least once
At least once (Standard) / Exactly once (FIFO)
Retention
24 hours to 365 days
1 minute to 14 days
Multiple Consumers
Yes (shared or enhanced fan-out)
No (single consumer per message)
Max Record/Message Size
1 MB
1 MiB
Scaling
Provisioned (manual) / On-demand (automatic)
Fully automatic
Data Replay
Yes
No (message deleted after processing)
Provisioning
Three modes: Provisioned, On-demand Standard, On-demand Advantage
Fully serverless, no provisioning
Fair Processing
N/A
Fair Queues for multi-tenant workloads
AWS Certification Exam Practice Questions
Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
Open to further feedback, discussion and correction.
You are deploying an application to track GPS coordinates of delivery trucks in the United States. Coordinates are transmitted from each delivery truck once every three seconds. You need to design an architecture that will enable real-time processing of these coordinates from multiple consumers. Which service should you use to implement data ingestion?
Amazon Kinesis
AWS Data Pipeline
Amazon AppStream
Amazon Simple Queue Service
Your customer is willing to consolidate their log streams (access logs, application logs, security logs etc.) in one single system. Once consolidated, the customer wants to analyze these logs in real time based on heuristics. From time to time, the customer needs to validate heuristics, which requires going back to data samples extracted from the last 12 hours? What is the best approach to meet your customer’s requirements?
Send all the log events to Amazon SQS. Setup an Auto Scaling group of EC2 servers to consume the logs and apply the heuristics.
Send all the log events to Amazon Kinesis develop a client process to apply heuristics on the logs (Can perform real time analysis and stores data for 24 hours which can be extended to 365 days)
Configure Amazon CloudTrail to receive custom logs, use EMR to apply heuristics the logs (CloudTrail is only for auditing)
Setup an Auto Scaling group of EC2 syslogd servers, store the logs on S3 use EMR to apply heuristics on the logs (EMR is for batch analysis)
A company runs a multi-tenant SaaS application where different customers submit varying volumes of jobs to an SQS queue. During peak hours, one large customer floods the queue with messages, causing increased dwell time for all other customers. Which SQS feature should the team enable to address this noisy neighbor problem?
SQS FIFO High Throughput mode
SQS Long Polling
SQS Fair Queues (Fair Queues dynamically reorder message delivery to mitigate noisy neighbor impact in multi-tenant standard queues)
SQS Dead-Letter Queue Redrive
A streaming analytics application needs to process real-time clickstream data with five independent consumer applications reading from the same stream simultaneously, each requiring dedicated throughput. The team wants to minimize operational overhead. Which configuration is most appropriate?
Amazon SQS with five separate queues using SNS fan-out
Kinesis Data Streams in Provisioned mode with Enhanced Fan-Out
Kinesis Data Streams in On-demand Advantage mode with Enhanced Fan-Out (On-demand Advantage provides automatic scaling with no shard management and supports up to 50 enhanced fan-out consumers with dedicated 2 MB/s per shard throughput)
Kinesis Data Streams in On-demand Standard mode with shared GetRecords
An application processes order events that are each approximately 500 KB in size. The events need to be placed in a queue for asynchronous processing by a Lambda function. Which approach meets the requirements with the LEAST operational overhead?
Use SQS with the Extended Client Library to store messages in S3
Use SQS directly, as it now supports message payloads up to 1 MiB (Since August 2025, SQS supports up to 1 MiB message payload natively, eliminating the need for S3 offloading for messages under 1 MiB)
Amazon Kinesis Data Streams is a streaming data service that enables real-time processing of streaming data at a massive scale.
Kinesis Streams enables building of custom applications that process or analyze streaming data for specialized needs.
Kinesis Streams features
handles provisioning, deployment, ongoing-maintenance of hardware, software, or other services for the data streams.
manages the infrastructure, storage, networking, and configuration needed to stream the data at the level of required data throughput.
synchronously replicates data across three AZs in an AWS Region, providing high availability and data durability.
stores records of a stream for up to 24 hours, by default, from the time they are added to the stream. The limit can be raised to up to 365 days by enabling extended data retention.
Data such as clickstreams, application logs, social media, etc can be added from multiple sources and within seconds is available for processing to the Kinesis Applications.
Kinesis provides the ordering of records, as well as the ability to read and/or replay records in the same order to multiple applications.
Kinesis is designed to process streaming big data and the pricing model allows heavy PUTs rate.
Multiple Kinesis Data Streams applications can consume data from a stream, so that multiple actions, like archiving and processing, can take place concurrently and independently.
Kinesis Data Streams application can start consuming the data from the stream almost immediately after the data is added and put-to-get delay is typically less than 1 second.
Kinesis Streams is useful for rapidly moving data off data producers and then continuously processing the data, be it to transform the data before emitting to a data store, run real-time metrics and analytics, or derive more complex data streams for further processing
Accelerated log and data feed intake: Data producers can push data to Kinesis stream as soon as it is produced, preventing any data loss and making it available for processing within seconds.
Real-time metrics and reporting: Metrics can be extracted and used to generate reports from data in real-time.
Real-time data analytics: Run real-time streaming data analytics.
Complex stream processing: Create Directed Acyclic Graphs (DAGs) of Kinesis Applications and data streams, with Kinesis applications adding to another Amazon Kinesis stream for further processing, enabling successive stages of stream processing.
Kinesis limits
stores records of a stream for up to 24 hours, by default, which can be extended to max 365 days
maximum size of a data blob (the data payload before Base64-encoding) within one record is 1 MB by default, configurable up to 10 MB
Each shard can support up to 1000 PUT records per second.
S3 is a cost-effective way to store the data, but not designed to handle a stream of data in real-time
Kinesis Data Streams Terminology
Data Record
A record is the unit of data stored in a Kinesis data stream.
A record is composed of a sequence number, partition key, and data blob, which is an immutable sequence of bytes.
Maximum size of a data blob is 1 MB by default, configurable up to 10 MB (announced Oct 2025).
Partition key
Partition key is used to segregate and route records to different shards of a stream.
A partition key is specified by the data producer while adding data to a Kinesis stream.
Sequence number
A sequence number is a unique identifier for each record.
Kinesis assigns a Sequence number, when a data producer calls PutRecord or PutRecords operation to add data to a stream.
Sequence numbers for the same partition key generally increase over time; the longer the time period between PutRecord or PutRecords requests, the larger the sequence numbers become.
Data Stream
Data stream represents a group of data records.
Data records in a data stream are distributed into shards.
Shard
A shard is a uniquely identified sequence of data records in a stream.
Streams are made of shards and are the base throughput unit of a Kinesis stream, as pricing is per shard basis.
Each shard supports up to 5 transactions per second for reads, up to a maximum total data read rate of 2 MB per second, and up to 1,000 records per second for writes, up to a maximum total data write rate of 1 MB per second (including partition keys)
Each shard provides a fixed unit of capacity. If the limits are exceeded, either by data throughput or the number of PUT records, the put data call will be rejected with a ProvisionedThroughputExceeded exception.
This can be handled by
Implementing a retry on the data producer side, if this is due to a temporary rise of the stream’s input data rate
Dynamically scaling the number of shared (resharding) to provide enough capacity for the put data calls to consistently succeed
Capacity Mode
A data stream capacity mode determines the pricing and how the capacity is managed
Kinesis Data Streams currently supports three capacity modes:
On-demand Standard
KDS automatically manages the shards in order to provide the necessary throughput.
You are charged only for the actual throughput used and KDS automatically accommodates the workloads’ throughput needs as they ramp up or down.
On-demand Advantage (launched Nov 2025)
Enables on-demand streams to handle instant throughput increases up to 10GB/s or 10 million events per second using warm throughput capability.
Offers 60% lower data pricing compared to On-demand Standard (ingest at $0.032/GB, retrieval at $0.016/GB).
Supports up to 50 enhanced fan-out consumers per stream (vs 20 for Standard/Provisioned).
Best suited for use cases ingesting at least 10 MiB/s in aggregate or having hundreds of data streams in a Region.
Provisioned mode
Number of shards for the data stream must be specified.
Total capacity of a data stream is the sum of the capacities of its shards.
Shards can be increased or decreased in a data stream as needed and you are charged for the number of shards at an hourly rate.
Retention Period
All data is stored for 24 hours, by default, and can be increased to 8760 hours (365 days) maximum.
Producers
A producer puts data records into Kinesis data streams.
To put data into the stream, the name of the stream, a partition key, and the data blob to be added to the stream should be specified.
Partition key is used to determine which shard in the stream the data record is added to.
Consumers
A consumer is an application built to read and process data records from Kinesis data streams.
Kinesis Data Streams Large Record Support
Kinesis Data Streams supports record sizes up to 10 MiB – a tenfold increase from the previous 1 MiB limit (announced Oct 2025).
Maximum PutRecords request size increased from 5 MiB to 10 MiB.
Large record support can be configured per stream by updating the maxRecordSize setting via Console, CLI, or SDK.
Individual shard throughput limits (1 MiB/s write, 2 MiB/s read) remain unchanged; large records use burst capacity (up to 10 MiB/s per shard) that refills at the standard rate.
Best practices:
Maintain large records at 1-2% of total record count for optimal performance.
Use random/uniformly distributed partition keys to spread large records across shards.
Implement backoff and retry logic in producer applications.
For continuous large record streams, store payloads in S3 and send metadata references through the stream.
Supported by KCL 2.x+, Amazon Data Firehose (S3 delivery), and AWS Lambda consumers.
supports encryption in transit using HTTPS endpoints.
supports Interface VPC endpoint to keep traffic between VPC and Kinesis Data Streams from leaving the Amazon network. Interface VPC endpoints don’t require an IGW, NAT device, VPN connection, or Direct Connect.
integrated with IAM to control access to Kinesis Data Streams resources.
supports Attribute-Based Access Control (ABAC) using stream tags and consumer tags, enabling fine-grained access control based on tag key-value pairs attached to IAM users, roles, and resources (Sep 2024 for streams, Apr 2025 for consumers).
supports resource-based policies for cross-account access, enabling processing data ingested in one account with consumers (e.g., Lambda) in another account without assuming IAM roles.
integrated with CloudTrail, which provides a record of actions taken by a user, role, or an AWS service in Kinesis Data Streams.
supports AWS Fault Injection Service (FIS) integration (Oct 2025) to test application resilience against common API errors (e.g., ProvisionedThroughputExceeded, expired iterators) in a controlled environment.
PutRecord & PutRecords operations are synchronous operation that sends single/multiple records to the stream per HTTP request.
use PutRecords to achieve a higher throughput per data producer
helps manage many aspects of Kinesis Data Streams (including creating streams, resharding, and putting and getting records)
Kinesis Agent
is a pre-built Java application that offers an easy way to collect and send data to the Kinesis stream.
can be installed on Linux-based server environments such as web servers, log servers, and database servers
can be configured to monitor certain files on the disk and then continuously send new data to the Kinesis stream
Kinesis Producer Library (KPL)
is an easy-to-use and highly configurable library that helps to put data into a Kinesis stream.
provides a layer of abstraction specifically for ingesting data
presents a simple, asynchronous, and reliable interface that helps achieve high producer throughput with minimal client resources.
batches messages, as it aggregates records to increase payload size and improve throughput.
Collects records and uses PutRecords to write multiple records to multiple shards per request
Writes to one or more Kinesis data streams with an automatic and configurable retry mechanism.
Integrates seamlessly with the Kinesis Client Library (KCL) to de-aggregate batched records on the consumer
Submits CloudWatch metrics to provide visibility into performance
⚠️ KPL 0.x reached end-of-support on January 30, 2026. Migrate to KPL 1.x. The upgrade requires no changes to data processing logic.
Third Party and Open source
Log4j appender
Apache Kafka
Flume, fluentd, etc.
Kinesis Consumers
Kinesis Application is a data consumer that reads and processes data from a Kinesis Data Stream and can be built using either Kinesis API or Kinesis Client Library (KCL)
Shards in a stream provide 2 MB/sec of read throughput per shard, by default, which is shared by all the consumers reading from a given shard.
Kinesis Client Library (KCL)
is a pre-built library with multiple language support
delivers all records for a given partition key to same record processor
makes it easier to build multiple applications reading from the same stream for e.g. to perform counting, aggregation, and filtering
handles complex issues such as adapting to changes in stream volume, load-balancing streaming data, coordinating distributed services, and processing data with fault-tolerance
uses a unique DynamoDB table to keep track of the application’s state, so if the Kinesis Data Streams application receives provisioned-throughput exceptions, increase the provisioned throughput for the DynamoDB table
KCL 3.0 (launched Nov 2024) introduces an enhanced load balancing algorithm that continuously monitors resource utilization and redistributes load from over-utilized workers to underutilized workers, reducing stream processing compute costs by up to 33%.
⚠️ KCL 1.x reached end-of-support on January 30, 2026. Migrate to KCL 3.x. Migration requires updating interfaces and security credential providers.
AWS Lambda, Amazon Data Firehose, and Amazon Managed Service for Apache Flink also act as consumers for Kinesis Data Streams
Kinesis Enhanced Fan-Out
allows customers to scale the number of consumers reading from a data stream in parallel, while maintaining high performance and without contending for read throughput with other consumers.
provides logical 2 MB/sec throughput pipes between consumers and shards for Kinesis Data Streams Consumers.
supports up to 20 enhanced fan-out consumers per stream with On-demand Standard and Provisioned modes.
supports up to 50 enhanced fan-out consumers per stream with On-demand Advantage mode (Nov 2025).
supports consumer tagging and ABAC for fine-grained access control and cost allocation (Apr 2025).
Kinesis Data Streams Sharding
Resharding helps to increase or decrease the number of shards in a stream in order to adapt to changes in the rate of data flowing through the stream.
Resharding operations support shard split and shard merge.
Shard split helps divide a single shard into two shards. It increases the capacity and the cost.
Shard merge helps combine two shards into a single shard. It reduces the capacity and the cost.
Resharding is always pairwise and always involves two shards.
The shard or pair of shards that the resharding operation acts on are referred to as parent shards. The shard or pair of shards that result from the resharding operation are referred to as child shards.
Kinesis Client Library tracks the shards in the stream using a DynamoDB table and discovers the new shards and populates new rows in the table.
KCL ensures that any data that existed in shards prior to the resharding is processed before the data from the new shards, thereby, preserving the order in which data records were added to the stream for a particular partition key.
Data records in the parent shard are accessible from the time they are added to the stream to the current retention period.
⚠️ Amazon Kinesis Data Analytics for SQL was discontinued on January 27, 2026.
AWS deleted all existing SQL applications. Customers should migrate to:
Amazon Managed Service for Apache Flink – for complex stream processing
Amazon Managed Service for Apache Flink Studio – for interactive SQL-based analytics
AWS Certification Exam Practice Questions
Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
Open to further feedback, discussion and correction.
Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
Open to further feedback, discussion and correction.
You are deploying an application to track GPS coordinates of delivery trucks in the United States. Coordinates are transmitted from each delivery truck once every three seconds. You need to design an architecture that will enable real-time processing of these coordinates from multiple consumers. Which service should you use to implement data ingestion?
Amazon Kinesis
AWS Data Pipeline
Amazon AppStream
Amazon Simple Queue Service
You are deploying an application to collect votes for a very popular television show. Millions of users will submit votes using mobile devices. The votes must be collected into a durable, scalable, and highly available data store for real-time public tabulation. Which service should you use?
Amazon DynamoDB
Amazon Redshift
Amazon Kinesis
Amazon Simple Queue Service
Your company is in the process of developing a next generation pet collar that collects biometric information to assist families with promoting healthy lifestyles for their pets. Each collar will push 30kb of biometric data In JSON format every 2 seconds to a collection platform that will process and analyze the data providing health trending information back to the pet owners and veterinarians via a web portal Management has tasked you to architect the collection platform ensuring the following requirements are met. Provide the ability for real-time analytics of the inbound biometric data Ensure processing of the biometric data is highly durable, elastic and parallel. The results of the analytic processing should be persisted for data mining. Which architecture outlined below will meet the initial requirements for the collection platform?
Utilize S3 to collect the inbound sensor data analyze the data from S3 with a daily scheduled Data Pipeline and save the results to a Redshift Cluster.
Utilize Amazon Kinesis to collect the inbound sensor data, analyze the data with Kinesis clients and save the results to a Redshift cluster using EMR. (refer link)
Utilize SQS to collect the inbound sensor data analyze the data from SQS with Amazon Kinesis and save the results to a Microsoft SQL Server RDS instance.
Utilize EMR to collect the inbound sensor data, analyze the data from EUR with Amazon Kinesis and save me results to DynamoDB.
Your customer is willing to consolidate their log streams (access logs, application logs, security logs etc.) in one single system. Once consolidated, the customer wants to analyze these logs in real time based on heuristics. From time to time, the customer needs to validate heuristics, which requires going back to data samples extracted from the last 12 hours? What is the best approach to meet your customer’s requirements?
Send all the log events to Amazon SQS. Setup an Auto Scaling group of EC2 servers to consume the logs and apply the heuristics.
Send all the log events to Amazon Kinesis develop a client process to apply heuristics on the logs (Can perform real time analysis and stores data for 24 hours which can be extended to 365 days)
Configure Amazon CloudTrail to receive custom logs, use EMR to apply heuristics the logs (CloudTrail is only for auditing)
Setup an Auto Scaling group of EC2 syslogd servers, store the logs on S3 use EMR to apply heuristics on the logs (EMR is for batch analysis)
You require the ability to analyze a customer’s clickstream data on a website so they can do behavioral analysis. Your customer needs to know what sequence of pages and ads their customer clicked on. This data will be used in real time to modify the page layouts as customers click through the site to increase stickiness and advertising click-through. Which option meets the requirements for captioning and analyzing this data?
Log clicks in weblogs by URL store to Amazon S3, and then analyze with Elastic MapReduce
Push web clicks by session to Amazon Kinesis and analyze behavior using Kinesis workers
Write click events directly to Amazon Redshift and then analyze with SQL
Publish web clicks by session to an Amazon SQS queue men periodically drain these events to Amazon RDS and analyze with SQL
Your social media monitoring application uses a Python app running on AWS Elastic Beanstalk to inject tweets, Facebook updates and RSS feeds into an Amazon Kinesis stream. A second AWS Elastic Beanstalk app generates key performance indicators into an Amazon DynamoDB table and powers a dashboard application. What is the most efficient option to prevent any data loss for this application?
Use AWS Data Pipeline to replicate your DynamoDB tables into another region.
Use the second AWS Elastic Beanstalk app to store a backup of Kinesis data onto Amazon Elastic Block Store (EBS), and then create snapshots from your Amazon EBS volumes.
Add a second Amazon Kinesis stream in another Availability Zone and use AWS data pipeline to replicate data across Kinesis streams.
Add a third AWS Elastic Beanstalk app that uses the Amazon Kinesis S3 connector to archive data from Amazon Kinesis into Amazon S3.
You need to replicate API calls across two systems in real time. What tool should you use as a buffer and transport mechanism for API call events?
AWS SQS
AWS Lambda
AWS Kinesis (AWS Kinesis is an event stream service. Streams can act as buffers and transport across systems for in-order programmatic events, making it ideal for replicating API calls across systems)
AWS SNS
You need to perform ad-hoc business analytics queries on well-structured data. Data comes in constantly at a high velocity. Your business intelligence team can understand SQL. What AWS service(s) should you look to first?
Kinesis Firehose + RDS
Kinesis Firehose + RedShift (Kinesis Firehose provides a managed service for aggregating streaming data and inserting it into RedShift. RedShift also supports ad-hoc queries over well-structured data using a SQL-compliant wire protocol, so the business team should be able to adopt this system easily. Refer link)
EMR using Hive
EMR running Apache Spark
A company streams IoT sensor data into Kinesis Data Streams using provisioned mode. They have 10 consumers reading from the same stream and experience latency issues. Which solution provides dedicated throughput to each consumer without increasing shards?
Switch to On-demand Standard mode
Use Kinesis Data Firehose instead
Enable Enhanced Fan-Out for each consumer (Enhanced fan-out provides a dedicated 2 MB/sec pipe per consumer per shard, eliminating shared throughput contention)
Increase the number of shards
A development team needs to process records that are occasionally 5 MB in size from their Kinesis Data Stream. What must they do to support this? (Choose 2)
Update the stream’s maxRecordSize setting to allow records up to 5 MiB or larger
Split records into smaller chunks before ingestion
Ensure downstream consumers (KCL 2.x+, Lambda, or Firehose) can handle larger records
Upgrade to a provisioned mode stream with more shards
Store the records in S3 and send only metadata to Kinesis
📢 Service Rename (February 2024): Amazon Kinesis Data Firehose has been renamed to Amazon Data Firehose. The functionality remains the same. This post uses both names for clarity.
Kinesis acts as a highly available conduit to stream messages between data producers and data consumers.
Data producers can be almost any source of data: system or web log data, social network data, financial trading information, geospatial data, mobile app data, or telemetry from connected IoT devices.
Data consumers will typically fall into the category of data processing and storage applications such as Apache Hadoop, Apache Storm, S3, and OpenSearch.
Purpose
Kinesis Data Streams is highly customizable and best suited for developers building custom applications or streaming data for specialized needs.
Amazon Data Firehose (formerly Kinesis Data Firehose) handles loading data streams directly into AWS products for processing. Firehose allows streaming to S3, OpenSearch Service, Redshift, Apache Iceberg tables, Amazon S3 Tables, Snowflake, and other destinations, where data can be copied for processing through additional services.
Provisioning & Scaling
Kinesis Data Streams offers three capacity modes:
Provisioned Mode: Requires manual configuration of shards and scaling. You specify the number of shards needed based on expected throughput. Each shard provides 1 MB/s write (1000 records/s) and 2 MB/s read.
On-Demand Standard Mode (Launched November 2021): Automatically scales to handle gigabytes of write and read throughput per minute without manual shard management. Default capacity of 4 MB/s write (4000 records/s), can scale up to 200 MB/s (or 1 GB/s with limit increase).
On-Demand Advantage Mode (Launched November 2025): Enables warm throughput capability for instant scaling up to 10 GB/s or 10 million events/second. Offers 60% lower pricing compared to On-demand Standard ($0.032/GB ingest, $0.016/GB retrieval). Removes per-stream fixed charge. Supports up to 50 enhanced fan-out consumers (vs. 20 for other modes). Best for workloads ingesting at least 10 MB/s aggregate, high fan-out use cases, or accounts with hundreds of streams.
Amazon Data Firehose is fully managed and sends data to S3, Redshift, OpenSearch, Apache Iceberg tables, Amazon S3 Tables, Snowflake, and other destinations. Scaling is handled automatically, up to gigabytes per second, and allows for batching, encrypting, and compressing.
Processing Delay
Kinesis Data Streams provides real-time processing with ~200 ms for shared throughput classic single consumer and ~70 ms for the enhanced fan-out consumer.
Amazon Data Firehose provides near real-time processing:
Standard Buffering: Minimum buffer time of 60 seconds (1 min), maximum 900 seconds (15 min)
Zero Buffering (Announced December 2023): Delivers data within ~5 seconds with no buffering delay, enabling real-time use cases
Record Size
Kinesis Data Streams supports record sizes up to 10 MiB (increased from 1 MiB in October 2025), enabling larger data payloads like images, documents, and complex event data without splitting records.
Amazon Data Firehose supports record sizes up to 1 MiB per record.
Data Storage
Kinesis Data Streams provides data storage. Data typically is made available in a stream for 24 hours, but for an additional cost, users can gain data availability for up to 365 days (8760 hours). On-demand Advantage mode offers 77% lower extended retention pricing ($0.023/GB-month vs $0.10/GB-month).
Amazon Data Firehose does not provide data storage.
Replay
Kinesis Data Streams supports replay capability
Amazon Data Firehose does not support replay capability
Producers & Consumers
Kinesis Data Streams & Amazon Data Firehose support multiple producer options including SDK, KPL, Kinesis Agent, IoT, etc.
Kinesis Data Streams supports multiple consumer options including SDK, KCL, and Lambda, and can write data to multiple destinations. However, they have to be coded.
Supports up to 50 enhanced fan-out consumers per stream with On-demand Advantage mode (up from 20 with On-demand Standard/Provisioned modes) — Launched November 2025
Enhanced fan-out provides dedicated 2 MB/s throughput per consumer per shard with ~70 ms latency
Amazon Data Firehose consumers are close-ended and support destinations including:
Amazon S3
Amazon Redshift
Amazon OpenSearch Service
Amazon OpenSearch Serverless
Apache Iceberg Tables (Added October 2024) — Stream data directly into Iceberg format tables in S3
Amazon S3 Tables (GA March 2025) — Stream data into S3 Tables with built-in Apache Iceberg support and automatic table maintenance (compaction, snapshot management)
Snowflake (with Snowpipe Streaming) — Real-time streaming to Snowflake
Amazon Data Firehose also supports database CDC replication (Preview, November 2024) — captures change data from MySQL and PostgreSQL databases and replicates directly to Apache Iceberg tables in S3, enabling near real-time data lake updates without custom ETL code.
Key Differences Summary
Feature
Kinesis Data Streams
Amazon Data Firehose
Capacity Mode
Provisioned, On-Demand Standard, or On-Demand Advantage
Fully managed (automatic)
Latency
Real-time (~70-200 ms)
Near real-time (60s default, ~5s with zero buffering)
Max Record Size
10 MiB (since Oct 2025)
1 MiB
Data Retention
24 hours to 365 days
No storage
Replay
✅ Supported
❌ Not supported
Max Throughput
Up to 10 GB/s (On-demand Advantage with warm throughput)
⚠️ Important: Kinesis Client Library (KCL) 1.x reached end-of-support on January 30, 2026. AWS strongly recommends migrating KCL applications to KCL 2.x or later. KCL 1.x entered maintenance mode on April 17, 2025 with only critical bug fixes and security updates.
Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
Open to further feedback, discussion and correction.
Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
Open to further feedback, discussion and correction.
Your organization needs to ingest a big data stream into its data lake on Amazon S3. The data may stream in at a rate of hundreds of megabytes per second. What AWS service will accomplish the goal with the least amount of management?
Amazon Data Firehose
Amazon Kinesis Data Streams
Amazon CloudFront
Amazon SQS
Your organization is looking for a solution that can help the business with streaming data several services will require access to read and process the same stream concurrently. What AWS service meets the business requirements?
Amazon Data Firehose
Amazon Kinesis Data Streams
Amazon CloudFront
Amazon SQS
Your application generates a 1 KB JSON payload that needs to be queued and delivered to EC2 instances for applications. At the end of the day, the application needs to replay the data for the past 24 hours. In the near future, you also need the ability for other multiple EC2 applications to consume the same stream concurrently. What is the best solution for this?
Kinesis Data Streams
Amazon Data Firehose
SNS
SQS
A company needs to stream data to Amazon S3 with the lowest possible latency (under 10 seconds). Which Kinesis service and configuration should they use? (Assume December 2023 or later)
Kinesis Data Streams with Lambda consumer
Amazon Data Firehose with zero buffering enabled
Amazon Data Firehose with 60-second buffer
Kinesis Data Streams with KCL consumer
A company wants to avoid manual shard management for their Kinesis Data Streams and needs to handle instant traffic surges up to 10 GB/s. Which capacity mode should they use?
Provisioned mode with Auto Scaling
On-Demand Standard mode
On-Demand Advantage mode with warm throughput
Enhanced fan-out mode
A data analytics team needs to stream real-time data into Apache Iceberg tables in S3 for analytics. Which AWS service supports this natively? (Assume October 2024 or later)
Kinesis Data Streams
Amazon Data Firehose
AWS Glue Streaming
Amazon MSK
A company streams data using Kinesis Data Streams with 30 independent consumer applications needing dedicated throughput. Which configuration supports this?
On-Demand Standard mode with enhanced fan-out
Provisioned mode with enhanced fan-out
On-Demand Advantage mode with enhanced fan-out
Create separate streams for each consumer
A company needs to stream IoT sensor data that occasionally includes 5 MiB payloads. Which streaming service supports this record size natively? (Assume October 2025 or later)
Amazon Data Firehose
Kinesis Data Streams
Amazon SQS
Amazon SNS
A company wants to stream real-time data into Amazon S3 Tables with built-in Apache Iceberg support and automatic table maintenance. Which service should they use?
Amazon Data Firehose (formerly Kinesis Data Firehose)
📢 Service Renamed (February 2024): Amazon Kinesis Data Firehose has been renamed to Amazon Data Firehose. The functionality remains the same. Existing applications, API endpoints, and IAM policies continue to work without changes.
Amazon Data Firehose is a fully managed service for delivering real-time streaming data to data stores and analytics tools.
Amazon Data Firehose automatically scales to match the throughput of the data (gigabytes per second or more) and requires no ongoing administration or need to write applications or manage resources.
is a data transfer solution for delivering real-time streaming data to destinations such as S3, Redshift, OpenSearch Service, OpenSearch Serverless, Apache Iceberg Tables, Amazon S3 Tables, Snowflake, Splunk, and third-party HTTP endpoints.
is NOT Real Time, but Near Real Time as it supports batching and buffers streaming data to a certain size (Buffer Size in MBs) or for a certain period of time (Buffer Interval in seconds) before delivering it to destinations.
Zero Buffering (December 2023): Firehose now supports zero buffering, delivering data within ~5 seconds with no buffering delay for real-time use cases.
supports data compression, minimizing the amount of storage used at the destination. It currently supports GZIP, ZIP, and SNAPPY compression formats. Only GZIP is supported if the data is further loaded to Redshift.
supports Apache Parquet and ORC format conversion — converts incoming JSON data to columnar formats optimized for analytics with Athena, Redshift Spectrum, and EMR before storing in S3.
supports data at rest encryption using KMS after the data is delivered to the S3 bucket.
supports 20+ data sources including Amazon Kinesis Data Streams, Amazon MSK (and MSK Serverless), Direct PUT API, Kinesis Agent, CloudWatch Logs, CloudWatch Events, AWS IoT Core, Amazon SNS, AWS WAF web ACL logs, and Amazon VPC Flow Logs.
supports out of box data transformation as well as custom transformation using the Lambda function to transform incoming source data and deliver the transformed data to destinations.
supports Dynamic Partitioning — groups streaming data by static or dynamically defined keys (e.g., customer_id, region) and delivers into key-unique S3 prefixes for optimized analytics.
supports source record backup with custom data transformation with Lambda, where Data Firehose will deliver the un-transformed incoming data to a separate S3 bucket.
uses at least once semantics for data delivery. In rare circumstances such as request timeout upon data delivery attempt, delivery retry by Firehose could introduce duplicates if the previous request eventually goes through.
Underlying entity of Data Firehose, where the data is sent
Record
Data sent by data producer to a Data Firehose delivery stream
Maximum size of a record (before Base64-encoding) is 1024 KB.
With Amazon MSK as source, maximum record size is 10 MB (6 MB if Lambda transformation is enabled).
Data producer
Producers send records to Data Firehose delivery streams.
Buffer size and buffer interval
Data Firehose buffers incoming streaming data to a certain size or for a certain time period before delivering it to destinations
Buffer size and buffer interval can be configured while creating the delivery stream
Buffer size is in MBs and ranges from 1MB to 128MB for the S3 destination and 1MB to 100MB for the OpenSearch Service destination.
Buffer interval is in seconds and ranges from 0 secs (zero buffering) to 900 secs
Zero Buffering (December 2023): Set buffer interval to 0 seconds to deliver data within ~5 seconds with no buffering delay
Firehose raises buffer size dynamically to catch up and make sure that all data is delivered to the destination, if data delivery to the destination is falling behind data writing to the delivery stream
Buffer size is applied before compression.
Source
Data Firehose supports 20+ data sources:
Amazon Kinesis Data Streams — read directly from a KDS stream
Amazon MSK / MSK Serverless — consume from Kafka topics
Direct PUT — via Firehose API, SDK, or Kinesis Agent
A destination is the data store where the data will be delivered.
supports the following destinations:
Amazon S3 — with optional dynamic partitioning and format conversion
Amazon Redshift — via intermediate S3 COPY
Amazon OpenSearch Service
Amazon OpenSearch Serverless (added November 2022)
Apache Iceberg Tables in S3 (GA September 2024) — stream into Iceberg format tables with ACID transactions
Amazon S3 Tables (GA March 2025) — purpose-built managed Iceberg tables with automatic optimization
Snowflake — real-time streaming via Snowpipe Streaming
Splunk
Third-party HTTP endpoints — Datadog, Dynatrace, New Relic, MongoDB, Coralogix, Elastic, and others
Zero Buffering (December 2023)
Amazon Data Firehose now supports zero buffering for real-time data delivery
Delivers data within ~5 seconds with no buffering delay
Available for destinations: S3, OpenSearch Service, Redshift, and third-party HTTP endpoints
Enables real-time use cases that previously required Kinesis Data Streams
Trade-off: More frequent deliveries may result in more small files and higher costs
Dynamic Partitioning
Dynamically partition streaming data before delivery to S3 using static or dynamically defined keys (e.g., customer_id, transaction_id, region)
Firehose groups data by these keys and delivers into key-unique S3 prefixes
Enables high-performance, cost-efficient analytics with Athena, EMR, and Redshift Spectrum
Supports inline parsing (JQ expressions) to extract keys from JSON records without Lambda
Can be combined with data transformation (Lambda) for complex routing logic
Available only for S3 destination
Format Conversion (Parquet and ORC)
Firehose can convert incoming JSON data to columnar formats (Apache Parquet or Apache ORC) before storing in S3
Columnar formats are optimized for analytics cost and performance with Athena, Redshift Spectrum, and EMR
Uses AWS Glue Data Catalog schema for conversion
Reduces storage costs and improves query performance compared to raw JSON
Apache Iceberg Tables Support (GA September 2024)
Amazon Data Firehose can stream data directly into Apache Iceberg tables in S3
Iceberg brings SQL table reliability and ACID transactions to S3 data lakes
Supports automatic schema management, partitioning, and compaction
Compatible with Athena, EMR, Redshift, Spark, Flink, and other analytics engines
Simplifies data lake ingestion without custom ETL code
Content-based routing: Route records from a single stream to different Iceberg tables based on record content
Row-level operations: Apply update or delete operations for data correction and right-to-forget scenarios
Use cases: Real-time data lake ingestion, streaming analytics, CDC to data lake
Amazon S3 Tables Support (GA March 2025)
Amazon Data Firehose can deliver streaming data directly into Amazon S3 Tables — a purpose-built, managed Apache Iceberg table store
S3 Tables provide storage optimized for analytics workloads with built-in Apache Iceberg support
Delivers up to 3x faster query performance and 10x higher transactions per second compared to self-managed Iceberg tables in general purpose S3 buckets
Automatic continuous table optimization (compaction, snapshot management) without additional infrastructure
Supports content-based routing to different S3 Tables and row-level update/delete operations
Integrated with AWS Glue Data Catalog multi-catalog hierarchy (May 2025) — no resource links needed between default catalog and S3TablesCatalog
Compatible with Athena, EMR, Redshift, SageMaker Lakehouse, and other analytics engines
Use cases: Real-time data lake analytics, IoT data ingestion, streaming to data lakehouse
A user is designing a new service that receives location updates from 3600 rental cars every hour. The cars location needs to be uploaded to an Amazon S3 bucket. Each location must also be checked for distance from the original rental location. Which services will process the updates and automatically scale?
Amazon EC2 and Amazon EBS
Amazon Data Firehose and Amazon S3
Amazon ECS and Amazon RDS
Amazon S3 events and AWS Lambda
You need to perform ad-hoc SQL queries on massive amounts of well-structured data. Additional data comes in constantly at a high velocity, and you don’t want to have to manage the infrastructure processing it if possible. Which solution should you use?
Data Firehose and RDS
EMR running Apache Spark
Data Firehose and Redshift
EMR using Hive
Your organization needs to ingest a big data stream into their data lake on Amazon S3. The data may stream in at a rate of hundreds of megabytes per second. What AWS service will accomplish the goal with the least amount of management?
Amazon Data Firehose
Amazon Kinesis Data Streams
Amazon CloudFront
Amazon SQS
A startup company is building an application to track the high scores for a popular video game. Their Solution Architect is tasked with designing a solution to allow real-time processing of scores from millions of players worldwide. Which AWS service should the Architect use to provide reliable data ingestion from the video game into the datastore?
AWS Data Pipeline
Amazon Data Firehose
Amazon DynamoDB Streams
Amazon OpenSearch Service
A company has an infrastructure that consists of machines which keep sending log information every 5 minutes. The number of these machines can run into thousands and it is required to ensure that the data can be analyzed at a later stage. Which of the following would help in fulfilling this requirement?
Use Data Firehose with S3 to take the logs and store them in S3 for further processing.
Launch an Elastic Beanstalk application to take the processing job of the logs.
Launch an EC2 instance with enough EBS volumes to consume the logs which can be used for further processing.
Use CloudTrail to store all the logs which can be analyzed at a later stage.
A company needs to stream data to Amazon S3 with the lowest possible latency (under 10 seconds). Which configuration should they use?
Data Firehose with 60-second buffer
Data Firehose with zero buffering enabled
Kinesis Data Streams with Lambda consumer
Direct PUT to S3
A data analytics team needs to stream real-time data into Apache Iceberg tables in S3 for analytics with automatic table optimization and up to 3x faster queries. Which destination should they use?
Apache Iceberg Tables in general-purpose S3 bucket
Amazon S3 Tables
Amazon Redshift with COPY command
AWS Glue Streaming ETL
A company streams millions of events per day from different applications. They need to route events to different analytics tables in S3 based on event type, with ACID transaction support. Which Data Firehose feature enables this?
Dynamic Partitioning to S3 prefixes
Content-based routing to Apache Iceberg Tables
Lambda transformation with multiple outputs
Multiple delivery streams
A company wants to consume streaming data from an Amazon MSK cluster and load it into S3 in Parquet format without managing consumers or infrastructure. Which solution requires the LEAST effort?
Write a custom Kafka consumer with Spark
Use Amazon MSK Connect with S3 Sink Connector
Amazon Data Firehose with MSK as source and Parquet format conversion
