Kinesis data streams is highly customizable and best suited for developers building custom applications or streaming data for specialized needs.
Kinesis Data Firehose handles loading data streams directly into AWS products for processing. Firehose also allows for streaming to S3, OpenSearch Service, Redshift, Apache Iceberg tables, Snowflake, and other destinations, where data can be copied for processing through additional services.
Provisioning & Scaling
Kinesis Data Streams offers two capacity modes:
Provisioned Mode: Requires manual configuration of shards and scaling. You specify the number of shards needed based on expected throughput.
On-Demand Mode (Launched November 2021): Automatically scales to handle gigabytes of write and read throughput per minute without manual shard management. Default capacity of 4 MB/s write (4000 records/s), can scale up to 200 MB/s (or 1 GB/s with limit increase).
Kinesis Data Firehose is fully managed and sends data to S3, Redshift, OpenSearch, Apache Iceberg tables, Snowflake, and other destinations. Scaling is handled automatically, up to gigabytes per second, and allows for batching, encrypting, and compressing.
Processing Delay
Kinesis Data Streams provides real-time processing with ~200 ms for shared throughput classic single consumer and ~70 ms for the enhanced fan-out consumer.
Kinesis Data Firehose provides near real-time processing:
Standard Buffering: Minimum buffer time of 60 seconds (1 min), maximum 900 seconds (15 min)
Zero Buffering (Announced December 2023): Delivers data within ~5 seconds with no buffering delay, enabling real-time use cases
Data Storage
Kinesis Data Streams provide data storage. Data typically is made available in a stream for 24 hours, but for an additional cost, users can gain data availability for up to 365 days (8760 hours).
Kinesis Data Firehose does not provide data storage.
Replay
Kinesis Data Streams supports replay capability
Kinesis Data Firehose does not support replay capability
Producers & Consumers
Kinesis Data Streams & Kinesis Data Firehose support multiple producer options including SDK, KPL, Kinesis Agent, IoT, etc.
Kinesis Data Streams support multiple consumers option including SDK, KCL, and Lambda, and can write data to multiple destinations. However, they have to be coded.
Kinesis Data Firehose consumers are close-ended and support destinations including:
Amazon S3
Amazon Redshift
Amazon OpenSearch Service
Amazon OpenSearch Serverless
Apache Iceberg Tables (Added October 2024) – Stream data directly into Iceberg format tables in S3
Snowflake (with Snowpipe Streaming) – Real-time streaming to Snowflake
Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
Open to further feedback, discussion and correction.
Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
Open to further feedback, discussion and correction.
Your organization needs to ingest a big data stream into its data lake on Amazon S3. The data may stream in at a rate of hundreds of megabytes per second. What AWS service will accomplish the goal with the least amount of management?
Amazon Kinesis Firehose
Amazon Kinesis Streams
Amazon CloudFront
Amazon SQS
Your organization is looking for a solution that can help the business with streaming data several services will require access to read and process the same stream concurrently. What AWS service meets the business requirements?
Amazon Kinesis Firehose
Amazon Kinesis Streams
Amazon CloudFront
Amazon SQS
Your application generates a 1 KB JSON payload that needs to be queued and delivered to EC2 instances for applications. At the end of the day, the application needs to replay the data for the past 24 hours. In the near future, you also need the ability for other multiple EC2 applications to consume the same stream concurrently. What is the best solution for this?
Kinesis Data Streams
Kinesis Firehose
SNS
SQS
A company needs to stream data to Amazon S3 with the lowest possible latency (under 10 seconds). Which Kinesis service and configuration should they use? (Assume December 2023 or later)
Kinesis Data Streams with Lambda consumer
Kinesis Data Firehose with zero buffering enabled
Kinesis Data Firehose with 60-second buffer
Kinesis Data Streams with KCL consumer
A company wants to avoid manual shard management for their Kinesis Data Streams. Which capacity mode should they use? (Assume November 2021 or later)
Provisioned mode with Auto Scaling
On-Demand mode
Enhanced fan-out mode
Standard mode
A data analytics team needs to stream real-time data into Apache Iceberg tables in S3 for analytics. Which AWS service supports this natively? (Assume October 2024 or later)
Kinesis acts as a highly available conduit to stream messages between data producers and data consumers.
Data producers can be almost any source of data: system or web log data, social network data, financial trading information, geospatial data, mobile app data, or telemetry from connected IoT devices.
Data consumers will typically fall into the category of data processing and storage applications such as Apache Hadoop, Apache Storm, S3, and ElasticSearch.
Purpose
Kinesis data streams is highly customizable and best suited for developers building custom applications or streaming data for specialized needs.
Kinesis Data Firehose handles loading data streams directly into AWS products for processing. Firehose also allows for streaming to S3, OpenSearch Service, Redshift, Apache Iceberg tables, Snowflake, and other destinations, where data can be copied for processing through additional services.
Provisioning & Scaling
Kinesis Data Streams offers two capacity modes:
Provisioned Mode: Requires manual configuration of shards and scaling. You specify the number of shards needed based on expected throughput.
On-Demand Mode (Launched November 2021): Automatically scales to handle gigabytes of write and read throughput per minute without manual shard management. Default capacity of 4 MB/s write (4000 records/s), can scale up to 200 MB/s (or 1 GB/s with limit increase).
Kinesis Data Firehose is fully managed and sends data to S3, Redshift, OpenSearch, Apache Iceberg tables, Snowflake, and other destinations. Scaling is handled automatically, up to gigabytes per second, and allows for batching, encrypting, and compressing.
Processing Delay
Kinesis Data Streams provides real-time processing with ~200 ms for shared throughput classic single consumer and ~70 ms for the enhanced fan-out consumer.
Kinesis Data Firehose provides near real-time processing:
Standard Buffering: Minimum buffer time of 60 seconds (1 min), maximum 900 seconds (15 min)
Zero Buffering (Announced December 2023): Delivers data within ~5 seconds with no buffering delay, enabling real-time use cases
Data Storage
Kinesis Data Streams provide data storage. Data typically is made available in a stream for 24 hours, but for an additional cost, users can gain data availability for up to 365 days (8760 hours).
Kinesis Data Firehose does not provide data storage.
Replay
Kinesis Data Streams supports replay capability
Kinesis Data Firehose does not support replay capability
Producers & Consumers
Kinesis Data Streams & Kinesis Data Firehose support multiple producer options including SDK, KPL, Kinesis Agent, IoT, etc.
Kinesis Data Streams support multiple consumers option including SDK, KCL, and Lambda, and can write data to multiple destinations. However, they have to be coded.
Kinesis Data Firehose consumers are close-ended and support destinations including:
Amazon S3
Amazon Redshift
Amazon OpenSearch Service
Amazon OpenSearch Serverless
Apache Iceberg Tables (Added October 2024) – Stream data directly into Iceberg format tables in S3
Snowflake (with Snowpipe Streaming) – Real-time streaming to Snowflake
Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
Open to further feedback, discussion and correction.
Your organization needs to ingest a big data stream into its data lake on Amazon S3. The data may stream in at a rate of hundreds of megabytes per second. What AWS service will accomplish the goal with the least amount of management?
Amazon Kinesis Firehose
Amazon Kinesis Streams
Amazon CloudFront
Amazon SQS
Your organization is looking for a solution that can help the business with streaming data several services will require access to read and process the same stream concurrently. What AWS service meets the business requirements?
Amazon Kinesis Firehose
Amazon Kinesis Streams
Amazon CloudFront
Amazon SQS
Your application generates a 1 KB JSON payload that needs to be queued and delivered to EC2 instances for applications. At the end of the day, the application needs to replay the data for the past 24 hours. In the near future, you also need the ability for other multiple EC2 applications to consume the same stream concurrently. What is the best solution for this?
Kinesis Data Streams
Kinesis Firehose
SNS
SQS
A company needs to stream data to Amazon S3 with the lowest possible latency (under 10 seconds). Which Kinesis service and configuration should they use? (Assume December 2023 or later)
Kinesis Data Streams with Lambda consumer
Kinesis Data Firehose with zero buffering enabled
Kinesis Data Firehose with 60-second buffer
Kinesis Data Streams with KCL consumer
A company wants to avoid manual shard management for their Kinesis Data Streams. Which capacity mode should they use? (Assume November 2021 or later)
Provisioned mode with Auto Scaling
On-Demand mode
Enhanced fan-out mode
Standard mode
A data analytics team needs to stream real-time data into Apache Iceberg tables in S3 for analytics. Which AWS service supports this natively? (Assume October 2024 or later)
Amazon Data Firehose (formerly Kinesis Data Firehose)
📢 Service Renamed (February 2024): Amazon Kinesis Data Firehose has been renamed to Amazon Data Firehose. The functionality remains the same.
Amazon Data Firehose is a fully managed service for delivering real-time streaming data
Amazon Data Firehose automatically scales to match the throughput of the data and requires no ongoing administration or need to write applications or manage resources
is a data transfer solution for delivering real-time streaming data to destinations such as S3, Redshift, OpenSearch Service, OpenSearch Serverless, Apache Iceberg tables, Snowflake, Splunk, and third-party HTTP endpoints.
is NOT Real Time, but Near Real Time as it supports batching and buffers streaming data to a certain size (Buffer Size in MBs) or for a certain period of time (Buffer Interval in seconds) before delivering it to destinations.
Zero Buffering (December 2023): Firehose now supports zero buffering, delivering data within ~5 seconds with no buffering delay for real-time use cases.
supports data compression, minimizing the amount of storage used at the destination. It currently supports GZIP, ZIP, and SNAPPY compression formats. Only GZIP is supported if the data is further loaded to Redshift.
supports data at rest encryption using KMS after the data is delivered to the S3 bucket.
supports multiple producers as datasource, which include Kinesis data stream, Kinesis Agent, or the Data Firehose API using the AWS SDK, CloudWatch Logs, CloudWatch Events, or AWS IoT
supports out of box data transformation as well as custom transformation using the Lambda function to transform incoming source data and deliver the transformed data to destinations
supports source record backup with custom data transformation with Lambda, where Data Firehose will deliver the un-transformed incoming data to a separate S3 bucket.
uses at least once semantics for data delivery. In rare circumstances such as request timeout upon data delivery attempt, delivery retry by Firehose could introduce duplicates if the previous request eventually goes through.
Underlying entity of Data Firehose, where the data is sent
Record
Data sent by data producer to a Data Firehose delivery stream
Maximum size of a record (before Base64-encoding) is 1024 KB.
Data producer
Producers send records to Data Firehose delivery streams.
Buffer size and buffer interval
Data Firehose buffers incoming streaming data to a certain size or for a certain time period before delivering it to destinations
Buffer size and buffer interval can be configured while creating the delivery stream
Buffer size is in MBs and ranges from 1MB to 128MB for the S3 destination and 1MB to 100MB for the OpenSearch Service destination.
Buffer interval is in seconds and ranges from 60 secs to 900 secs (standard buffering)
Zero Buffering (December 2023): Set buffer interval to 0 seconds to deliver data within ~5 seconds with no buffering delay
Firehose raises buffer size dynamically to catch up and make sure that all data is delivered to the destination, if data delivery to the destination is falling behind data writing to the delivery stream
Buffer size is applied before compression.
Destination
A destination is the data store where the data will be delivered.
supports the following destinations:
Amazon S3
Amazon Redshift
Amazon OpenSearch Service
Amazon OpenSearch Serverless (added November 2022)
Apache Iceberg Tables (added October 2024) – Stream data directly into Iceberg format tables in S3
Snowflake – Real-time streaming to Snowflake via Snowpipe Streaming
Splunk
Third-party HTTP endpoints – Datadog, Dynatrace, New Relic, MongoDB, Coralogix, Elastic, and others
Zero Buffering (December 2023)
Amazon Data Firehose now supports zero buffering for real-time data delivery
Delivers data within ~5 seconds with no buffering delay
Available for destinations: S3, OpenSearch Service, Redshift, and third-party HTTP endpoints
Enables real-time use cases that previously required Kinesis Data Streams
Trade-off: More frequent deliveries may result in more small files and higher costs
Apache Iceberg Tables Support (October 2024)
Amazon Data Firehose can now stream data directly into Apache Iceberg tables in S3
Iceberg brings SQL table reliability and ACID transactions to S3 data lakes
Supports automatic schema management, partitioning, and compaction
Compatible with Athena, EMR, Redshift, Spark, Flink, and other analytics engines
Simplifies data lake ingestion without custom ETL code
Use cases: Real-time data lake ingestion, streaming analytics, CDC to data lake
A user is designing a new service that receives location updates from 3600 rental cars every hour. The cars location needs to be uploaded to an Amazon S3 bucket. Each location must also be checked for distance from the original rental location. Which services will process the updates and automatically scale?
Amazon EC2 and Amazon EBS
Amazon Data Firehose and Amazon S3
Amazon ECS and Amazon RDS
Amazon S3 events and AWS Lambda
You need to perform ad-hoc SQL queries on massive amounts of well-structured data. Additional data comes in constantly at a high velocity, and you don’t want to have to manage the infrastructure processing it if possible. Which solution should you use?
Data Firehose and RDS
EMR running Apache Spark
Data Firehose and Redshift
EMR using Hive
Your organization needs to ingest a big data stream into their data lake on Amazon S3. The data may stream in at a rate of hundreds of megabytes per second. What AWS service will accomplish the goal with the least amount of management?
Amazon Data Firehose
Amazon Kinesis Data Streams
Amazon CloudFront
Amazon SQS
A startup company is building an application to track the high scores for a popular video game. Their Solution Architect is tasked with designing a solution to allow real-time processing of scores from millions of players worldwide. Which AWS service should the Architect use to provide reliable data ingestion from the video game into the datastore?
AWS Data Pipeline
Amazon Data Firehose
Amazon DynamoDB Streams
Amazon Elasticsearch Service
A company has an infrastructure that consists of machines which keep sending log information every 5 minutes. The number of these machines can run into thousands and it is required to ensure that the data can be analyzed at a later stage. Which of the following would help in fulfilling this requirement?
Use Data Firehose with S3 to take the logs and store them in S3 for further processing.
Launch an Elastic Beanstalk application to take the processing job of the logs.
Launch an EC2 instance with enough EBS volumes to consume the logs which can be used for further processing.
Use CloudTrail to store all the logs which can be analyzed at a later stage.
A company needs to stream data to Amazon S3 with the lowest possible latency (under 10 seconds). Which configuration should they use? (Assume December 2023 or later)
Data Firehose with 60-second buffer
Data Firehose with zero buffering enabled
Kinesis Data Streams with Lambda consumer
Direct PUT to S3
A data analytics team needs to stream real-time data into Apache Iceberg tables in S3 for analytics. Which AWS service supports this natively? (Assume October 2024 or later)
