Kinesis Data Firehose

Kinesis Data Streams vs Firehose – Comparison

Kinesis Data Streams vs. Kinesis Data Firehose
~ Last updated on : ~ jayendrapatil ~ 11 Comments

AWS Kinesis Data Streams vs Amazon Data Firehose

📢 Service Rename (February 2024): Amazon Kinesis Data Firehose has been renamed to Amazon Data Firehose. The functionality remains the same. This post uses both names for clarity.

  • Kinesis acts as a highly available conduit to stream messages between data producers and data consumers.
  • Data producers can be almost any source of data: system or web log data, social network data, financial trading information, geospatial data, mobile app data, or telemetry from connected IoT devices.
  • Data consumers will typically fall into the category of data processing and storage applications such as Apache Hadoop, Apache Storm, S3, and OpenSearch.

Kinesis Data Streams vs. Firehose

Purpose

  • Kinesis Data Streams is highly customizable and best suited for developers building custom applications or streaming data for specialized needs.
  • Amazon Data Firehose (formerly Kinesis Data Firehose) handles loading data streams directly into AWS products for processing. Firehose allows streaming to S3, OpenSearch Service, Redshift, Apache Iceberg tables, Amazon S3 Tables, Snowflake, and other destinations, where data can be copied for processing through additional services.

Provisioning & Scaling

  • Kinesis Data Streams offers three capacity modes:
    • Provisioned Mode: Requires manual configuration of shards and scaling. You specify the number of shards needed based on expected throughput. Each shard provides 1 MB/s write (1000 records/s) and 2 MB/s read.
    • On-Demand Standard Mode (Launched November 2021): Automatically scales to handle gigabytes of write and read throughput per minute without manual shard management. Default capacity of 4 MB/s write (4000 records/s), can scale up to 200 MB/s (or 1 GB/s with limit increase).
    • On-Demand Advantage Mode (Launched November 2025): Enables warm throughput capability for instant scaling up to 10 GB/s or 10 million events/second. Offers 60% lower pricing compared to On-demand Standard ($0.032/GB ingest, $0.016/GB retrieval). Removes per-stream fixed charge. Supports up to 50 enhanced fan-out consumers (vs. 20 for other modes). Best for workloads ingesting at least 10 MB/s aggregate, high fan-out use cases, or accounts with hundreds of streams.
  • Amazon Data Firehose is fully managed and sends data to S3, Redshift, OpenSearch, Apache Iceberg tables, Amazon S3 Tables, Snowflake, and other destinations. Scaling is handled automatically, up to gigabytes per second, and allows for batching, encrypting, and compressing.

Processing Delay

  • Kinesis Data Streams provides real-time processing with ~200 ms for shared throughput classic single consumer and ~70 ms for the enhanced fan-out consumer.
  • Amazon Data Firehose provides near real-time processing:
    • Standard Buffering: Minimum buffer time of 60 seconds (1 min), maximum 900 seconds (15 min)
    • Zero Buffering (Announced December 2023): Delivers data within ~5 seconds with no buffering delay, enabling real-time use cases

Record Size

  • Kinesis Data Streams supports record sizes up to 10 MiB (increased from 1 MiB in October 2025), enabling larger data payloads like images, documents, and complex event data without splitting records.
  • Amazon Data Firehose supports record sizes up to 1 MiB per record.

Data Storage

  • Kinesis Data Streams provides data storage. Data typically is made available in a stream for 24 hours, but for an additional cost, users can gain data availability for up to 365 days (8760 hours). On-demand Advantage mode offers 77% lower extended retention pricing ($0.023/GB-month vs $0.10/GB-month).
  • Amazon Data Firehose does not provide data storage.

Replay

  • Kinesis Data Streams supports replay capability
  • Amazon Data Firehose does not support replay capability

Producers & Consumers

  • Kinesis Data Streams & Amazon Data Firehose support multiple producer options including SDK, KPL, Kinesis Agent, IoT, etc.
  • Kinesis Data Streams supports multiple consumer options including SDK, KCL, and Lambda, and can write data to multiple destinations. However, they have to be coded.
    • Supports up to 50 enhanced fan-out consumers per stream with On-demand Advantage mode (up from 20 with On-demand Standard/Provisioned modes) — Launched November 2025
    • Enhanced fan-out provides dedicated 2 MB/s throughput per consumer per shard with ~70 ms latency
  • Amazon Data Firehose consumers are close-ended and support destinations including:
    • Amazon S3
    • Amazon Redshift
    • Amazon OpenSearch Service
    • Amazon OpenSearch Serverless
    • Apache Iceberg Tables (Added October 2024) — Stream data directly into Iceberg format tables in S3
    • Amazon S3 Tables (GA March 2025) — Stream data into S3 Tables with built-in Apache Iceberg support and automatic table maintenance (compaction, snapshot management)
    • Snowflake (with Snowpipe Streaming) — Real-time streaming to Snowflake
    • Splunk
    • Third-party HTTP endpoints (Datadog, Dynatrace, New Relic, MongoDB, Coralogix, Elastic, etc.)
  • Amazon Data Firehose also supports database CDC replication (Preview, November 2024) — captures change data from MySQL and PostgreSQL databases and replicates directly to Apache Iceberg tables in S3, enabling near real-time data lake updates without custom ETL code.

Key Differences Summary

Feature Kinesis Data Streams Amazon Data Firehose
Capacity Mode Provisioned, On-Demand Standard, or On-Demand Advantage Fully managed (automatic)
Latency Real-time (~70-200 ms) Near real-time (60s default, ~5s with zero buffering)
Max Record Size 10 MiB (since Oct 2025) 1 MiB
Data Retention 24 hours to 365 days No storage
Replay ✅ Supported ❌ Not supported
Max Throughput Up to 10 GB/s (On-demand Advantage with warm throughput) Automatic scaling (gigabytes/second)
Enhanced Fan-Out Consumers Up to 50 (Advantage) / 20 (Standard/Provisioned) N/A — pre-defined destinations
Consumers Custom (SDK, KCL, Lambda) Pre-defined (S3, Redshift, OpenSearch, Iceberg, S3 Tables, Snowflake, etc.)
Use Case Custom processing, real-time analytics ETL, loading to data stores, CDC replication

KCL 1.x End of Support

⚠️ Important: Kinesis Client Library (KCL) 1.x reached end-of-support on January 30, 2026. AWS strongly recommends migrating KCL applications to KCL 2.x or later. KCL 1.x entered maintenance mode on April 17, 2025 with only critical bug fixes and security updates.

Migration: See Migrate consumers from KCL 1.x to KCL 2.x

AWS Certification Exam Practice Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  1. Your organization needs to ingest a big data stream into its data lake on Amazon S3. The data may stream in at a rate of hundreds of megabytes per second. What AWS service will accomplish the goal with the least amount of management?
    1. Amazon Data Firehose
    2. Amazon Kinesis Data Streams
    3. Amazon CloudFront
    4. Amazon SQS
  2. Your organization is looking for a solution that can help the business with streaming data several services will require access to read and process the same stream concurrently. What AWS service meets the business requirements?
    1. Amazon Data Firehose
    2. Amazon Kinesis Data Streams
    3. Amazon CloudFront
    4. Amazon SQS
  3. Your application generates a 1 KB JSON payload that needs to be queued and delivered to EC2 instances for applications. At the end of the day, the application needs to replay the data for the past 24 hours. In the near future, you also need the ability for other multiple EC2 applications to consume the same stream concurrently. What is the best solution for this?
    1. Kinesis Data Streams
    2. Amazon Data Firehose
    3. SNS
    4. SQS
  4. A company needs to stream data to Amazon S3 with the lowest possible latency (under 10 seconds). Which Kinesis service and configuration should they use? (Assume December 2023 or later)
    1. Kinesis Data Streams with Lambda consumer
    2. Amazon Data Firehose with zero buffering enabled
    3. Amazon Data Firehose with 60-second buffer
    4. Kinesis Data Streams with KCL consumer
  5. A company wants to avoid manual shard management for their Kinesis Data Streams and needs to handle instant traffic surges up to 10 GB/s. Which capacity mode should they use?
    1. Provisioned mode with Auto Scaling
    2. On-Demand Standard mode
    3. On-Demand Advantage mode with warm throughput
    4. Enhanced fan-out mode
  6. A data analytics team needs to stream real-time data into Apache Iceberg tables in S3 for analytics. Which AWS service supports this natively? (Assume October 2024 or later)
    1. Kinesis Data Streams
    2. Amazon Data Firehose
    3. AWS Glue Streaming
    4. Amazon MSK
  7. A company streams data using Kinesis Data Streams with 30 independent consumer applications needing dedicated throughput. Which configuration supports this?
    1. On-Demand Standard mode with enhanced fan-out
    2. Provisioned mode with enhanced fan-out
    3. On-Demand Advantage mode with enhanced fan-out
    4. Create separate streams for each consumer
  8. A company needs to stream IoT sensor data that occasionally includes 5 MiB payloads. Which streaming service supports this record size natively? (Assume October 2025 or later)
    1. Amazon Data Firehose
    2. Kinesis Data Streams
    3. Amazon SQS
    4. Amazon SNS
  9. A company wants to stream real-time data into Amazon S3 Tables with built-in Apache Iceberg support and automatic table maintenance. Which service should they use?
    1. Kinesis Data Streams with custom Lambda
    2. Amazon Data Firehose
    3. AWS Glue ETL job
    4. Amazon EMR Streaming

References

Amazon Data Firehose – Delivery & Transformation

Kinesis Data Firehose
~ Last updated on : ~ jayendrapatil ~ 8 Comments

Amazon Data Firehose (formerly Kinesis Data Firehose)

📢 Service Renamed (February 2024): Amazon Kinesis Data Firehose has been renamed to Amazon Data Firehose. The functionality remains the same. Existing applications, API endpoints, and IAM policies continue to work without changes.

  • Amazon Data Firehose is a fully managed service for delivering real-time streaming data to data stores and analytics tools.
  • Amazon Data Firehose automatically scales to match the throughput of the data (gigabytes per second or more) and requires no ongoing administration or need to write applications or manage resources.
  • is a data transfer solution for delivering real-time streaming data to destinations such as S3, Redshift, OpenSearch Service, OpenSearch Serverless, Apache Iceberg Tables, Amazon S3 Tables, Snowflake, Splunk, and third-party HTTP endpoints.
  • is NOT Real Time, but Near Real Time as it supports batching and buffers streaming data to a certain size (Buffer Size in MBs) or for a certain period of time (Buffer Interval in seconds) before delivering it to destinations.
    • Zero Buffering (December 2023): Firehose now supports zero buffering, delivering data within ~5 seconds with no buffering delay for real-time use cases.
  • supports data compression, minimizing the amount of storage used at the destination. It currently supports GZIP, ZIP, and SNAPPY compression formats. Only GZIP is supported if the data is further loaded to Redshift.
  • supports Apache Parquet and ORC format conversion — converts incoming JSON data to columnar formats optimized for analytics with Athena, Redshift Spectrum, and EMR before storing in S3.
  • supports data at rest encryption using KMS after the data is delivered to the S3 bucket.
  • supports 20+ data sources including Amazon Kinesis Data Streams, Amazon MSK (and MSK Serverless), Direct PUT API, Kinesis Agent, CloudWatch Logs, CloudWatch Events, AWS IoT Core, Amazon SNS, AWS WAF web ACL logs, and Amazon VPC Flow Logs.
  • supports out of box data transformation as well as custom transformation using the Lambda function to transform incoming source data and deliver the transformed data to destinations.
  • supports Dynamic Partitioning — groups streaming data by static or dynamically defined keys (e.g., customer_id, region) and delivers into key-unique S3 prefixes for optimized analytics.
  • supports source record backup with custom data transformation with Lambda, where Data Firehose will deliver the un-transformed incoming data to a separate S3 bucket.
  • uses at least once semantics for data delivery. In rare circumstances such as request timeout upon data delivery attempt, delivery retry by Firehose could introduce duplicates if the previous request eventually goes through.
  • supports Interface VPC Interface Endpoint (AWS Private Link) to keep traffic between the VPC and Data Firehose from leaving the Amazon network.

Amazon Data Firehose

Amazon Data Firehose Key Concepts

  • Data Firehose delivery stream
    • Underlying entity of Data Firehose, where the data is sent
  • Record
    • Data sent by data producer to a Data Firehose delivery stream
    • Maximum size of a record (before Base64-encoding) is 1024 KB.
    • With Amazon MSK as source, maximum record size is 10 MB (6 MB if Lambda transformation is enabled).
  • Data producer
    • Producers send records to Data Firehose delivery streams.
  • Buffer size and buffer interval
    • Data Firehose buffers incoming streaming data to a certain size or for a certain time period before delivering it to destinations
    • Buffer size and buffer interval can be configured while creating the delivery stream
    • Buffer size is in MBs and ranges from 1MB to 128MB for the S3 destination and 1MB to 100MB for the OpenSearch Service destination.
    • Buffer interval is in seconds and ranges from 0 secs (zero buffering) to 900 secs
    • Zero Buffering (December 2023): Set buffer interval to 0 seconds to deliver data within ~5 seconds with no buffering delay
    • Firehose raises buffer size dynamically to catch up and make sure that all data is delivered to the destination, if data delivery to the destination is falling behind data writing to the delivery stream
    • Buffer size is applied before compression.
  • Source
    • Data Firehose supports 20+ data sources:
      • Amazon Kinesis Data Streams — read directly from a KDS stream
      • Amazon MSK / MSK Serverless — consume from Kafka topics
      • Direct PUT — via Firehose API, SDK, or Kinesis Agent
      • AWS Services — CloudWatch Logs, CloudWatch Events, AWS IoT Core, Amazon SNS, AWS WAF web ACL logs, Amazon VPC Flow Logs, and others
  • Destination
    • A destination is the data store where the data will be delivered.
    • supports the following destinations:
      • Amazon S3 — with optional dynamic partitioning and format conversion
      • Amazon Redshift — via intermediate S3 COPY
      • Amazon OpenSearch Service
      • Amazon OpenSearch Serverless (added November 2022)
      • Apache Iceberg Tables in S3 (GA September 2024) — stream into Iceberg format tables with ACID transactions
      • Amazon S3 Tables (GA March 2025) — purpose-built managed Iceberg tables with automatic optimization
      • Snowflake — real-time streaming via Snowpipe Streaming
      • Splunk
      • Third-party HTTP endpoints — Datadog, Dynatrace, New Relic, MongoDB, Coralogix, Elastic, and others

Zero Buffering (December 2023)

  • Amazon Data Firehose now supports zero buffering for real-time data delivery
  • Delivers data within ~5 seconds with no buffering delay
  • Available for destinations: S3, OpenSearch Service, Redshift, and third-party HTTP endpoints
  • Enables real-time use cases that previously required Kinesis Data Streams
  • Trade-off: More frequent deliveries may result in more small files and higher costs

Dynamic Partitioning

  • Dynamically partition streaming data before delivery to S3 using static or dynamically defined keys (e.g., customer_id, transaction_id, region)
  • Firehose groups data by these keys and delivers into key-unique S3 prefixes
  • Enables high-performance, cost-efficient analytics with Athena, EMR, and Redshift Spectrum
  • Supports inline parsing (JQ expressions) to extract keys from JSON records without Lambda
  • Can be combined with data transformation (Lambda) for complex routing logic
  • Available only for S3 destination

Format Conversion (Parquet and ORC)

  • Firehose can convert incoming JSON data to columnar formats (Apache Parquet or Apache ORC) before storing in S3
  • Columnar formats are optimized for analytics cost and performance with Athena, Redshift Spectrum, and EMR
  • Uses AWS Glue Data Catalog schema for conversion
  • Reduces storage costs and improves query performance compared to raw JSON

Apache Iceberg Tables Support (GA September 2024)

  • Amazon Data Firehose can stream data directly into Apache Iceberg tables in S3
  • Iceberg brings SQL table reliability and ACID transactions to S3 data lakes
  • Supports automatic schema management, partitioning, and compaction
  • Compatible with Athena, EMR, Redshift, Spark, Flink, and other analytics engines
  • Simplifies data lake ingestion without custom ETL code
  • Content-based routing: Route records from a single stream to different Iceberg tables based on record content
  • Row-level operations: Apply update or delete operations for data correction and right-to-forget scenarios
  • Use cases: Real-time data lake ingestion, streaming analytics, CDC to data lake

Amazon S3 Tables Support (GA March 2025)

  • Amazon Data Firehose can deliver streaming data directly into Amazon S3 Tables — a purpose-built, managed Apache Iceberg table store
  • S3 Tables provide storage optimized for analytics workloads with built-in Apache Iceberg support
  • Delivers up to 3x faster query performance and 10x higher transactions per second compared to self-managed Iceberg tables in general purpose S3 buckets
  • Automatic continuous table optimization (compaction, snapshot management) without additional infrastructure
  • Supports content-based routing to different S3 Tables and row-level update/delete operations
  • Integrated with AWS Glue Data Catalog multi-catalog hierarchy (May 2025) — no resource links needed between default catalog and S3TablesCatalog
  • Compatible with Athena, EMR, Redshift, SageMaker Lakehouse, and other analytics engines
  • Use cases: Real-time data lake analytics, IoT data ingestion, streaming to data lakehouse

Amazon Data Firehose vs Kinesis Data Streams

Kinesis Data Streams vs. Amazon Data Firehose

AWS Certification Exam Practice Questions

  1. A user is designing a new service that receives location updates from 3600 rental cars every hour. The cars location needs to be uploaded to an Amazon S3 bucket. Each location must also be checked for distance from the original rental location. Which services will process the updates and automatically scale?
    1. Amazon EC2 and Amazon EBS
    2. Amazon Data Firehose and Amazon S3
    3. Amazon ECS and Amazon RDS
    4. Amazon S3 events and AWS Lambda
  2. You need to perform ad-hoc SQL queries on massive amounts of well-structured data. Additional data comes in constantly at a high velocity, and you don’t want to have to manage the infrastructure processing it if possible. Which solution should you use?
    1. Data Firehose and RDS
    2. EMR running Apache Spark
    3. Data Firehose and Redshift
    4. EMR using Hive
  3. Your organization needs to ingest a big data stream into their data lake on Amazon S3. The data may stream in at a rate of hundreds of megabytes per second. What AWS service will accomplish the goal with the least amount of management?
    1. Amazon Data Firehose
    2. Amazon Kinesis Data Streams
    3. Amazon CloudFront
    4. Amazon SQS
  4. A startup company is building an application to track the high scores for a popular video game. Their Solution Architect is tasked with designing a solution to allow real-time processing of scores from millions of players worldwide. Which AWS service should the Architect use to provide reliable data ingestion from the video game into the datastore?
    1. AWS Data Pipeline
    2. Amazon Data Firehose
    3. Amazon DynamoDB Streams
    4. Amazon OpenSearch Service
  5. A company has an infrastructure that consists of machines which keep sending log information every 5 minutes. The number of these machines can run into thousands and it is required to ensure that the data can be analyzed at a later stage. Which of the following would help in fulfilling this requirement?
    1. Use Data Firehose with S3 to take the logs and store them in S3 for further processing.
    2. Launch an Elastic Beanstalk application to take the processing job of the logs.
    3. Launch an EC2 instance with enough EBS volumes to consume the logs which can be used for further processing.
    4. Use CloudTrail to store all the logs which can be analyzed at a later stage.
  6. A company needs to stream data to Amazon S3 with the lowest possible latency (under 10 seconds). Which configuration should they use?
    1. Data Firehose with 60-second buffer
    2. Data Firehose with zero buffering enabled
    3. Kinesis Data Streams with Lambda consumer
    4. Direct PUT to S3
  7. A data analytics team needs to stream real-time data into Apache Iceberg tables in S3 for analytics with automatic table optimization and up to 3x faster queries. Which destination should they use?
    1. Apache Iceberg Tables in general-purpose S3 bucket
    2. Amazon S3 Tables
    3. Amazon Redshift with COPY command
    4. AWS Glue Streaming ETL
  8. A company streams millions of events per day from different applications. They need to route events to different analytics tables in S3 based on event type, with ACID transaction support. Which Data Firehose feature enables this?
    1. Dynamic Partitioning to S3 prefixes
    2. Content-based routing to Apache Iceberg Tables
    3. Lambda transformation with multiple outputs
    4. Multiple delivery streams
  9. A company wants to consume streaming data from an Amazon MSK cluster and load it into S3 in Parquet format without managing consumers or infrastructure. Which solution requires the LEAST effort?
    1. Write a custom Kafka consumer with Spark
    2. Use Amazon MSK Connect with S3 Sink Connector
    3. Amazon Data Firehose with MSK as source and Parquet format conversion
    4. AWS Glue Streaming ETL job

References