Kinesis Data Analytics

Kinesis Streams vs Firehose vs Flink – Compared

~ Last updated on : ~ Kiro Agent ~ Leave a comment

AWS Kinesis Data Streams vs Firehose vs Managed Apache Flink

AWS offers three core streaming services under the Kinesis umbrella, each designed for different use cases in real-time data processing pipelines. Understanding when to use Amazon Kinesis Data Streams, Amazon Data Firehose (formerly Kinesis Data Firehose), or Amazon Managed Service for Apache Flink (formerly Kinesis Data Analytics) is critical for AWS certification exams and real-world architecture decisions.

📢 Service Naming Updates:

  • February 2024: Amazon Kinesis Data Firehose renamed to Amazon Data Firehose
  • August 2023: Amazon Kinesis Data Analytics renamed to Amazon Managed Service for Apache Flink
  • January 2026: Kinesis Data Analytics for SQL discontinued — migrate to Managed Service for Apache Flink
Kinesis Data Streams vs Firehose vs Managed Flink
Data Streams
Producers
↓
Shards
(retain 24h-365d)
↓
KCL / Lambda / EFO
Real-time (~70ms)
Firehose
Sources (Streams/Direct)
↓
Buffer + Transform
(Lambda/format)
↓
S3 / Redshift / OpenSearch
Near real-time (60s-900s)
Managed Flink
Streams / MSK / S3
↓
Apache Flink
(SQL/Java/Python)
↓
Any destination
Complex processing (ms)

Service Overview

Amazon Kinesis Data Streams

A real-time data streaming service that enables you to collect, process, and analyze streaming data with sub-second latency. You build custom consumer applications using KCL, AWS Lambda, or the SDK. It provides full control over data retention (up to 365 days), ordering guarantees via partition keys, and supports multiple simultaneous consumers.

Amazon Data Firehose

A fully managed delivery service that captures, transforms, and loads streaming data into AWS destinations (S3, Redshift, OpenSearch, Apache Iceberg tables, Snowflake) and third-party HTTP endpoints. It requires zero administration — no shards, no capacity planning, and no custom consumer code.

Amazon Managed Service for Apache Flink

A fully managed stream processing engine that runs Apache Flink applications for complex event processing, real-time analytics, and stateful computations. Supports Java, Scala, Python (PyFlink), and SQL. Provides exactly-once processing semantics, windowed aggregations, and joins across multiple streams.

Architecture Comparison

Kinesis Data Streams Architecture

  • Shard-based model: Each shard provides 1 MB/s write (1,000 records/s) and 2 MB/s read capacity
  • Producers → Stream (Shards) → Consumers — fully decoupled
  • Data is stored in shards and replicated across 3 Availability Zones
  • Records are ordered within a shard by sequence number
  • Three capacity modes: Provisioned, On-Demand Standard, On-Demand Advantage
  • Consumers pull data via GetRecords (shared throughput) or SubscribeToShard (enhanced fan-out with push model)

Amazon Data Firehose Architecture

  • Serverless pipeline: No shards, no partitions, fully managed infrastructure
  • Producers → Firehose Stream → Buffer → Transform (optional) → Destination
  • Automatically scales to match incoming data throughput (up to GB/s)
  • Buffers data by size (1–128 MB) or time (0–900 seconds) before delivery
  • Can source directly from producers (Direct PUT) or from Kinesis Data Streams
  • Supports inline transformation via AWS Lambda functions

Managed Apache Flink Architecture

  • Application-based model: Deploy Apache Flink applications (JAR/ZIP) to a managed cluster
  • Sources → Flink Application (operators, state) → Sinks
  • Uses Kinesis Processing Units (KPUs) — each KPU = 1 vCPU + 4 GB memory
  • Supports parallel processing across multiple KPUs with automatic scaling
  • Maintains application state with checkpointing and savepoints
  • Integrates with Kinesis Data Streams, MSK, S3, DynamoDB, and custom connectors as sources/sinks

Data Retention

Feature Kinesis Data Streams Amazon Data Firehose Managed Apache Flink
Data Storage Yes — stores data in stream No — transit/delivery only Application state only (checkpoints)
Default Retention 24 hours N/A (buffered up to 15 min max) N/A (state stored in checkpoints)
Extended Retention Up to 7 days (additional charge) N/A N/A
Long-term Retention Up to 365 days N/A Snapshots stored in S3 (indefinite)
Replay Capability ✅ Yes — within retention window ❌ No ✅ Yes — via savepoints/snapshots

Scaling Model

Kinesis Data Streams — Shards & On-Demand

  • Provisioned Mode: Manually add/remove shards. Each shard = 1 MB/s in, 2 MB/s out. You pay per shard-hour ($0.015/shard-hour in us-east-1).
  • On-Demand Standard Mode: Auto-scales up to 200 MB/s write (expandable to 1 GB/s). Per-stream hourly charge + per-GB pricing ($0.08/GB ingest, $0.04/GB retrieval).
  • On-Demand Advantage Mode (Nov 2025): Instant scaling up to 10 GB/s or 10M events/s via warm throughput. 60% lower per-GB pricing ($0.032/GB ingest, $0.016/GB retrieval). No per-stream hourly fee. Requires 25 MB/s minimum aggregate commitment.

Amazon Data Firehose — Fully Automatic

  • No capacity units to manage — scales automatically from KB/s to GB/s
  • No shard splitting, merging, or resharding operations
  • Pricing based purely on data volume ingested (per GB)
  • Scales buffer size and parallelism transparently

Managed Apache Flink — KPU-based Auto-scaling

  • Uses Kinesis Processing Units (KPUs): 1 KPU = 1 vCPU + 4 GB memory
  • Default limit: 64 KPUs per application (can request increase)
  • Auto-scaling: Scales out when CPU utilization exceeds 75% for 15 minutes
  • Parallelism configurable per operator (default 1, max 8 per KPU)
  • Also supports metric-based scaling and scheduled scaling policies

Consumers & Processing

Kinesis Data Streams Consumers

  • Shared Throughput (Classic): Multiple consumers share 2 MB/s per shard. ~200 ms latency via GetRecords polling.
  • Enhanced Fan-Out (EFO): Dedicated 2 MB/s per consumer per shard. Push-based via HTTP/2 SubscribeToShard. ~70 ms latency.
  • Consumer Options:
    • KCL (Kinesis Client Library): Managed checkpointing, load balancing, shard assignment. KCL 1.x EOL January 30, 2026 — migrate to KCL 3.x.
    • AWS Lambda: Event source mapping with automatic scaling, batching, and error handling.
    • SDK: Low-level GetRecords/SubscribeToShard API calls for custom consumers.
    • Amazon Data Firehose: As a consumer for delivery to destinations.
    • Managed Apache Flink: As a consumer for complex stream processing.
  • Fan-out limits: Up to 50 EFO consumers per stream (On-Demand Advantage), 20 for other modes.

Amazon Data Firehose Consumers

  • No custom consumers — Firehose IS the consumer; it delivers to pre-defined destinations
  • Supported Destinations:
    • Amazon S3 (including S3 Tables)
    • Amazon Redshift (via S3 COPY)
    • Amazon OpenSearch Service / OpenSearch Serverless
    • Apache Iceberg Tables in S3
    • Snowflake (via Snowpipe Streaming)
    • Splunk
    • HTTP endpoints (Datadog, Dynatrace, New Relic, MongoDB, Coralogix, Elastic, etc.)
  • Data Transformation: Inline via AWS Lambda (up to 3 min timeout per invocation)

Managed Apache Flink Processing

  • Sources: Kinesis Data Streams, Amazon MSK, Amazon S3, DynamoDB Streams, JDBC, custom Apache Flink connectors
  • Sinks: Kinesis Data Streams, Amazon Data Firehose, S3, DynamoDB, OpenSearch, JDBC, custom connectors
  • Languages: Java, Scala, Python (PyFlink), SQL (via Flink Studio notebooks)
  • Processing: Windowed aggregations, joins, pattern detection (CEP), stateful transformations

Latency Comparison

Service Latency Type Typical Latency Best For
Kinesis Data Streams Real-time ~70 ms (EFO) / ~200 ms (shared) Sub-second event processing, real-time dashboards
Amazon Data Firehose Near real-time / Micro-batch ~5 seconds (zero buffering) / 60–900 seconds (standard) Log delivery, data lake loading, ETL pipelines
Managed Apache Flink Real-time (continuous processing) Milliseconds to seconds (application-dependent) Complex event processing, windowed analytics, real-time ML inference

Pricing Models (US East – N. Virginia)

Kinesis Data Streams Pricing

Mode Ingestion Retrieval Other Charges
Provisioned $0.014 per 1M PUT payload units Included (2 MB/s per shard) $0.015/shard-hour
On-Demand Standard $0.08/GB $0.04/GB $0.04/stream-hour
On-Demand Advantage $0.032/GB $0.016/GB No per-stream fee; 25 MB/s min commitment

Additional charges: Extended retention (24h–7d): $0.020/shard-hour (Provisioned) or $0.023/GB-month (Advantage). Long-term retention (7d–365d): $0.023/GB-month (Advantage) vs $0.10/GB-month (Standard). Enhanced Fan-Out: $0.013/shard-hour + $0.016/GB (Advantage has no EFO surcharge).

Amazon Data Firehose Pricing

  • Ingestion: Tiered pricing starting at ~$0.029/GB for first 500 TB/month (billed in 5 KB increments)
  • Format conversion: $0.018/GB (Parquet/ORC conversion)
  • VPC delivery: Additional charges for VPC endpoints
  • Dynamic partitioning: $0.020/GB
  • No charges for: Delivery to destinations, scaling, or idle streams

Managed Apache Flink Pricing

  • KPU-hour: $0.11/KPU-hour (1 KPU = 1 vCPU + 4 GB RAM)
  • Running application storage: $0.10/GB-month (for application state/checkpoints)
  • Durable application backups (snapshots): $0.023/GB-month
  • Orchestration: $0.011/orchestration-hour per KPU
  • Billed per second with a 1-minute minimum
  • Auto-scaling adjusts KPUs automatically — you pay only for KPUs consumed

Data Transformation Capabilities

Capability Kinesis Data Streams Amazon Data Firehose Managed Apache Flink
Built-in Transformation ❌ None (requires consumer code) ✅ Lambda-based transformation ✅ Full Apache Flink operators
Format Conversion ❌ Not built-in ✅ JSON → Parquet/ORC (via Glue schema) ✅ Any format (programmatic)
Compression ❌ Client-side only ✅ GZIP, Snappy, Zip, Hadoop Snappy ✅ Programmatic
Aggregation/Windowing ❌ Not built-in ❌ No (only batching by buffer) ✅ Tumbling, Sliding, Session windows
Joins ❌ Not built-in ❌ No ✅ Stream-stream & stream-table joins
Stateful Processing ❌ Consumer must manage state ❌ Stateless delivery ✅ Managed state with checkpointing
Complex Event Processing ❌ Custom code required ❌ No ✅ Pattern detection via Flink CEP

Ordering & Delivery Guarantees

Guarantee Kinesis Data Streams Amazon Data Firehose Managed Apache Flink
Ordering ✅ Per-shard ordering via partition key ⚠️ Best-effort ordering (no strict guarantee) ✅ Per-key ordering with watermarks and event-time processing
Delivery Semantics At-least-once (exactly-once with KCL deduplication) At-least-once Exactly-once for application state (with checkpointing)
Deduplication Consumer responsibility (sequence numbers) Not guaranteed (possible duplicates on retry) Built-in via Flink’s checkpoint mechanism
Data Durability 3 AZ replication, up to 365 days retention Retry with S3 backup for failed deliveries Checkpoints stored in durable storage, snapshots in S3

Comprehensive Comparison Table

Feature Kinesis Data Streams Amazon Data Firehose Managed Apache Flink
Type Data streaming & ingestion Data delivery & loading Stream processing & analytics
Management Semi-managed (shard management in Provisioned mode) Fully managed (zero admin) Fully managed (auto-scaling KPUs)
Scaling Shards (manual) or On-Demand (auto up to 10 GB/s) Automatic (unlimited, up to GB/s) KPU auto-scaling (default max 64 KPUs)
Latency ~70 ms (EFO) to ~200 ms (shared) ~5s (zero buffer) to 60–900s Milliseconds (continuous processing)
Data Retention 24 hours – 365 days None (pass-through) State in checkpoints/snapshots
Max Record Size 10 MiB (since Oct 2025) 1 MiB Limited by available memory
Consumers KCL, Lambda, SDK, EFO (up to 50) Built-in delivery (no custom consumers) Flink application sinks
Destinations Any (custom consumer code) S3, Redshift, OpenSearch, Iceberg, Snowflake, Splunk, HTTP endpoints Kinesis, S3, DynamoDB, OpenSearch, MSK, JDBC, custom connectors
Transformation None (consumer responsibility) Lambda functions, format conversion Full Flink operators (map, filter, join, window, CEP)
Ordering Per-shard (partition key) Best-effort Per-key with event-time semantics
Delivery Semantics At-least-once At-least-once Exactly-once (state)
Replay ✅ Yes ❌ No ✅ Yes (savepoints)
Pricing Model Per shard-hour OR per GB (on-demand) Per GB ingested (5 KB increments) Per KPU-hour + storage
Languages/SDK Any (via SDK/KCL) N/A (configuration-based) Java, Scala, Python, SQL
Use Case Custom real-time apps, multiple consumers, replay Simple data delivery/ETL to storage Complex analytics, CEP, ML inference, joins

When to Choose Each Service

Choose Kinesis Data Streams When:

  • You need sub-second latency (real-time processing with ~70 ms via EFO)
  • You need multiple consumers reading the same stream independently
  • You require data replay capability (reprocess historical data)
  • You need ordering guarantees per partition key
  • You need data retention beyond delivery (audit, compliance — up to 365 days)
  • You want to build custom processing logic with KCL, Lambda, or SDK
  • You need large record support (up to 10 MiB)
  • Your architecture needs a durable message bus between producers and multiple downstream services

Choose Amazon Data Firehose When:

  • You need the simplest path to load data into S3, Redshift, OpenSearch, or Iceberg
  • You want zero operational overhead — no code, no capacity planning
  • Near real-time delivery (seconds to minutes) is acceptable
  • You need built-in format conversion (JSON to Parquet/ORC)
  • You need automatic compression and batching before storage
  • You’re building a data lake ingestion pipeline with minimal complexity
  • You need to deliver to third-party destinations (Datadog, Splunk, Snowflake, etc.)
  • You don’t need replay, multiple consumers, or strict ordering

Choose Managed Apache Flink When:

  • You need complex stream processing — windowed aggregations, joins, pattern detection
  • You need exactly-once processing guarantees for application state
  • You’re building real-time analytics with SQL or programmatic queries
  • You need to join multiple streams or enrich streams with reference data
  • You need event-time processing with watermarks (handling late/out-of-order events)
  • You’re implementing real-time ML scoring or anomaly detection
  • You need stateful computations (running totals, session tracking, deduplication)
  • You want an interactive notebook environment for stream exploration (Flink Studio)

Common Architecture Pattern: Using All Three Together

These services are complementary, not mutually exclusive. A common production pattern:

  1. Kinesis Data Streams — Ingests and buffers high-volume event data from producers
  2. Managed Apache Flink — Reads from the stream, performs real-time analytics (windowed counts, anomaly detection, enrichment), writes results to another stream or DynamoDB
  3. Amazon Data Firehose — Consumes from Kinesis Data Streams and delivers raw/processed data to S3/Redshift data lake for batch analytics

AWS Certification Exam Practice Questions

Question 1:

A company collects clickstream data from its website at 500,000 events per second. The data must be processed in real-time to update a fraud detection dashboard with sub-second latency, while also being stored in S3 in Parquet format for batch analytics. Which architecture meets these requirements?

  1. Send data directly to Amazon Data Firehose with Lambda transformation and zero buffering enabled
  2. Send data to Kinesis Data Streams, use a Lambda consumer for real-time fraud detection, and configure Amazon Data Firehose as a second consumer to deliver to S3 in Parquet format
  3. Send data to Amazon Data Firehose with two delivery destinations: one for the dashboard and one for S3
  4. Send data directly to Amazon Managed Service for Apache Flink which writes to both the dashboard and S3
Show Answer

Answer: B –

Explanation: Kinesis Data Streams supports multiple consumers — Lambda provides sub-second processing for real-time fraud detection, while Firehose as a second consumer handles S3 delivery with Parquet format conversion. Firehose alone (A, C) cannot provide sub-second latency for the dashboard. Option D is possible but adds unnecessary complexity when Lambda suffices for the fraud detection logic.

Question 2:

A data engineering team needs to continuously aggregate IoT sensor readings into 5-minute tumbling windows, join the aggregated data with a reference table in DynamoDB, and detect anomalous patterns across multiple sensors. Which AWS service is BEST suited for this requirement?

  1. Amazon Kinesis Data Streams with KCL consumers
  2. Amazon Data Firehose with Lambda transformation
  3. Amazon Managed Service for Apache Flink
  4. AWS Lambda with Kinesis Data Streams event source mapping
Show Answer

Answer: C –

Explanation: Amazon Managed Service for Apache Flink (formerly Kinesis Data Analytics) is purpose-built for windowed aggregations, stream-table joins, and complex event processing (CEP). KCL (A) and Lambda (D) would require custom state management for windows. Firehose (B) cannot perform windowed aggregations or joins.

Question 3:

A company wants to stream application logs to Amazon OpenSearch Service with minimal operational overhead and within 60 seconds of generation. They do not need to reprocess or replay the data. Which is the MOST operationally efficient solution?

  1. Kinesis Data Streams with a KCL consumer that writes to OpenSearch
  2. Amazon Data Firehose with OpenSearch as the destination
  3. Amazon Managed Service for Apache Flink reading from Kinesis Data Streams and writing to OpenSearch
  4. Kinesis Data Streams with an AWS Lambda consumer writing to OpenSearch
Show Answer

Answer: B –

Explanation: Amazon Data Firehose provides the most operationally efficient solution — it natively supports OpenSearch as a destination, requires no custom code, auto-scales, and can deliver within 60 seconds with standard buffering (or ~5 seconds with zero buffering). Since replay is not needed, Kinesis Data Streams (A, D) adds unnecessary complexity. Managed Flink (C) is overkill for simple log delivery.

Question 4:

A company uses Kinesis Data Streams with 5 enhanced fan-out consumers. During peak traffic, Consumer A is experiencing high latency while Consumer B processes normally. What is the MOST likely cause?

  1. Enhanced fan-out consumers share throughput, so Consumer A is being throttled by Consumer B’s usage
  2. Consumer A has a processing bottleneck in its application code, causing its iterator to fall behind
  3. The stream has insufficient shards to support 5 enhanced fan-out consumers
  4. Enhanced fan-out only supports up to 2 consumers per stream
Show Answer

Answer: B –

Explanation: Enhanced fan-out provides dedicated 2 MB/s throughput per consumer per shard — consumers do NOT share throughput (eliminating A). Since Consumer B is fine, the stream has adequate shards (eliminating C). Enhanced fan-out supports up to 20 consumers (Standard/Provisioned) or 50 (Advantage mode), not 2 (eliminating D). The issue is Consumer A’s own processing performance.

Question 5:

A company is evaluating pricing for their streaming workload: 50 GB/hour of data ingested continuously, single consumer, 24-hour retention, in us-east-1. They want the lowest-cost Kinesis Data Streams option. Which capacity mode should they choose?

  1. Provisioned mode with calculated shard count
  2. On-Demand Standard mode
  3. On-Demand Advantage mode
  4. Amazon Data Firehose instead of Kinesis Data Streams
Show Answer

Answer: C –

Explanation: At 50 GB/hour (~13.9 MB/s), the workload exceeds On-Demand Advantage’s 25 MB/s minimum aggregate commitment threshold (when considering retrieval). On-Demand Advantage costs $0.032/GB ingest + $0.016/GB retrieval = ~$57.60/day, vs On-Demand Standard at $0.08/GB + $0.04/GB + stream-hour fees = ~$145/day+. Provisioned mode (A) could be cheaper for very predictable loads but requires operational overhead for shard management. Firehose (D) is a different service that doesn’t provide retention or replay capabilities the company may need.

Related Posts

References

Related Posts

AWS Data Analytics Services Cheat Sheet

AWS Data Analytics Services
~ Last updated on : ~ jayendrapatil ~ 7 Comments

AWS Data Analytics Services Cheat Sheet

📋 Last Updated: June 2026. This post has been updated to reflect service renamings (Kinesis Data Firehose → Amazon Data Firehose, Kinesis Data Analytics → Amazon Managed Service for Apache Flink, Elasticsearch → OpenSearch Service, QuickSight → Quick Suite), deprecations (AWS Data Pipeline, Kinesis Data Analytics for SQL), and major new features (Zero-ETL integrations, MSK Express brokers, Glue 5.0, SageMaker Lakehouse).

AWS Data Analytics Services

Kinesis Data Streams – KDS

  • enables real-time processing of streaming data at a massive scale
  • provides ordering of records per shard
  • provides an ability to read and/or replay records in the same order
  • allows multiple applications to consume the same data
  • data is replicated across three data centers within a region
  • data is preserved for 24 hours, by default, and can be extended to 365 days
  • data inserted in Kinesis, it can’t be deleted (immutability) but only expires
  • streams can be scaled using multiple shards, based on the partition key
  • each shard provides the capacity of 1MB/sec data input and 2MB/sec data output with 1000 PUT requests per second
  • supports two capacity modes:
    • Provisioned mode – you manage the number of shards
    • On-demand mode – automatically scales to accommodate up to 10 GB/s write and 20 GB/s read throughput per stream
  • On-demand Advantage mode (launched Nov 2025) – enables on-demand streams to handle instant throughput increases via warm throughput capability, with up to 10GB/s or 10 million events/second, eliminating over-provisioning needs and offering 60%+ cost savings for consistent workloads
  • supports record sizes up to 10 MiB (increased from 1 MiB in Oct 2025)
  • supports up to 50 enhanced fan-out consumers per stream (increased from 20 in Nov 2025)
  • Kinesis vs SQS
    • real-time processing of streaming big data vs reliable, highly scalable hosted queue for storing messages
    • ordered records, as well as the ability to read and/or replay records in the same order vs no guarantee on data ordering (with the standard queues before the FIFO queue feature was released)
    • data storage up to 24 hours, extended to 365 days vs 1 minute to extended to 14 days but cleared if deleted by the consumer.
    • supports multiple consumers vs a single consumer at a time and requires multiple queues to deliver messages to multiple consumers.
  • Kinesis Producer
    • API
      • PutRecord and PutRecords are synchronous
      • PutRecords uses batching and increases throughput
      • might experience ProvisionedThroughputExceeded Exceptions, when sending more data. Use retries with backoff, resharding, or change partition key.
    • KPL
      • producer supports synchronous or asynchronous use cases
      • supports inbuilt batching and retry mechanism
      • ⚠️ KPL 0.x reached end-of-support on January 30, 2026. Migrate to KPL 1.x.
    • Kinesis Agent can help monitor log files and send them to KDS
    • supports third-party libraries like Spark, Flume, Kafka connect, etc.
  • Kinesis Consumers
    • Kinesis SDK
      • Records are polled by consumers from a shard
    • Kinesis Client Library (KCL)
      • Read records from Kinesis produced with the KPL (de-aggregation)
      • supports the checkpointing feature to keep track of the application’s state and resume progress using the DynamoDB table.
      • if application receives provisioned-throughput exceptions, increase the provisioned throughput for the DynamoDB table
      • ⚠️ KCL 1.x reached end-of-support on January 30, 2026. Migrate to KCL 2.x.
    • Kinesis Connector Library – can be replaced using Firehose or Lambda
    • Third-party libraries: Spark, Log4J Appenders, Flume, Kafka Connect…
    • Amazon Data Firehose, AWS Lambda
    • Kinesis Consumer Enhanced Fan-Out
      • supports Multiple Consumer applications for the same Stream
      • provides Low Latency ~70ms
      • Higher costs
      • now supports up to 50 consumers per stream
  • Kinesis Security
    • allows access/authorization control using IAM policies
    • supports Encryption in flight using HTTPS endpoints
    • supports data encryption at rest using either server-side encryption with KMS or using client-side encryption before pushing the data to data streams.
    • supports VPC Endpoints to access within VPC

Amazon Data Firehose

(Previously known as Amazon Kinesis Data Firehose, renamed February 2024)

  • data transfer solution for delivering near real-time streaming data to destinations such as S3, Redshift, OpenSearch Service, Splunk, Snowflake, and other 3rd-party analytics services.
  • is a fully managed service that automatically scales to match the throughput of your data and requires no ongoing administration
  • is Near Real Time (min. 60 secs) as it buffers incoming streaming data to a certain size or for a certain period of time before delivering it
  • supports batching, compression, and encryption of the data before loading it, minimizing the amount of storage used at the destination and increasing security
  • supports data compression, minimizing the amount of storage used at the destination. It currently supports GZIP, ZIP, and SNAPPY compression formats. Only GZIP is supported if the data is further loaded to Redshift.
  • supports out of box data transformation as well as custom transformation using Lambda function to transform incoming source data and deliver the transformed data to destinations
  • uses at least once semantics for data delivery.
  • supports multiple producers as datasource, which include Kinesis data stream, KPL, Kinesis Agent, or the Data Firehose API using the AWS SDK, CloudWatch Logs, CloudWatch Events, or AWS IoT
  • does NOT support consumers like Spark and KCL
  • supports interface VPC endpoint to keep traffic between the VPC and Data Firehose from leaving the Amazon network.
  • Apache Iceberg Tables destination (launched 2024) – delivers streaming data directly into Apache Iceberg format tables in S3 and S3 Tables, supporting record routing to different Iceberg tables, CDC replication from databases, schema evolution, and ACID transactions.
  • Database CDC replication (Preview 2024) – supports continuous replication of database changes from MySQL and PostgreSQL directly into Apache Iceberg Tables in S3.

Kinesis Data Streams vs Amazon Data Firehose

Amazon Managed Service for Apache Flink

(Previously known as Amazon Kinesis Data Analytics, renamed August 2023)

⚠️ Kinesis Data Analytics for SQL was discontinued on January 27, 2026. Migrate to Amazon Managed Service for Apache Flink or Apache Flink Studio for real-time stream processing.

  • helps analyze streaming data, gain actionable insights, and respond to the business and customer needs in real time.
  • is a fully managed and serverless service for building and running real-time streaming applications using Apache Flink.
  • reduces the complexity of building, managing, and integrating streaming applications with other AWS services.
  • supports Apache Flink applications written in Java, Scala, Python, and SQL.
  • provides automatic scaling, high availability, and exactly-once processing semantics.
  • integrates with Kinesis Data Streams, Amazon MSK, and Amazon S3 as data sources and sinks.

Managed Streaming for Kafka – MSK

  • Managed Streaming for Kafka- MSK is an AWS streaming data service that manages Apache Kafka infrastructure and operations.
  • makes it easy for developers and DevOps managers to run Kafka applications and Kafka Connect connectors on AWS, without the need to become experts in operating Kafka.
  • operates, maintains, and scales Kafka clusters, provides enterprise-grade security features out of the box, and has built-in AWS integrations that accelerate development of streaming data applications.
  • always runs within a VPC managed by the MSK and is available to your own selected VPC, subnet, and security group when the cluster is setup.
  • IP addresses from the VPC are attached to the MSK resources through elastic network interfaces (ENIs), and all network traffic stays within the AWS network and is not accessible to the internet by default.
  • integrates with CloudWatch for monitoring, metrics, and logging.
  • MSK Serverless is a cluster type for MSK that makes it easy for you to run Apache Kafka clusters without having to manage compute and storage capacity.
  • MSK Express Brokers (GA November 2024) – a new broker type for MSK Provisioned designed to deliver:
    • up to 3x more throughput per broker (500 MBps ingress, 1000 MBps egress on m7g instances)
    • up to 20x faster scaling
    • 90% faster recovery from failures
    • up to 5x more partitions per broker
    • virtually unlimited storage with instant storage scaling
    • supports Intelligent Rebalancing for 180x faster operation performance
  • supports EBS server-side encryption using KMS to encrypt storage.
  • supports encryption in transit enabled via TLS for inter-broker communication.
  • For provisioned clusters, you have three options:
    • IAM Access Control for both AuthN/Z (recommended),
    • TLS certificate authentication (CA) for AuthN and access control lists for AuthZ
    • SASL/SCRAM for AuthN and access control lists for AuthZ.
  • For serverless clusters, IAM Access Control can be used for both authentication and authorization.

Redshift

  • Redshift is a fast, fully managed data warehouse
  • provides simple and cost-effective solutions to analyze all the data using standard SQL and the existing Business Intelligence (BI) tools.
  • manages the work needed to set up, operate, and scale a data warehouse, from provisioning the infrastructure capacity to automating ongoing administrative tasks such as backups, and patching.
  • automatically monitors your nodes and drives to help you recover from failures.
  • only supported Single-AZ deployments. However, now supports Multi-AZ deployments.
  • replicates all the data within the data warehouse cluster when it is loaded and also continuously backs up your data to S3.
  • attempts to maintain at least three copies of your data (the original and replica on the compute nodes and a backup in S3).
  • supports cross-region snapshot replication to another region for disaster recovery
  • Redshift supports four distribution styles; AUTO, EVEN, KEY, or ALL.
    • KEY distribution uses a single column as distribution key (DISTKEY) and helps place matching values on the same node slice
    • Even distribution distributes the rows across the slices in a round-robin fashion, regardless of the values in any particular column
    • ALL distribution replicates whole table in every compute node.
    • AUTO distribution lets Redshift assigns an optimal distribution style based on the size of the table data
  • Redshift supports Compound and Interleaved sort keys
    • Compound key
      • is made up of all of the columns listed in the sort key definition, in the order they are listed and is more efficient when query predicates use a prefix, or query’s filter applies conditions, such as filters and joins, which is a subset of the sort key columns in order.
    • Interleaved sort key
      • gives equal weight to each column in the sort key, so query predicates can use any subset of the columns that make up the sort key, in any order.
      • Not ideal for monotonically increasing attributes
  • Import/Export Data
    • UNLOAD helps copy data from Redshift table to S3
    • COPY command
      • helps copy data from S3 to Redshift
      • also supports EMR, DynamoDB, remote hosts using SSH
      • parallelized and efficient
      • can decrypt data as it is loaded from S3
      • DON’T use multiple concurrent COPY commands to load one table from multiple files as Redshift is forced to perform a serialized load, which is much slower.
      • supports data decryption when loading data, if data encrypted
      • supports decompressing data, if data is compressed.
    • Split the Load Data into Multiple Files
    • Load the data in sort key order to avoid needing to vacuum.
    • Use a Manifest File
      • provides Data consistency, to avoid S3 eventual consistency issues
      • helps specify different S3 locations in a more efficient way that with the use of S3 prefixes.
  • Zero-ETL Integrations (2024-2025)
    • enable near real-time analytics by connecting operational databases and applications to Redshift without building data pipelines
    • supports integrations from Aurora (MySQL/PostgreSQL), DynamoDB, RDS, and third-party applications (Salesforce, SAP, Zendesk)
    • works with both Redshift Serverless workgroups and provisioned clusters using RA3 instance types
    • includes SQL features: QUERY_ALL_STATES, TRUNCATECOLUMNS, and ACCEPTINVCHARS for zero-ETL data handling
    • integrates with Amazon SageMaker Lakehouse for unified analytics and AI/ML
  • Redshift Distribution Style determines how data is distributed across compute nodes and helps minimize the impact of the redistribution step by locating the data where it needs to be before the query is executed.
  • Redshift Enhanced VPC routing forces all COPY and UNLOAD traffic between the cluster and the data repositories through the VPC.
  • Workload management (WLM) enables users to flexibly manage priorities within workloads so that short, fast-running queries won’t get stuck in queues behind long-running queries.
  • Redshift Spectrum helps query and retrieve structured and semistructured data from files in S3 without having to load the data into Redshift tables.
    • Redshift Spectrum external tables are read-only. You can’t COPY or INSERT to an external table.
  • Federated Query feature allows querying and analyzing data across operational databases, data warehouses, and data lakes.
  • Short query acceleration (SQA) prioritizes selected short-running queries ahead of longer-running queries.
  • Redshift Serverless is a serverless option of Redshift that makes it more efficient to run and scale analytics in seconds without the need to set up and manage data warehouse infrastructure.

EMR

  • is a web service that utilizes a hosted Hadoop framework running on the web-scale infrastructure of EC2 and S3
  • launches all nodes for a given cluster in the same Availability Zone, which improves performance as it provides a higher data access rate.
  • seamlessly supports Reserved, On-Demand, and Spot Instances
  • consists of Master/Primary Node for management and Slave nodes, which consist of Core nodes holding data and providing compute and Task nodes for performing tasks only.
  • is fault tolerant for slave node failures and continues job execution if a slave node goes down
  • supports Persistent and Transient cluster types
    • Persistent EMR clusters continue to run after the data processing job is complete
    • Transient EMR clusters shut down when the job or the steps (series of jobs) are complete
  • supports EMRFS which allows S3 to be used as a durable HA data storage
  • EMR Serverless helps run big data frameworks such as Apache Spark and Apache Hive without configuring, managing, and scaling clusters.
    • now supports Spark Connect (2026) for interactive PySpark development from local environments, IDEs, and SageMaker Unified Studio Notebooks
    • eliminates local storage provisioning, reducing costs by up to 20%
  • Apache Spark 4.0 support (GA 2026) – includes VARIANT data type, state-management improvements, and Spark Connect, with EMR optimized runtime running workloads up to 4.5x faster than open-source Spark
  • EMR Studio is an IDE that helps data scientists and data engineers to develop, visualize, and debug data engineering and data science applications written in R, Python, Scala, and PySpark.
  • EMR Notebooks provide a managed environment, based on Jupyter Notebook, that helps prepare and visualize data, collaborate with peers, build applications, and perform interactive analysis using EMR clusters.

Glue

  • AWS Glue is a fully managed, ETL service that automates the time-consuming steps of data preparation for analytics.
  • is serverless and supports a pay-as-you-go model.
  • handles provisioning, configuration, and scaling of the resources required to run the ETL jobs on a fully managed, scale-out Apache Spark environment.
  • helps setup, orchestrate, and monitor complex data flows.
  • supports custom Scala or Python code and import custom libraries and Jar files into the AWS Glue ETL jobs to access data sources not natively supported by AWS Glue.
  • supports server side encryption for data at rest and SSL for data in motion.
  • provides development endpoints to edit, debug, and test the code it generates.
  • AWS Glue natively supports data stored in RDS, Redshift, DynamoDB, S3, MySQL, Oracle, Microsoft SQL Server, and PostgreSQL databases in the VPC running on EC2 and Data streams from MSK, Kinesis Data Streams, and Apache Kafka.
  • Glue ETL engine to Extract, Transform, and Load data that can automatically generate Scala or Python code.
  • AWS Glue 5.0/5.1 (2024-2026):
    • provides performance-optimized Apache Spark 3.5 runtime for batch and stream processing
    • native support for open table formats: Apache Iceberg, Delta Lake, and Apache Hudi
    • Spark-native fine-grained access control (FGAC) integration with AWS Lake Formation
    • faster job start times and automatic partition pruning
    • Glue 5.1 adds support for Apache Iceberg format version 3.0, deletion vectors, and row lineage tracking
    • new worker types: G.12X, G.16X general compute, and R.1X/R.2X/R.4X/R.8X memory-optimized
  • Glue Materialized Views (2025) – Apache Iceberg-based materialized views for transforming data and accelerating query performance
  • supports generative AI assistance for data integration tasks
  • Glue Data Catalog is a central repository and persistent metadata store to store structural and operational metadata for all the data assets.
  • Glue Crawlers scan various data stores to automatically infer schemas and partition structures to populate the Data Catalog with corresponding table definitions and statistics.
  • Glue Job Bookmark tracks data that has already been processed during a previous run of an ETL job by persisting state information from the job run.
  • AWS Glue Streaming ETL enables performing ETL operations on streaming data using continuously-running jobs.
  • Glue provides flexible scheduler that handles dependency resolution, job monitoring, and retries.
  • Glue Studio offers a graphical interface for authoring AWS Glue jobs to process data allowing you to define the flow of the data sources, transformations, and targets in the visual interface and generating Apache Spark code on your behalf.
  • Glue Data Quality helps reduces manual data quality effort by automatically measuring and monitoring the quality of data in data lakes and pipelines.
  • Glue DataBrew is a visual data preparation tool that makes it easy for data analysts and data scientists to prepare, visualize, clean, and normalize terabytes, and even petabytes of data directly from your data lake, data warehouses, and databases, including S3, Redshift, Aurora, and RDS.
  • ⚠️ AWS Glue for Ray will no longer accept new customers starting April 30, 2026. Existing customers can continue using the service. Explore Amazon EKS for similar capabilities.

Lake Formation

  • AWS Lake Formation helps create secure data lakes, making data available for wide-ranging analytics.
  • is an integrated data lake service that helps to discover, ingest, clean, catalog, transform, and secure data and make it available for analysis and ML.
  • automatically manages access to the registered data in S3 through services including AWS Glue, Athena, Redshift, QuickSight, and EMR using Zeppelin notebooks with Apache Spark to ensure compliance with your defined policies.
  • helps configure and manage your data lake without manually integrating multiple underlying AWS services.
  • uses a shared infrastructure with AWS Glue, including console controls, ETL code creation and job monitoring, blueprints to create workflows for data ingest, the same data catalog, and a serverless architecture.
  • can manage data ingestion through AWS Glue. Data is automatically classified, and relevant data definitions, schema, and metadata are stored in the central Glue Data Catalog. Once the data is in the S3 data lake, access policies, including table-and-column-level access controls can be defined, and encryption for data at rest enforced.
  • integrates with IAM so authenticated users and roles can be automatically mapped to data protection policies that are stored in the data catalog. The IAM integration also supports Microsoft Active Directory or LDAP to federate into IAM using SAML.
  • helps centralize data access policy controls. Users and roles can be defined to control access, down to the table and column level.
  • supports fine-grained access control (FGAC) including row-level and cell-level security, and tag-based access control (LF-Tags) for scalable permission management.
  • supports private endpoints in the VPC and records all activity in AWS CloudTrail for network isolation and auditability.
  • ⚠️ Lake Formation’s Governed Tables feature was deprecated in February 2025. Use Apache Iceberg tables with Lake Formation for transactional data lake capabilities.

Amazon Quick Suite (formerly QuickSight)

(Amazon QuickSight evolved to Amazon Quick Suite on October 9, 2025, expanding from a single BI product to a comprehensive analytics and AI platform)

  • is a cloud-powered business analytics service that integrates BI capabilities with AI agents for business insights, deep research, and automation in one unified experience.
  • delivers fast and responsive query performance by using a robust in-memory engine (SPICE).
    • “SPICE” stands for a Super-fast, Parallel, In-memory Calculation Engine
    • can also be configured to keep the data in SPICE up-to-date as the data in the underlying sources change.
    • automatically replicates data for high availability and enables Quick Suite to scale to support users to perform simultaneous fast interactive analysis across a wide variety of AWS data sources.
  • Amazon Q in QuickSight (GA April 2024) – generative BI capabilities powered by Amazon Bedrock:
    • multi-visual data Q&A for asking questions of data not in dashboards
    • executive summaries for quick trend and insight discovery
    • automated stories – documents and slides explaining data
    • natural language generation for pixel-perfect reports
    • available to all Enterprise Edition users without separate Q add-on
  • supports
    • Excel files and flat files like CSV, TSV, CLF, ELF
    • on-premises databases like PostgreSQL, SQL Server and MySQL
    • SaaS applications like Salesforce
    • and AWS data sources such as Redshift, RDS, Aurora, Athena, and S3
  • supports various functions to format and transform the data.
  • supports assorted visualizations that facilitate different analytical approaches:
    • Comparison and distribution – Bar charts (several assorted variants)
    • Changes over time – Line graphs, Area line charts
    • Correlation – Scatter plots, Heat maps
    • Aggregation – Pie graphs, Tree maps
    • Tabular – Pivot tables
  • Amazon Quick Sight (a capability within Quick Suite) now offers visual data preparation experience for advanced data transformations without code.

Data Pipeline

⚠️ AWS Data Pipeline – No Longer Available to New Customers (July 25, 2024)

AWS Data Pipeline is in maintenance mode and is no longer available to new customers. Console access was removed on April 30, 2023. Existing customers can continue to use the service via CLI and API.

Migration Options:

  • Amazon MWAA (Managed Workflows for Apache Airflow) – for complex workflow orchestration
  • AWS Step Functions – for serverless workflow orchestration
  • AWS Glue – for ETL-focused data movement pipelines
  • Amazon EventBridge – for event-driven scheduling
  • orchestration service that helps define data-driven workflows to automate and schedule regular data movement and data processing activities
  • integrates with on-premises and cloud-based storage systems
  • allows scheduling, retry, and failure logic for the workflows

Amazon OpenSearch Service

(Previously known as Amazon Elasticsearch Service, renamed September 8, 2021)

  • Amazon OpenSearch Service is a managed service that makes it easy to deploy, operate, and scale OpenSearch clusters in the AWS Cloud.
  • OpenSearch Service provides
    • real-time, distributed search and analytics engine
    • ability to provision all the resources for OpenSearch cluster and launches the cluster
    • easy to use cluster scaling options. Scaling OpenSearch Service domain by adding or modifying instances, and storage volumes is an online operation that does not require any downtime.
    • provides self-healing clusters, which automatically detects and replaces failed nodes, reducing the overhead associated with self-managed infrastructures
    • domain snapshots to back up and restore domains and replicate domains across AZs
    • enhanced security with IAM, Network, Domain access policies, and fine-grained access control
    • storage volumes for the data using EBS volumes
    • ability to span cluster nodes across multiple AZs in the same region, known as zone awareness, for high availability and redundancy. OpenSearch Service automatically distributes the primary and replica shards across instances in different AZs.
    • dedicated master nodes to improve cluster stability
    • data visualization using OpenSearch Dashboards (formerly Kibana)
    • integration with CloudWatch for monitoring domain metrics
    • integration with CloudTrail for auditing configuration API calls to domains
    • integration with S3, Kinesis, and DynamoDB for loading streaming data
    • ability to handle structured and Unstructured data
    • supports encryption at rest through KMS, node-to-node encryption over TLS, and the ability to require clients to communicate with HTTPS
  • Amazon OpenSearch Serverless
    • automatically scales without managing infrastructure
    • NextGen architecture (2026) – decoupled compute from storage, provisions in seconds, scales to zero when idle, up to 20x faster autoscaling, and up to 60% lower cost than provisioned clusters
    • two collection architectures: Classic (original) and NextGen (default for new collections)
  • Vector Database Capabilities
    • stores vector embeddings from LLMs for semantic/similarity search
    • supports hybrid search combining vector, lexical, and agentic retrieval
    • GPU-accelerated vector indexes for billion-scale databases (2025)
    • auto-optimized vector indexes for search quality/speed/cost tradeoffs
    • integrates with Amazon Bedrock for RAG and agentic AI applications
  • Zero-ETL integrations – direct data access from other AWS services without pipeline management
  • Extended Support – Standard Support ends Nov 7, 2025 for legacy Elasticsearch versions up to 6.7, ES 7.1-7.8, and OpenSearch 1.0-1.2

Athena

  • Amazon Athena is a serverless, interactive analytics service built on open-source frameworks, supporting open-table and file formats.
  • provides a simplified, flexible way to analyze petabytes of data in an S3 data lake and 30 data sources, including on-premises data sources or other cloud systems using SQL or Python without loading the data.
  • is built on open-source Trino and Presto engines and Apache Spark frameworks, with no provisioning or configuration effort required.
  • is highly available and runs queries using compute resources across multiple facilities, automatically routing queries appropriately if a particular facility is unreachable
  • can process unstructured, semi-structured, and structured datasets.
  • integrates with QuickSight for visualizing the data or creating dashboards.
  • supports various standard data formats, including CSV, TSV, JSON, ORC, Avro, and Parquet.
  • supports compressed data in Snappy, Zlib, LZO, and GZIP formats. You can improve performance and reduce costs by compressing, partitioning, and using columnar formats.
  • can handle complex analysis, including large joins, window functions, and arrays
  • uses a managed Glue Data Catalog to store information and schemas about the databases and tables that you create for the data stored in S3
  • uses schema-on-read technology, which means that the table definitions are applied to the data in S3 when queries are being applied. There’s no data loading or transformation required. Table definitions and schema can be deleted without impacting the underlying data stored in S3.
  • supports fine-grained access control with AWS Lake Formation which allows for centrally managing permissions and access control for data catalog resources in the S3 data lake.
  • integrates with Amazon SageMaker Lakehouse for governed federated queries across data sources

Amazon SageMaker Lakehouse

(Launched at re:Invent 2024, GA March 2025)

  • part of the next generation of Amazon SageMaker – a unified platform for data, analytics, and AI
  • unifies data across S3 data lakes (including S3 Tables), Redshift data warehouses, and operational databases
  • supports zero-ETL integrations from Aurora, DynamoDB, RDS, and third-party applications (Salesforce, SAP, Zendesk) for near real-time data access
  • enables querying, analyzing, and joining data using Redshift, Athena, EMR, and AWS Glue
  • provides unified access through Amazon SageMaker Unified Studio – a single development experience for data engineers, data scientists, and analysts
  • supports Apache Iceberg open table format for interoperability
  • integrates with Lake Formation for fine-grained access control and governance