AWS

AWS SQS Standard vs FIFO Queue – Ordering & Dedup

SQS Standard vs FIFO Queues
~ Last updated on : ~ jayendrapatil ~ 6 Comments

AWS SQS Standard vs FIFO Queue

SQS offers two types of queues – Standard & FIFO queues

SQS Standard vs FIFO Queues

SQS Standard vs FIFO Queue Features

Message Order

  • Standard queues provide best-effort ordering which ensures that messages are generally delivered in the same order as they are sent. Occasionally (because of the highly-distributed architecture that allows high throughput), more than one copy of a message might be delivered out of order
  • FIFO queues offer first-in-first-out delivery and exactly-once processing: the order in which messages are sent and received is strictly preserved

Delivery

  • Standard queues guarantee that a message is delivered at least once and duplicates can be introduced into the queue
  • FIFO queues ensure a message is delivered exactly once and remains available until a consumer processes and deletes it; duplicates are not introduced into the queue

Transactions Per Second (TPS)

  • Standard queues allow nearly-unlimited number of transactions per second
  • FIFO queues by default are limited to 300 transactions per second per API action (SendMessage, ReceiveMessage, DeleteMessage).
  • With High Throughput Mode enabled, FIFO queues can support up to 70,000 TPS per API action without batching in select regions (US East N. Virginia, US West Oregon, Europe Ireland), and up to 700,000 messages per second with batching.
  • High Throughput Mode can be enabled from the SQS console and uses message group-level partitioning to achieve higher throughput.
  • In other regions, high throughput quotas vary (up to 18,000 TPS in several regions).

In-Flight Messages

  • Standard queues support approximately 120,000 in-flight messages.
  • FIFO queues now support up to 120,000 in-flight messages (increased from 20,000 in November 2024). In-flight messages are those received by a consumer but not yet deleted from the queue.

Regions

  • Standard & FIFO queues are available in all regions where Amazon SQS is available.

SQS Buffered Asynchronous Client

  • FIFO queues are not compatible with the SQS Buffered Asynchronous Client, where messages are buffered at the client side and sent as a single request to the SQS queue to reduce cost.

Dead-Letter Queue (DLQ) Support

  • Both Standard and FIFO queues support dead-letter queues for handling messages that cannot be processed after a configured number of retries.
  • FIFO queues now support DLQ redrive, allowing messages to be moved from a FIFO dead-letter queue back to the FIFO source queue or a custom FIFO destination queue (launched 2023, expanded to GovCloud in April 2024).

CloudWatch Metrics

  • Standard queues support all standard SQS CloudWatch metrics.
  • FIFO queues support additional metrics (added July 2024):
    • NumberOfDeduplicatedSentMessages – tracks deduplicated messages
    • ApproximateNumberOfGroupsWithInflightMessages – tracks active message groups

AWS Services Supported

  • Standard Queues are supported by all AWS services
  • FIFO Queues now have broader service integration than at launch, but some limitations remain:
    • Supported:
      • Amazon SNS FIFO Topics (can subscribe SQS FIFO queues for ordered fan-out)
      • AWS Lambda (SQS FIFO as event source mapping, with ordered processing per message group)
      • Amazon EventBridge (SQS FIFO as rule target; EventBridge Pipes supports FIFO as source)
      • Auto Scaling Lifecycle Hooks (SQS queue target)
    • Not Supported:
      • S3 Event Notifications (cannot directly target SQS FIFO; use EventBridge as intermediary)
      • Lambda Asynchronous Invocation Destinations (does not support SQS FIFO or SNS FIFO as destination)

Use Cases

  • Standard queues can be used in any scenario, as long as the application can process messages that arrive more than once and out of order
    • Decouple live user requests from intensive background work: Let users upload media while resizing or encoding it.
    • Allocate tasks to multiple worker nodes: Process a high number of credit card validation requests.
    • Batch messages for future processing: Schedule multiple entries to be added to a database.
  • FIFO queues are designed to enhance messaging between applications when the order of operations and events is critical, or where duplicates can’t be tolerated
    • Ensure that user-entered commands are executed in the right order.
    • Display the correct product price by sending price modifications in the right order.
    • Prevent a student from enrolling in a course before registering for an account.
    • E-commerce order management systems where order processing sequence is critical.
    • Online ticketing systems where tickets are distributed on a first-come-first-served basis.

AWS Certification Exam Practice Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  1. A restaurant reservation application needs the ability to maintain a waiting list. When a customer tries to reserve a table, and none are available, the customer must be put on the waiting list, and the application must notify the customer when a table becomes free. What service should the Solutions Architect recommend ensuring that the system respects the order in which the customer requests are put onto the waiting list?
    1. Amazon SNS
    2. AWS Lambda with sequential dispatch
    3. A FIFO queue in Amazon SQS
    4. A standard queue in Amazon SQS
  2. A solutions architect is designing an application for a two-step order process. The first step is synchronous and must return to the user with little latency. The second step takes longer, so it will be implemented in a separate component. Orders must be processed exactly once and in the order in which they are received. How should the solutions architect integrate these components?
    1. Use Amazon SQS FIFO queues.
    2. Use an AWS Lambda function along with Amazon SQS standard queues.
    3. Create an SNS topic and subscribe an Amazon SQS FIFO queue to that topic.
    4. Create an SNS topic and subscribe an Amazon SQS Standard queue to that topic.
  3. A company needs to process over 50,000 messages per second with strict ordering within each customer’s message stream. The system uses Amazon SQS FIFO queues. What should a solutions architect recommend to meet the throughput requirement?
    1. Use multiple standard queues with application-level ordering
    2. Use Amazon Kinesis Data Streams instead of SQS
    3. Enable High Throughput Mode on the FIFO queue and use unique message group IDs per customer
    4. Increase the visibility timeout to allow more concurrent processing
  4. An application sends S3 event notifications to an SQS FIFO queue for ordered processing of uploaded files. The team reports that messages are not being delivered. What is the most likely cause?
    1. The SQS FIFO queue has reached its throughput limit
    2. S3 Event Notifications do not support SQS FIFO queues as a direct destination
    3. The queue’s message deduplication ID is not configured
    4. The IAM role lacks permissions to publish to the queue

References

Kinesis Data Streams vs SQS – Streaming vs Queuing

Kinesis Data Streams vs SQS
~ Last updated on : ~ jayendrapatil ~ 6 Comments

Kinesis Data Streams vs SQS

Purpose

  • Amazon Kinesis Data Streams
    • allows real-time processing of streaming big data and the ability to read and replay records to multiple Amazon Kinesis Applications.
    • Amazon Kinesis Client Library (KCL) delivers all records for a given partition key to the same record processor, making it easier to build multiple applications that read from the same Amazon Kinesis stream (for example, to perform counting, aggregation, and filtering).
    • designed for high-volume, real-time data ingestion and processing with multiple concurrent consumers reading the same data.
  • Amazon SQS
    • offers a reliable, highly-scalable hosted queue for storing messages as they travel between applications or microservices.
    • It moves data between distributed application components and helps decouple these components.
    • provides common middleware constructs such as dead-letter queues, poison-pill management, and dead-letter queue redrive (including FIFO DLQ redrive).
    • provides a generic web services API and can be accessed by any programming language that the AWS SDK supports.
    • supports both standard and FIFO queues
    • supports Fair Queues (launched Jul 2025) for standard queues to mitigate noisy neighbor impact in multi-tenant systems.

Scaling

  • Kinesis Data Streams offers three capacity modes:
    • Provisioned mode – requires manual shard management and scaling.
    • On-demand Standard mode – automatically scales throughput (up to 200 MB/s write), eliminating the need for manual shard provisioning.
    • On-demand Advantage mode (launched Nov 2025) – provides warm throughput for instant scaling to handle traffic surges up to 10 GB/s, with 60%+ lower pricing compared to On-demand Standard for high-volume workloads.
  • SQS is fully managed, highly scalable and requires no administrative overhead and little configuration. It scales transparently to handle any volume of messages.
  • SQS FIFO queues support High Throughput mode with up to 70,000 transactions per second per API action (without batching), and up to 700,000 messages per second with batching in select regions.

Ordering

  • Kinesis provides ordering of records within a shard (by partition key), as well as the ability to read and/or replay records in the same order to multiple Kinesis Applications
  • SQS Standard Queue provides best-effort ordering but does not guarantee strict data ordering and provides at least once delivery of messages
  • SQS FIFO Queue guarantees strict data ordering within the message group

Data Retention Period

  • Kinesis Data Streams stores the data for up to 24 hours, by default, and can be extended to 365 days (with extended retention up to 7 days, and long-term retention from 7 to 365 days)
  • SQS stores the message for up to 4 days, by default, and can be configured from 1 minute to 14 days but clears the message once deleted by the consumer

Delivery Semantics

  • Kinesis and SQS Standard Queue both guarantee at least one delivery of the message.
  • SQS FIFO Queue guarantees Exactly once delivery (exactly-once processing via deduplication)

Parallel Clients

  • Kinesis supports multiple consumers reading from the same stream simultaneously
    • With shared throughput (GetRecords), all consumers share the 2 MB/s per shard read capacity
    • With Enhanced Fan-Out, each consumer gets a dedicated 2 MB/s per shard throughput via SubscribeToShard (push-based)
    • On-demand Advantage mode supports up to 50 enhanced fan-out consumers per stream (vs 20 on On-demand Standard or Provisioned)
  • SQS allows the messages to be delivered to only one consumer at a time and requires multiple queues to deliver messages to multiple consumers
  • SQS Fair Queues (Jul 2025) dynamically reorder message delivery to ensure fair processing across tenants when one tenant becomes a noisy neighbor

Message/Record Size

  • Kinesis supports a maximum record size of 1 MB per data record
  • SQS supports a maximum message payload size of 1 MiB (increased from 256 KiB in August 2025). For larger payloads, the Amazon SQS Extended Client Library can be used to store the payload in S3.

Throughput

  • Kinesis Data Streams
    • Provisioned: 1 MB/s write and 2 MB/s read per shard
    • On-demand Standard: automatically scales, default 4 MB/s write, can burst to 200 MB/s
    • On-demand Advantage: supports instant scaling with warm throughput up to 10 GB/s
  • SQS
    • Standard queues: nearly unlimited throughput (no per-queue limits)
    • FIFO queues: 300 TPS default, up to 70,000 TPS with High Throughput mode enabled

Integration with AWS Lambda

  • Kinesis integrates with Lambda via event source mapping with parallelization factor, tumbling windows, and failure handling (bisect on error, max retry)
  • SQS integrates with Lambda via event source mapping with support for batch windows and Provisioned Mode (Nov 2025) that provides 3x faster scaling (up to 1,000 concurrent executions per minute) and 16x higher concurrency (up to 20,000 concurrent executions)

Use Cases

  • Kinesis use cases requirements
    • Ordering of records.
    • Ability to consume records in the same order a few hours later (data replay)
    • Ability for multiple applications to consume the same stream concurrently
    • Routing related records to the same record processor (as in streaming MapReduce)
    • Real-time analytics, log and event data aggregation, IoT telemetry ingestion
  • SQS uses cases requirements
    • Messaging semantics like message-level ack/fail and visibility timeout
    • Leveraging SQS’s ability to scale transparently
    • Dynamically increasing concurrency/throughput at read time
    • Individual message delay, which can be delayed
    • Multi-tenant workloads requiring fair message processing (Fair Queues)
    • Decoupling microservices and serverless event-driven architectures

Kinesis Data Streams vs SQS Comparison Table

Feature Kinesis Data Streams SQS
Primary Use Case Real-time streaming data processing Message queuing and decoupling
Ordering Per-shard (partition key) Best-effort (Standard) / Strict per message group (FIFO)
Delivery At least once At least once (Standard) / Exactly once (FIFO)
Retention 24 hours to 365 days 1 minute to 14 days
Multiple Consumers Yes (shared or enhanced fan-out) No (single consumer per message)
Max Record/Message Size 1 MB 1 MiB
Scaling Provisioned (manual) / On-demand (automatic) Fully automatic
Data Replay Yes No (message deleted after processing)
Provisioning Three modes: Provisioned, On-demand Standard, On-demand Advantage Fully serverless, no provisioning
Fair Processing N/A Fair Queues for multi-tenant workloads

AWS Certification Exam Practice Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  1. You are deploying an application to track GPS coordinates of delivery trucks in the United States. Coordinates are transmitted from each delivery truck once every three seconds. You need to design an architecture that will enable real-time processing of these coordinates from multiple consumers. Which service should you use to implement data ingestion?
    1. Amazon Kinesis
    2. AWS Data Pipeline
    3. Amazon AppStream
    4. Amazon Simple Queue Service
  2. Your customer is willing to consolidate their log streams (access logs, application logs, security logs etc.) in one single system. Once consolidated, the customer wants to analyze these logs in real time based on heuristics. From time to time, the customer needs to validate heuristics, which requires going back to data samples extracted from the last 12 hours? What is the best approach to meet your customer’s requirements?
    1. Send all the log events to Amazon SQS. Setup an Auto Scaling group of EC2 servers to consume the logs and apply the heuristics.
    2. Send all the log events to Amazon Kinesis develop a client process to apply heuristics on the logs (Can perform real time analysis and stores data for 24 hours which can be extended to 365 days)
    3. Configure Amazon CloudTrail to receive custom logs, use EMR to apply heuristics the logs (CloudTrail is only for auditing)
    4. Setup an Auto Scaling group of EC2 syslogd servers, store the logs on S3 use EMR to apply heuristics on the logs (EMR is for batch analysis)
  3. A company runs a multi-tenant SaaS application where different customers submit varying volumes of jobs to an SQS queue. During peak hours, one large customer floods the queue with messages, causing increased dwell time for all other customers. Which SQS feature should the team enable to address this noisy neighbor problem?
    1. SQS FIFO High Throughput mode
    2. SQS Long Polling
    3. SQS Fair Queues (Fair Queues dynamically reorder message delivery to mitigate noisy neighbor impact in multi-tenant standard queues)
    4. SQS Dead-Letter Queue Redrive
  4. A streaming analytics application needs to process real-time clickstream data with five independent consumer applications reading from the same stream simultaneously, each requiring dedicated throughput. The team wants to minimize operational overhead. Which configuration is most appropriate?
    1. Amazon SQS with five separate queues using SNS fan-out
    2. Kinesis Data Streams in Provisioned mode with Enhanced Fan-Out
    3. Kinesis Data Streams in On-demand Advantage mode with Enhanced Fan-Out (On-demand Advantage provides automatic scaling with no shard management and supports up to 50 enhanced fan-out consumers with dedicated 2 MB/s per shard throughput)
    4. Kinesis Data Streams in On-demand Standard mode with shared GetRecords
  5. An application processes order events that are each approximately 500 KB in size. The events need to be placed in a queue for asynchronous processing by a Lambda function. Which approach meets the requirements with the LEAST operational overhead?
    1. Use SQS with the Extended Client Library to store messages in S3
    2. Use SQS directly, as it now supports message payloads up to 1 MiB (Since August 2025, SQS supports up to 1 MiB message payload natively, eliminating the need for S3 offloading for messages under 1 MiB)
    3. Use Kinesis Data Streams with 1 MB record limit
    4. Use Amazon SNS to fan out to multiple SQS queues

References

AWS Kinesis Data Streams – Real-time Data Ingestion

~ Last updated on : ~ jayendrapatil ~ 22 Comments

AWS Kinesis Data Streams – KDS

  • Amazon Kinesis Data Streams is a streaming data service that enables real-time processing of streaming data at a massive scale.
  • Kinesis Streams enables building of custom applications that process or analyze streaming data for specialized needs.
  • Kinesis Streams features
    • handles provisioning, deployment, ongoing-maintenance of hardware, software, or other services for the data streams.
    • manages the infrastructure, storage, networking, and configuration needed to stream the data at the level of required data throughput.
    • synchronously replicates data across three AZs in an AWS Region, providing high availability and data durability.
    • stores records of a stream for up to 24 hours, by default, from the time they are added to the stream. The limit can be raised to up to 365 days by enabling extended data retention.
  • Data such as clickstreams, application logs, social media, etc can be added from multiple sources and within seconds is available for processing to the Kinesis Applications.
  • Kinesis provides the ordering of records, as well as the ability to read and/or replay records in the same order to multiple applications.
  • Kinesis is designed to process streaming big data and the pricing model allows heavy PUTs rate.
  • Multiple Kinesis Data Streams applications can consume data from a stream, so that multiple actions, like archiving and processing, can take place concurrently and independently.
  • Kinesis Data Streams application can start consuming the data from the stream almost immediately after the data is added and put-to-get delay is typically less than 1 second.
  • Kinesis Streams is useful for rapidly moving data off data producers and then continuously processing the data, be it to transform the data before emitting to a data store, run real-time metrics and analytics, or derive more complex data streams for further processing
    • Accelerated log and data feed intake: Data producers can push data to Kinesis stream as soon as it is produced, preventing any data loss and making it available for processing within seconds.
    • Real-time metrics and reporting: Metrics can be extracted and used to generate reports from data in real-time.
    • Real-time data analytics: Run real-time streaming data analytics.
    • Complex stream processing: Create Directed Acyclic Graphs (DAGs) of Kinesis Applications and data streams, with Kinesis applications adding to another Amazon Kinesis stream for further processing, enabling successive stages of stream processing.
  • Kinesis limits
    • stores records of a stream for up to 24 hours, by default, which can be extended to max 365 days
    • maximum size of a data blob (the data payload before Base64-encoding) within one record is 1 MB by default, configurable up to 10 MB
    • Each shard can support up to 1000 PUT records per second.
  • S3 is a cost-effective way to store the data, but not designed to handle a stream of data in real-time

Kinesis Data Streams Terminology

 

Kinesis Architecture

 

  • Data Record
    • A record is the unit of data stored in a Kinesis data stream.
    • A record is composed of a sequence number, partition key, and data blob, which is an immutable sequence of bytes.
    • Maximum size of a data blob is 1 MB by default, configurable up to 10 MB (announced Oct 2025).
    • Partition key
      • Partition key is used to segregate and route records to different shards of a stream.
      • A partition key is specified by the data producer while adding data to a Kinesis stream.
    • Sequence number
      • A sequence number is a unique identifier for each record.
      • Kinesis assigns a Sequence number, when a data producer calls PutRecord or PutRecords operation to add data to a stream.
      • Sequence numbers for the same partition key generally increase over time; the longer the time period between PutRecord or PutRecords requests, the larger the sequence numbers become.
  • Data Stream
    • Data stream represents a group of data records.
    • Data records in a data stream are distributed into shards.
  • Shard
    • A shard is a uniquely identified sequence of data records in a stream.
    • Streams are made of shards and are the base throughput unit of a Kinesis stream, as pricing is per shard basis.
    • Each shard supports up to 5 transactions per second for reads, up to a maximum total data read rate of 2 MB per second, and up to 1,000 records per second for writes, up to a maximum total data write rate of 1 MB per second (including partition keys)
    • Each shard provides a fixed unit of capacity. If the limits are exceeded, either by data throughput or the number of PUT records, the put data call will be rejected with a ProvisionedThroughputExceeded exception.
    • This can be handled by
      • Implementing a retry on the data producer side, if this is due to a temporary rise of the stream’s input data rate
      • Dynamically scaling the number of shared (resharding) to provide enough capacity for the put data calls to consistently succeed
  • Capacity Mode
    • A data stream capacity mode determines the pricing and how the capacity is managed
    • Kinesis Data Streams currently supports three capacity modes:
      • On-demand Standard
        • KDS automatically manages the shards in order to provide the necessary throughput.
        • You are charged only for the actual throughput used and KDS automatically accommodates the workloads’ throughput needs as they ramp up or down.
      • On-demand Advantage (launched Nov 2025)
        • Enables on-demand streams to handle instant throughput increases up to 10GB/s or 10 million events per second using warm throughput capability.
        • Offers 60% lower data pricing compared to On-demand Standard (ingest at $0.032/GB, retrieval at $0.016/GB).
        • Supports up to 50 enhanced fan-out consumers per stream (vs 20 for Standard/Provisioned).
        • Best suited for use cases ingesting at least 10 MiB/s in aggregate or having hundreds of data streams in a Region.
      • Provisioned mode
        • Number of shards for the data stream must be specified.
        • Total capacity of a data stream is the sum of the capacities of its shards.
        • Shards can be increased or decreased in a data stream as needed and you are charged for the number of shards at an hourly rate.
  • Retention Period
    • All data is stored for 24 hours, by default, and can be increased to 8760 hours (365 days) maximum.
  • Producers
    • A producer puts data records into Kinesis data streams.
    • To put data into the stream, the name of the stream, a partition key, and the data blob to be added to the stream should be specified.
    • Partition key is used to determine which shard in the stream the data record is added to.
  • Consumers
    • A consumer is an application built to read and process data records from Kinesis data streams.

Kinesis Data Streams Large Record Support

  • Kinesis Data Streams supports record sizes up to 10 MiB – a tenfold increase from the previous 1 MiB limit (announced Oct 2025).
  • Maximum PutRecords request size increased from 5 MiB to 10 MiB.
  • Large record support can be configured per stream by updating the maxRecordSize setting via Console, CLI, or SDK.
  • Individual shard throughput limits (1 MiB/s write, 2 MiB/s read) remain unchanged; large records use burst capacity (up to 10 MiB/s per shard) that refills at the standard rate.
  • Best practices:
    • Maintain large records at 1-2% of total record count for optimal performance.
    • Use random/uniformly distributed partition keys to spread large records across shards.
    • Implement backoff and retry logic in producer applications.
    • For continuous large record streams, store payloads in S3 and send metadata references through the stream.
  • Supported by KCL 2.x+, Amazon Data Firehose (S3 delivery), and AWS Lambda consumers.

Kinesis Security

  • supports Server-side encryption using Key Management Service (KMS) for encrypting the data at rest.
  • supports writing encrypted data to a data stream by encrypting and decrypting on the client side.
  • supports encryption in transit using HTTPS endpoints.
  • supports Interface VPC endpoint to keep traffic between VPC and Kinesis Data Streams from leaving the Amazon network. Interface VPC endpoints don’t require an IGW, NAT device, VPN connection, or Direct Connect.
  • integrated with IAM to control access to Kinesis Data Streams resources.
  • supports Attribute-Based Access Control (ABAC) using stream tags and consumer tags, enabling fine-grained access control based on tag key-value pairs attached to IAM users, roles, and resources (Sep 2024 for streams, Apr 2025 for consumers).
  • supports resource-based policies for cross-account access, enabling processing data ingested in one account with consumers (e.g., Lambda) in another account without assuming IAM roles.
  • integrated with CloudTrail, which provides a record of actions taken by a user, role, or an AWS service in Kinesis Data Streams.
  • supports AWS Fault Injection Service (FIS) integration (Oct 2025) to test application resilience against common API errors (e.g., ProvisionedThroughputExceeded, expired iterators) in a controlled environment.

Kinesis Producer

Data to Kinesis Data Streams can be added via API/SDK (PutRecord and PutRecords) operations, Kinesis Producer Library (KPL), or Kinesis Agent.

  • API
    • PutRecord & PutRecords operations are synchronous operation that sends single/multiple records to the stream per HTTP request.
    • use PutRecords to achieve a higher throughput per data producer
    • helps manage many aspects of Kinesis Data Streams (including creating streams, resharding, and putting and getting records)
  • Kinesis Agent
    • is a pre-built Java application that offers an easy way to collect and send data to the Kinesis stream.
    • can be installed on Linux-based server environments such as web servers, log servers, and database servers
    • can be configured to monitor certain files on the disk and then continuously send new data to the Kinesis stream
  • Kinesis Producer Library (KPL)
    • is an easy-to-use and highly configurable library that helps to put data into a Kinesis stream.
    • provides a layer of abstraction specifically for ingesting data
    • presents a simple, asynchronous, and reliable interface that helps achieve high producer throughput with minimal client resources.
    • batches messages, as it aggregates records to increase payload size and improve throughput.
    • Collects records and uses PutRecords to write multiple records to multiple shards per request
    • Writes to one or more Kinesis data streams with an automatic and configurable retry mechanism.
    • Integrates seamlessly with the Kinesis Client Library (KCL) to de-aggregate batched records on the consumer
    • Submits CloudWatch metrics to provide visibility into performance
    • ⚠️ KPL 0.x reached end-of-support on January 30, 2026. Migrate to KPL 1.x. The upgrade requires no changes to data processing logic.
  • Third Party and Open source
    • Log4j appender
    • Apache Kafka
    • Flume, fluentd, etc.

Kinesis Consumers

  • Kinesis Application is a data consumer that reads and processes data from a Kinesis Data Stream and can be built using either Kinesis API or Kinesis Client Library (KCL)
  • Shards in a stream provide 2 MB/sec of read throughput per shard, by default, which is shared by all the consumers reading from a given shard.
  • Kinesis Client Library (KCL)
    • is a pre-built library with multiple language support
    • delivers all records for a given partition key to same record processor
    • makes it easier to build multiple applications reading from the same stream for e.g. to perform counting, aggregation, and filtering
    • handles complex issues such as adapting to changes in stream volume, load-balancing streaming data, coordinating distributed services, and processing data with fault-tolerance
    • uses a unique DynamoDB table to keep track of the application’s state, so if the Kinesis Data Streams application receives provisioned-throughput exceptions, increase the provisioned throughput for the DynamoDB table
    • KCL 3.0 (launched Nov 2024) introduces an enhanced load balancing algorithm that continuously monitors resource utilization and redistributes load from over-utilized workers to underutilized workers, reducing stream processing compute costs by up to 33%.
    • ⚠️ KCL 1.x reached end-of-support on January 30, 2026. Migrate to KCL 3.x. Migration requires updating interfaces and security credential providers.
  • Kinesis Connector Library
    • is a pre-built library that helps you easily integrate Kinesis Streams with other AWS services and third-party tools.
    • Kinesis Client Library is required for Kinesis Connector Library
    • is legacy and can be replaced by Lambda or Kinesis Data Firehose
  • Kinesis Storm Spout is a pre-built library that helps you easily integrate Kinesis Streams with Apache Storm
  • AWS Lambda, Amazon Data Firehose, and Amazon Managed Service for Apache Flink also act as consumers for Kinesis Data Streams

Kinesis Enhanced Fan-Out

  • allows customers to scale the number of consumers reading from a data stream in parallel, while maintaining high performance and without contending for read throughput with other consumers.
  • provides logical 2 MB/sec throughput pipes between consumers and shards for Kinesis Data Streams Consumers.
  • supports up to 20 enhanced fan-out consumers per stream with On-demand Standard and Provisioned modes.
  • supports up to 50 enhanced fan-out consumers per stream with On-demand Advantage mode (Nov 2025).
  • supports consumer tagging and ABAC for fine-grained access control and cost allocation (Apr 2025).

AWS Kinesis Shared Throughput vs Enhanced Fan-out

Kinesis Data Streams Sharding

  • Resharding helps to increase or decrease the number of shards in a stream in order to adapt to changes in the rate of data flowing through the stream.
  • Resharding operations support shard split and shard merge.
    • Shard split helps divide a single shard into two shards. It increases the capacity and the cost.
    • Shard merge helps combine two shards into a single shard. It reduces the capacity and the cost.
  • Resharding is always pairwise and always involves two shards.
  • The shard or pair of shards that the resharding operation acts on are referred to as parent shards. The shard or pair of shards that result from the resharding operation are referred to as child shards.
  • Kinesis Client Library tracks the shards in the stream using a DynamoDB table and discovers the new shards and populates new rows in the table.
  • KCL ensures that any data that existed in shards prior to the resharding is processed before the data from the new shards, thereby, preserving the order in which data records were added to the stream for a particular partition key.
  • Data records in the parent shard are accessible from the time they are added to the stream to the current retention period.

Kinesis Data Streams vs Kinesis Firehose

Refer post @ Kinesis Data Streams vs Kinesis Firehose

Kinesis Data Streams vs. Firehose

Kinesis Data Streams vs SQS

Refer post @ Kinesis Data Streams vs SQS

Kinesis vs S3

Amazon Kinesis vs S3

Kinesis Data Analytics for SQL – Discontinued

⚠️ Amazon Kinesis Data Analytics for SQL was discontinued on January 27, 2026.

AWS deleted all existing SQL applications. Customers should migrate to:

  • Amazon Managed Service for Apache Flink – for complex stream processing
  • Amazon Managed Service for Apache Flink Studio – for interactive SQL-based analytics

AWS Certification Exam Practice Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  1. You are deploying an application to track GPS coordinates of delivery trucks in the United States. Coordinates are transmitted from each delivery truck once every three seconds. You need to design an architecture that will enable real-time processing of these coordinates from multiple consumers. Which service should you use to implement data ingestion?
    1. Amazon Kinesis
    2. AWS Data Pipeline
    3. Amazon AppStream
    4. Amazon Simple Queue Service
  2. You are deploying an application to collect votes for a very popular television show. Millions of users will submit votes using mobile devices. The votes must be collected into a durable, scalable, and highly available data store for real-time public tabulation. Which service should you use?
    1. Amazon DynamoDB
    2. Amazon Redshift
    3. Amazon Kinesis
    4. Amazon Simple Queue Service
  3. Your company is in the process of developing a next generation pet collar that collects biometric information to assist families with promoting healthy lifestyles for their pets. Each collar will push 30kb of biometric data In JSON format every 2 seconds to a collection platform that will process and analyze the data providing health trending information back to the pet owners and veterinarians via a web portal Management has tasked you to architect the collection platform ensuring the following requirements are met. Provide the ability for real-time analytics of the inbound biometric data Ensure processing of the biometric data is highly durable, elastic and parallel. The results of the analytic processing should be persisted for data mining. Which architecture outlined below will meet the initial requirements for the collection platform?
    1. Utilize S3 to collect the inbound sensor data analyze the data from S3 with a daily scheduled Data Pipeline and save the results to a Redshift Cluster.
    2. Utilize Amazon Kinesis to collect the inbound sensor data, analyze the data with Kinesis clients and save the results to a Redshift cluster using EMR. (refer link)
    3. Utilize SQS to collect the inbound sensor data analyze the data from SQS with Amazon Kinesis and save the results to a Microsoft SQL Server RDS instance.
    4. Utilize EMR to collect the inbound sensor data, analyze the data from EUR with Amazon Kinesis and save me results to DynamoDB.
  4. Your customer is willing to consolidate their log streams (access logs, application logs, security logs etc.) in one single system. Once consolidated, the customer wants to analyze these logs in real time based on heuristics. From time to time, the customer needs to validate heuristics, which requires going back to data samples extracted from the last 12 hours? What is the best approach to meet your customer’s requirements?
    1. Send all the log events to Amazon SQS. Setup an Auto Scaling group of EC2 servers to consume the logs and apply the heuristics.
    2. Send all the log events to Amazon Kinesis develop a client process to apply heuristics on the logs (Can perform real time analysis and stores data for 24 hours which can be extended to 365 days)
    3. Configure Amazon CloudTrail to receive custom logs, use EMR to apply heuristics the logs (CloudTrail is only for auditing)
    4. Setup an Auto Scaling group of EC2 syslogd servers, store the logs on S3 use EMR to apply heuristics on the logs (EMR is for batch analysis)
  5. You require the ability to analyze a customer’s clickstream data on a website so they can do behavioral analysis. Your customer needs to know what sequence of pages and ads their customer clicked on. This data will be used in real time to modify the page layouts as customers click through the site to increase stickiness and advertising click-through. Which option meets the requirements for captioning and analyzing this data?
    1. Log clicks in weblogs by URL store to Amazon S3, and then analyze with Elastic MapReduce
    2. Push web clicks by session to Amazon Kinesis and analyze behavior using Kinesis workers
    3. Write click events directly to Amazon Redshift and then analyze with SQL
    4. Publish web clicks by session to an Amazon SQS queue men periodically drain these events to Amazon RDS and analyze with SQL
  6. Your social media monitoring application uses a Python app running on AWS Elastic Beanstalk to inject tweets, Facebook updates and RSS feeds into an Amazon Kinesis stream. A second AWS Elastic Beanstalk app generates key performance indicators into an Amazon DynamoDB table and powers a dashboard application. What is the most efficient option to prevent any data loss for this application?
    1. Use AWS Data Pipeline to replicate your DynamoDB tables into another region.
    2. Use the second AWS Elastic Beanstalk app to store a backup of Kinesis data onto Amazon Elastic Block Store (EBS), and then create snapshots from your Amazon EBS volumes.
    3. Add a second Amazon Kinesis stream in another Availability Zone and use AWS data pipeline to replicate data across Kinesis streams.
    4. Add a third AWS Elastic Beanstalk app that uses the Amazon Kinesis S3 connector to archive data from Amazon Kinesis into Amazon S3.
  7. You need to replicate API calls across two systems in real time. What tool should you use as a buffer and transport mechanism for API call events?
    1. AWS SQS
    2. AWS Lambda
    3. AWS Kinesis (AWS Kinesis is an event stream service. Streams can act as buffers and transport across systems for in-order programmatic events, making it ideal for replicating API calls across systems)
    4. AWS SNS
  8. You need to perform ad-hoc business analytics queries on well-structured data. Data comes in constantly at a high velocity. Your business intelligence team can understand SQL. What AWS service(s) should you look to first?
    1. Kinesis Firehose + RDS
    2. Kinesis Firehose + RedShift (Kinesis Firehose provides a managed service for aggregating streaming data and inserting it into RedShift. RedShift also supports ad-hoc queries over well-structured data using a SQL-compliant wire protocol, so the business team should be able to adopt this system easily. Refer link)
    3. EMR using Hive
    4. EMR running Apache Spark
  9. A company streams IoT sensor data into Kinesis Data Streams using provisioned mode. They have 10 consumers reading from the same stream and experience latency issues. Which solution provides dedicated throughput to each consumer without increasing shards?
    1. Switch to On-demand Standard mode
    2. Use Kinesis Data Firehose instead
    3. Enable Enhanced Fan-Out for each consumer (Enhanced fan-out provides a dedicated 2 MB/sec pipe per consumer per shard, eliminating shared throughput contention)
    4. Increase the number of shards
  10. A development team needs to process records that are occasionally 5 MB in size from their Kinesis Data Stream. What must they do to support this? (Choose 2)
    1. Update the stream’s maxRecordSize setting to allow records up to 5 MiB or larger
    2. Split records into smaller chunks before ingestion
    3. Ensure downstream consumers (KCL 2.x+, Lambda, or Firehose) can handle larger records
    4. Upgrade to a provisioned mode stream with more shards
    5. Store the records in S3 and send only metadata to Kinesis

References

AWS Redshift

Redshift Architecture
~ Last updated on : ~ jayendrapatil ~ 5 Comments

AWS Redshift

  • Amazon Redshift is a fully managed, fast, and powerful, petabyte-scale data warehouse service.
  • Redshift is an OLAP data warehouse solution based on PostgreSQL.
  • Redshift is built on cloud economics that scale with usage — powering modern analytics and autonomous agentic AI workloads on the data warehouse.
  • Redshift delivers up to 2.2x better price-performance and 7x better throughput than other cloud data warehouses.
  • Redshift automatically helps
    • set up, operate, and scale a data warehouse, from provisioning the infrastructure capacity.
    • patches and backs up the data warehouse, storing the backups for a user-defined retention period.
    • monitors the nodes and drives to help recovery from failures.
    • significantly lowers the cost of a data warehouse, but also makes it easy to analyze large amounts of data very quickly
    • provide fast querying capabilities over structured and semi-structured data using familiar SQL-based clients and business intelligence (BI) tools using standard ODBC and JDBC connections.
    • uses replication and continuous backups to enhance availability and improve data durability and can automatically recover from node and component failures.
    • scale up or down with a few clicks in the AWS Management Console or with a single API call
    • distributes & parallelize queries across multiple physical resources
    • supports VPC, SSL, AES-256 encryption, and Hardware Security Modules (HSMs) to protect the data in transit and at rest.
  • Redshift supported only Single-AZ deployments before and the nodes are available within the same AZ, if the AZ supports Redshift clusters. However, Multi-AZ deployments are now supported for RA3 clusters.
  • Redshift provides monitoring using CloudWatch and metrics for compute utilization, storage utilization, and read/write traffic to the cluster are available with the ability to add user-defined custom metrics
  • Redshift provides Audit logging and AWS CloudTrail integration
  • Redshift can be easily enabled to a second region for disaster recovery.

Redshift Node Types

  • RG Instances (Current Generation – GA May 2026)
    • Newest generation powered by AWS Graviton processors.
    • Delivers up to 2.2x better performance for data warehouse workloads and up to 2.4x faster for data lake workloads compared to RA3, at 30% lower price per vCPU.
    • Includes a custom-built vectorized data lake query engine that processes Apache Iceberg and Parquet data on cluster nodes.
    • Enables running SQL analytics across data warehouse and data lake using a single engine.
    • Uses Redshift Managed Storage (RMS) with independent scaling of compute and storage.
  • RA3 Instances
    • Previous generation with managed storage, allowing independent scaling of compute and storage.
    • Available in RA3.xlplus, RA3.4xlarge, RA3.16xlarge, and RA3.large sizes.
    • RA3.large launched in 2024 as a cost-effective option for smaller workloads.
    • Uses Redshift Managed Storage (RMS) backed by S3.
    • Supports data sharing, Multi-AZ, zero-ETL integrations.
  • DC2 Instances (DEPRECATED – EOL April 24, 2026)
    • Dense Compute nodes with fast CPUs, large amounts of RAM, and solid-state disks (SSDs).
    • DC2 nodes reached End of Life on April 24, 2026.
    • After May 15, 2025, creation of new DC2 clusters, resizing existing ones, or adding nodes was no longer allowed.
    • Migration options: Upgrade to RA3, RG instances, or Redshift Serverless using elastic resize.
  • DS2 Instances (DEPRECATED)
    • Dense Storage nodes using HDDs, previously used for large data warehouses.
    • DS2 has been deprecated and creation of new DS2 clusters is no longer allowed.
    • Migration options: Upgrade to RA3 or RG instances using elastic resize.

Redshift Performance

  • Massively Parallel Processing (MPP)
    • automatically distributes data and query load across all nodes.
    • makes it easy to add nodes to the data warehouse and enables fast query performance as the data warehouse grows.
  • Columnar Data Storage
    • organizes the data by column, as column-based systems are ideal for data warehousing and analytics, where queries often involve aggregates performed over large data sets
    • columnar data is stored sequentially on the storage media, and require far fewer I/Os, greatly improving query performance
  • Advance Compression
    • Columnar data stores can be compressed much more than row-based data stores because similar data is stored sequentially on a disk.
    • employs multiple compression techniques and can often achieve significant compression relative to traditional relational data stores.
    • doesn’t require indexes or materialized views and so uses less space than traditional relational database systems.
    • automatically samples the data and selects the most appropriate compression scheme, when the data is loaded into an empty table
  • Query Optimizer
    • Redshift query run engine incorporates a query optimizer that is MPP-aware and also takes advantage of columnar-oriented data storage.
  • Result Caching
    • Redshift caches the results of certain types of queries in memory on the leader node.
    • When a user submits a query, Redshift checks the results cache for a valid, cached copy of the query results. If a match is found in the result cache, Redshift uses the cached results and doesn’t run the query.
    • Result caching is transparent to the user.
  • Compiled Code
    • Leader node distributes fully optimized compiled code across all of the nodes of a cluster. Compiling the query decreases the overhead associated with an interpreter and therefore increases the runtime speed, especially for complex queries.
  • AQUA (Advanced Query Accelerator)
    • AQUA is a distributed and hardware-accelerated cache that boosts certain types of queries.
    • Amazon Redshift automatically determines whether to use AQUA — no manual configuration is required.
    • AQUA pushes the computation needed to handle reduction and aggregation queries closer to the data, reducing network traffic and offloading work from the cluster.
  • Autonomics
    • Redshift is a self-learning, self-tuning system that automatically generates and implements optimal data layout recommendations for distribution and sort keys.
    • Autonomics now extend to multi-cluster environments, eliminating manual performance tuning across consumer clusters.

Redshift Architecture

Redshift Architecture

  • Clusters
    • Core infrastructure component of a Redshift data warehouse
    • Cluster is composed of one or more compute nodes.
    • If a cluster is provisioned with two or more compute nodes, an additional leader node coordinates the compute nodes and handles external communication.
    • Client applications interact directly only with the leader node.
    • Compute nodes are transparent to external applications.
  • Leader node
    • Leader node manages communications with client programs and all communication with compute nodes.
    • It parses and develops execution plans to carry out database operations
    • Based on the execution plan, the leader node compiles code, distributes the compiled code to the compute nodes, and assigns a portion of the data to each compute node.
    • Leader node distributes SQL statements to the compute nodes only when a query references tables that are stored on the compute nodes. All other queries run exclusively on the leader node.
  • Compute nodes
    • Leader node compiles code for individual elements of the execution plan and assigns the code to individual compute nodes.
    • Compute nodes execute the compiled code and send intermediate results back to the leader node for final aggregation.
    • Each compute node has its own dedicated CPU, memory, and attached disk storage, which is determined by the node type.
    • As the workload grows, the compute and storage capacity of a cluster can be increased by increasing the number of nodes, upgrading the node type, or both.
  • Node slices
    • A compute node is partitioned into slices.
    • Each slice is allocated a portion of the node’s memory and disk space, where it processes a portion of the workload assigned to the node.
    • Leader node manages distributing data to the slices and apportions the workload for any queries or other database operations to the slices. The slices then work in parallel to complete the operation.
    • Number of slices per node is determined by the node size of the cluster.
    • When a table is created, one column can optionally be specified as the distribution key. When the table is loaded with data, the rows are distributed to the node slices according to the distribution key that is defined for a table.
    • A good distribution key enables Redshift to use parallel processing to load data and execute queries efficiently.
  • Managed Storage
    • Data warehouse data is stored in a separate storage tier Redshift Managed Storage (RMS).
    • RMS provides the ability to scale the storage to petabytes using S3 storage.
    • RMS enables scale, pay for compute and storage independently so that the cluster can be sized based only on the computing needs.
    • RMS automatically uses high-performance SSD-based local storage as tier-1 cache.
    • It also takes advantage of optimizations, such as data block temperature, data block age, and workload patterns to deliver high performance while scaling storage automatically to S3 when needed without requiring any action.
    • RMS tables can now be accessed through Apache Iceberg REST APIs via Amazon SageMaker Lakehouse, enabling other engines (Spark, EMR) to read Redshift data.

Redshift Serverless

  • Redshift Serverless is a serverless option of Redshift that makes it more efficient to run and scale analytics in seconds without the need to set up and manage data warehouse infrastructure.
  • Redshift Serverless automatically provisions and intelligently scales data warehouse capacity to deliver high performance for demanding and unpredictable workloads.
  • Redshift Serverless helps any user to get insights from data by simply loading and querying data in the data warehouse.
  • Redshift Serverless supports concurrency Scaling feature that can support unlimited concurrent users and concurrent queries, with consistently fast query performance.
  • When concurrency scaling is enabled, Redshift automatically adds cluster capacity when the cluster experiences an increase in query queuing.
  • Redshift Serverless measures data warehouse capacity in Redshift Processing Units (RPUs). RPUs are resources used to handle workloads.
  • Redshift Serverless supports workgroups and namespaces to isolate workloads and manage different resources.
  • Redshift Serverless supports a capacity configuration of up to 1024 RPUs for larger workloads.
  • AI-Driven Scaling and Optimization (GA 2024)
    • Redshift Serverless now includes AI-driven scaling and optimization that automatically scales compute resources based on query complexity, data volume, and workload patterns.
    • Users can choose to optimize for cost, performance, or keep it balanced using a simple slider.
    • Delivers up to 10x price-performance improvements for variable workloads.
    • Scales the compute not only based on queuing, but also factoring data volume and query complexity.
  • AWS Graviton is available in Serverless, offering up to 30% better price-performance.

Redshift Single vs Multi-Node Cluster

  • Single Node
    • Single node configuration enables getting started quickly and cost-effectively & scale up to a multi-node configuration as the needs grow
  • Multi-Node
    • Multi-node configuration requires a leader node that manages client connections and receives queries, and two or more compute nodes that store data and perform queries and computations.
    • Leader node
      • provisioned automatically and not charged for
      • receives queries from client applications, parses the queries, and develops execution plans, which are an ordered set of steps to process these queries.
      • coordinates the parallel execution of these plans with the compute nodes, aggregates the intermediate results from these nodes, and finally returns the results back to the client applications.
    • Compute node
      • can contain from 1-128 compute nodes, depending on the node type
      • executes the steps specified in the execution plans and transmits data among themselves to serve these queries.
      • intermediate results are sent back to the leader node for aggregation before being sent back to the client applications.
      • Current node types: RG (Graviton-based, latest), RA3 (managed storage)
        • RG instances — Latest generation, Graviton-powered, up to 2.4x faster than RA3 at 30% lower cost per vCPU, includes integrated data lake query engine
        • RA3 instances — Managed storage with independent compute/storage scaling, supports data sharing and Multi-AZ
        • DC2 (DEPRECATED – EOL April 24, 2026) — Dense Compute with SSDs, no longer available for new clusters
        • DS2 (DEPRECATED) — Dense Storage with HDDs, no longer available for new clusters
      • direct access to compute nodes is not allowed

Redshift Multi-AZ

  • Redshift Multi-AZ deployment runs the data warehouse in multiple AWS AZs simultaneously and continues operating in unforeseen failure scenarios.
  • Multi-AZ deployment is managed as a single data warehouse with one endpoint and does not require any application changes.
  • Multi-AZ deployments support high availability requirements and reduce recovery time by guaranteeing capacity to automatically recover and are intended for customers with business-critical analytics applications that require the highest levels of availability and resiliency to AZ failures.
  • Redshift Multi-AZ supports RPO = 0 meaning data is guaranteed to be current and up to date in the event of a failure. RTO is under a minute.
  • Multi-AZ is supported for RA3 clusters (GA 2024).

Redshift Availability & Durability

  • Redshift replicates the data within the data warehouse cluster and continuously backs up the data to S3 (11 9’s durability).
  • Redshift mirrors each drive’s data to other nodes within the cluster.
  • Redshift will automatically detect and replace a failed drive or node.
  • RA3 and RG clusters and Redshift serverless are not impacted the same way since the data is stored in S3 and the local drive is just used as a data cache.
  • If a drive fails,
    • cluster will remain available in the event of a drive failure.
    • the queries will continue with a slight latency increase while Redshift rebuilds the drive from the replica of the data on that drive which is stored on other drives within that node.
    • single node clusters do not support data replication and the cluster needs to be restored from a snapshot on S3.
  • In case of node failure(s), Redshift
    • automatically provisions new node(s) and begins restoring data from other drives within the cluster or from S3.
    • prioritizes restoring the most frequently queried data so the most frequently executed queries will become performant quickly.
    • cluster will be unavailable for queries and updates until a replacement node is provisioned and added to the cluster.
  • In case of Redshift cluster AZ goes down, Redshift
    • cluster is unavailable until power and network access to the AZ are restored
    • cluster’s data is preserved and can be used once AZ becomes available
    • cluster can be restored from any existing snapshots to a new AZ within the same region
    • Multi-AZ deployments (RA3) continue operating automatically during AZ failures

Redshift Backup & Restore

  • Redshift always attempts to maintain at least three copies of the data – Original, Replica on the compute nodes, and a backup in S3.
  • Redshift replicates all the data within the data warehouse cluster when it is loaded and also continuously backs up the data to S3.
  • Redshift enables automated backups of the data warehouse cluster with a 1-day retention period, by default, which can be extended to max 35 days.
  • Automated backups can be turned off by setting the retention period as 0.
  • Redshift can also asynchronously replicate the snapshots to S3 in another region for disaster recovery.
  • Redshift only backs up data that has changed.
  • Restoring the backup will provision a new data warehouse cluster.
  • Starting June 8, 2026, Amazon Redshift introduced incremental snapshot billing for Serverless and RG instances — customers pay only for unique data blocks across active manual snapshots.

Redshift Scalability

  • Redshift allows scaling of the cluster either by
    • increasing the node instance type (Vertical scaling)
    • increasing the number of nodes (Horizontal scaling)
  • Redshift scaling changes are usually applied during the maintenance window or can be applied immediately
  • Elastic Resize
    • Elastic resize allows changing node types within minutes with a single operation.
    • Supports migration from deprecated DC2/DS2 to RA3 or RG instances.
    • Can be run at any time or scheduled for a future time.
  • Redshift scaling process
    • existing cluster remains available for read operations only, while a new data warehouse cluster gets created during scaling operations
    • data from the compute nodes in the existing data warehouse cluster is moved in parallel to the compute nodes in the new cluster
    • when the new data warehouse cluster is ready, the existing cluster will be temporarily unavailable while the canonical name record of the existing cluster is flipped to point to the new data warehouse cluster
  • Concurrency Scaling
    • Automatically adds query processing power in seconds when concurrency increases.
    • Supports thousands of concurrent users and queries with consistently fast performance.
    • Extra processing power is automatically removed when demand subsides — pay only for usage.
    • Each cluster earns up to one hour of free Concurrency Scaling credits per day.
    • Now supports auto-copy, zero-ETL ingestion workloads, and COPY queries from S3 (2026).

Redshift Zero-ETL Integrations

  • Zero-ETL integrations seamlessly make transactional or operational data available in Redshift without building and managing complex ETL pipelines.
  • Data replicates in near real-time from the source into Redshift.
  • Supported Sources (GA):
    • Amazon Aurora MySQL-Compatible Edition — First zero-ETL source, supports petabytes of transactional data
    • Amazon Aurora PostgreSQL-Compatible Edition — GA October 2024
    • Amazon RDS for MySQL — GA September 2024
    • Amazon DynamoDB — GA October 2024
    • Enterprise Applications — Salesforce, Zendesk, ServiceNow, SAP, Facebook Ads, Instagram Ads, Pardot, Zoho CRM (GA December 2024)
  • Key Features:
    • Data filtering to selectively extract tables and schemas using regular expressions
    • Support for incremental and auto-refresh materialized views on replicated data
    • Configurable change data capture (CDC) refresh rates
    • Supports Enhanced VPC Routing warehouses
    • Concurrency scaling support for zero-ETL (2026)
  • Zero-ETL integrations can be created on Redshift Serverless workgroups or provisioned clusters using RA3/RG instance types.

Redshift Data Ingestion

  • Auto-Copy (GA 2024)
    • Simplifies continuous data ingestion from Amazon S3 into Redshift.
    • Set up ingestion rules to track S3 paths and automatically load new files without additional tools or custom solutions.
    • Uses S3 event integrations to detect and copy new files automatically.
    • Concurrency scaling support for auto-copy (2026).
  • Streaming Ingestion
    • Enables near real-time analytics by ingesting streaming data directly into Redshift without staging in S3.
    • Creates materialized views directly on top of data streams.
    • Supports Amazon Kinesis Data Streams, Amazon MSK, Confluent Managed Cloud, and self-managed Apache Kafka clusters.
    • Achieves low latency (measured in seconds) while ingesting hundreds of megabytes per second.
  • COPY Command — Traditional bulk loading from S3, DynamoDB, EMR, or remote hosts via SSH.

Redshift Data Sharing

  • Redshift data sharing allows sharing live, transactionally consistent data across different Redshift clusters without data copies or movement.
  • Supports sharing within the same AWS account, across accounts, and across regions.
  • Multi-Data Warehouse Writes (GA 2024) — Enables writing to shared Redshift databases from multiple Redshift data warehouses. Written data is available to all data warehouses as soon as committed.
  • Data sharing with data lake tables is now generally available — enables sharing Apache Iceberg/open format data.
  • Supports data sharing for use cases like customer 360, data monetization, and distributed ETL workloads.
  • First query execution for data sharing is up to 4x faster with improved metadata handling (2024).

Redshift and Amazon SageMaker Lakehouse

  • Amazon SageMaker Lakehouse (GA re:Invent 2024) unifies data across Amazon S3 data lakes and Redshift data warehouses.
  • Provides flexibility to access and query data using Apache Iceberg open standards.
  • Existing Redshift data warehouses can be made available through SageMaker Lakehouse via a simple publish step, exposing all data warehouse data with Iceberg REST API.
  • New data lake tables can be created using Redshift Managed Storage (RMS) as a native storage option.
  • Integrates with AWS Glue Data Catalog for unified metadata management.
  • Supports fine-grained permissions consistently applied across all analytics engines and AI tools.
  • Apache Iceberg Support in Redshift:
    • Query Iceberg tables stored in S3 and S3 Table Buckets.
    • Write support: INSERT, DELETE, UPDATE, and MERGE operations on Iceberg tables (2025-2026).
    • 2x faster data lake query performance for Iceberg workloads (2025).
    • Incremental refresh for materialized views on data lake tables.

Redshift Generative AI and ML

  • Amazon Q Generative SQL for Redshift (GA 2024)
    • Uses generative AI to analyze user intent, query patterns, and schema metadata.
    • Allows users to express queries in natural language and receive SQL code recommendations.
    • Leverages query history and custom context (table/column descriptions, sample queries) for more relevant SQL recommendations.
    • Available in the Redshift Query Editor.
  • Amazon Bedrock Integration (GA 2024)
    • Enables invoking Large Language Models (LLMs) from simple SQL commands on Redshift data.
    • Supports generative AI tasks: language translation, text generation, summarization, sentiment analysis, and data enrichment.
    • Supports models from Anthropic Claude, Amazon Titan, Meta Llama, and Mistral AI.
  • Redshift as Knowledge Base in Amazon Bedrock
    • Amazon Bedrock Knowledge Bases supports natural language querying to retrieve structured data from Redshift.
    • Transforms natural language queries into SQL queries automatically.
    • Users can ask questions like “What were my top 5 selling products?” without writing SQL.
  • Redshift ML
    • Provides integration between Redshift and Amazon SageMaker.
    • Enables creating, training, and deploying ML models using SQL.
    • Supports inference within the Redshift cluster for predictions in queries and applications.
    • Note: Scalar Python UDFs will reach end of support after June 30, 2026. Migrate to Lambda UDFs or Redshift ML.

Redshift Security

  • Redshift supports encryption at rest and in transit
  • Redshift provides support for role-based access control – RBAC. Row-level access control helps assign one or more roles to a user and assign system and object permissions by role.
  • Redshift supports Lambda User-defined Functions – UDFs to enable external tokenization, data masking, identification or de-identification of data by integrating with vendors like Protegrity, and protect or unprotect sensitive data based on a user’s permissions and groups, in query time.
  • Redshift supports Single Sign-On (SSO) and integrates with third-party corporate or SAML-compliant identity providers.
  • Redshift supports federated permissions with AWS IAM Identity Center, now available in multiple AWS Regions (2026).
  • Redshift supports multi-factor authentication (MFA) for additional security when authenticating to the Redshift cluster.
  • Redshift supports encrypting an unencrypted cluster using KMS. However, you can’t enable hardware security module (HSM) encryption by modifying the cluster. Instead, create a new, HSM-encrypted cluster and migrate your data to the new cluster.
  • Redshift enhanced VPC routing forces all COPY and UNLOAD traffic between the cluster and the data repositories through the VPC.
  • Minimum TLS version requirement changes effective January 31, 2026 — ensure clients support TLS 1.2 or higher.

Redshift Advanced Topics

Redshift Best Practices

Refer blog post Redshift Best Practices

Redshift vs EMR vs RDS

  • RDS is ideal for
    • structured data and running traditional relational databases while offloading database administration
    • for online-transaction processing (OLTP) and for reporting and analysis
  • Redshift is ideal for
    • large volumes of structured data that needs to be persisted and queried using standard SQL and existing BI tools
    • analytic and reporting workloads against very large data sets by harnessing the scale and resources of multiple nodes and using a variety of optimizations to provide improvements over RDS
    • preventing reporting and analytic processing from interfering with the performance of the OLTP workload
    • unified analytics across data warehouses and data lakes using a single engine (especially with RG instances)
  • EMR is ideal for
    • processing and transforming unstructured or semi-structured data to bring in to Amazon Redshift and
    • for data sets that are relatively transitory, not stored for long-term use.

AWS Certification Exam Practice Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  1. With which AWS services CloudHSM can be used (select 2)
    1. S3
    2. DynamoDB
    3. RDS
    4. ElastiCache
    5. Amazon Redshift
  2. You have recently joined a startup company building sensors to measure street noise and air quality in urban areas. The company has been running a pilot deployment of around 100 sensors for 3 months. Each sensor uploads 1KB of sensor data every minute to a backend hosted on AWS. During the pilot, you measured a peak of 10 IOPS on the database, and you stored an average of 3GB of sensor data per month in the database. The current deployment consists of a load-balanced auto scaled Ingestion layer using EC2 instances and a PostgreSQL RDS database with 500GB standard storage. The pilot is considered a success and your CEO has managed to get the attention or some potential investors. The business plan requires a deployment of at least 100K sensors, which needs to be supported by the backend. You also need to store sensor data for at least two years to be able to compare year over year Improvements. To secure funding, you have to make sure that the platform meets these requirements and leaves room for further scaling. Which setup will meet the requirements?
    1. Add an SQS queue to the ingestion layer to buffer writes to the RDS instance (RDS instance will not support data for 2 years)
    2. Ingest data into a DynamoDB table and move old data to a Redshift cluster (Handle 10K IOPS ingestion and store data into Redshift for analysis)
    3. Replace the RDS instance with a 6 node Redshift cluster with 96TB of storage (Does not handle the ingestion issue)
    4. Keep the current architecture but upgrade RDS storage to 3TB and 10K provisioned IOPS (RDS instance will not support data for 2 years)
  3. Which two AWS services provide out-of-the-box user configurable automatic backup-as-a-service and backup rotation options? Choose 2 answers
    1. Amazon S3
    2. Amazon RDS
    3. Amazon EBS
    4. Amazon Redshift
  4. Your department creates regular analytics reports from your company’s log files. All log data is collected in Amazon S3 and processed by daily Amazon Elastic Map Reduce (EMR) jobs that generate daily PDF reports and aggregated tables in CSV format for an Amazon Redshift data warehouse. Your CFO requests that you optimize the cost structure for this system. Which of the following alternatives will lower costs without compromising average performance of the system or data integrity for the raw data?
    1. Use reduced redundancy storage (RRS) for PDF and CSV data in Amazon S3. Add Spot instances to Amazon EMR jobs. Use Reserved Instances for Amazon Redshift. (Spot instances impacts performance)
    2. Use reduced redundancy storage (RRS) for all data in S3. Use a combination of Spot instances and Reserved Instances for Amazon EMR jobs. Use Reserved instances for Amazon Redshift (Combination of the Spot and reserved with guarantee performance and help reduce cost. Also, RRS would reduce cost and guarantee data integrity, which is different from data durability)
    3. Use reduced redundancy storage (RRS) for all data in Amazon S3. Add Spot Instances to Amazon EMR jobs. Use Reserved Instances for Amazon Redshift (Spot instances impacts performance)
    4. Use reduced redundancy storage (RRS) for PDF and CSV data in S3. Add Spot Instances to EMR jobs. Use Spot Instances for Amazon Redshift. (Spot instances impacts performance and Spot instance not available for Redshift)
  5. A company needs to make their operational database data available in near real-time in their Redshift data warehouse for analytics. They want to eliminate custom ETL pipeline management. Which approach is recommended?
    1. Use AWS DMS to continuously replicate data from the source database to Redshift
    2. Use zero-ETL integration between the source database and Amazon Redshift (Zero-ETL eliminates the need to build and manage data pipelines and makes transactional data available in near real-time)
    3. Export data to S3 and use auto-copy to load into Redshift
    4. Use Kinesis Data Streams to stream data into Redshift
  6. A company wants to run analytics queries across both their Redshift data warehouse and Apache Iceberg tables stored in S3, using a single engine. Which Amazon Redshift capability best supports this requirement?
    1. Redshift Spectrum
    2. Redshift Federated Query
    3. Amazon SageMaker Lakehouse with Redshift (SageMaker Lakehouse unifies data across S3 data lakes and Redshift warehouses using Apache Iceberg open standards, and RG instances include an integrated data lake query engine)
    4. AWS Glue ETL to move data between S3 and Redshift
  7. A company is migrating from Amazon Redshift DC2 node types that are reaching End of Life. Which of the following are valid migration options? (Select 2)
    1. Migrate to RA3 instances using elastic resize
    2. Create a new DC2 cluster with more nodes
    3. Migrate to Amazon Redshift Serverless
    4. Migrate to DS2 instances
    5. Contact AWS to extend DC2 support
  8. An analytics team wants to use natural language queries to get insights from their Redshift data warehouse without writing SQL. Which AWS service integration enables this?
    1. Amazon Athena with natural language processing
    2. Amazon Bedrock Knowledge Bases with Amazon Redshift as a structured data source (Bedrock Knowledge Bases supports natural language querying to retrieve structured data from Redshift by automatically translating natural language to SQL)
    3. Amazon Q Business connected to Redshift
    4. Amazon SageMaker Canvas with Redshift

References

Kinesis Data Streams vs Firehose – Comparison

Kinesis Data Streams vs. Kinesis Data Firehose
~ Last updated on : ~ jayendrapatil ~ 11 Comments

AWS Kinesis Data Streams vs Amazon Data Firehose

📢 Service Rename (February 2024): Amazon Kinesis Data Firehose has been renamed to Amazon Data Firehose. The functionality remains the same. This post uses both names for clarity.

  • Kinesis acts as a highly available conduit to stream messages between data producers and data consumers.
  • Data producers can be almost any source of data: system or web log data, social network data, financial trading information, geospatial data, mobile app data, or telemetry from connected IoT devices.
  • Data consumers will typically fall into the category of data processing and storage applications such as Apache Hadoop, Apache Storm, S3, and OpenSearch.

Kinesis Data Streams vs. Firehose

Purpose

  • Kinesis Data Streams is highly customizable and best suited for developers building custom applications or streaming data for specialized needs.
  • Amazon Data Firehose (formerly Kinesis Data Firehose) handles loading data streams directly into AWS products for processing. Firehose allows streaming to S3, OpenSearch Service, Redshift, Apache Iceberg tables, Amazon S3 Tables, Snowflake, and other destinations, where data can be copied for processing through additional services.

Provisioning & Scaling

  • Kinesis Data Streams offers three capacity modes:
    • Provisioned Mode: Requires manual configuration of shards and scaling. You specify the number of shards needed based on expected throughput. Each shard provides 1 MB/s write (1000 records/s) and 2 MB/s read.
    • On-Demand Standard Mode (Launched November 2021): Automatically scales to handle gigabytes of write and read throughput per minute without manual shard management. Default capacity of 4 MB/s write (4000 records/s), can scale up to 200 MB/s (or 1 GB/s with limit increase).
    • On-Demand Advantage Mode (Launched November 2025): Enables warm throughput capability for instant scaling up to 10 GB/s or 10 million events/second. Offers 60% lower pricing compared to On-demand Standard ($0.032/GB ingest, $0.016/GB retrieval). Removes per-stream fixed charge. Supports up to 50 enhanced fan-out consumers (vs. 20 for other modes). Best for workloads ingesting at least 10 MB/s aggregate, high fan-out use cases, or accounts with hundreds of streams.
  • Amazon Data Firehose is fully managed and sends data to S3, Redshift, OpenSearch, Apache Iceberg tables, Amazon S3 Tables, Snowflake, and other destinations. Scaling is handled automatically, up to gigabytes per second, and allows for batching, encrypting, and compressing.

Processing Delay

  • Kinesis Data Streams provides real-time processing with ~200 ms for shared throughput classic single consumer and ~70 ms for the enhanced fan-out consumer.
  • Amazon Data Firehose provides near real-time processing:
    • Standard Buffering: Minimum buffer time of 60 seconds (1 min), maximum 900 seconds (15 min)
    • Zero Buffering (Announced December 2023): Delivers data within ~5 seconds with no buffering delay, enabling real-time use cases

Record Size

  • Kinesis Data Streams supports record sizes up to 10 MiB (increased from 1 MiB in October 2025), enabling larger data payloads like images, documents, and complex event data without splitting records.
  • Amazon Data Firehose supports record sizes up to 1 MiB per record.

Data Storage

  • Kinesis Data Streams provides data storage. Data typically is made available in a stream for 24 hours, but for an additional cost, users can gain data availability for up to 365 days (8760 hours). On-demand Advantage mode offers 77% lower extended retention pricing ($0.023/GB-month vs $0.10/GB-month).
  • Amazon Data Firehose does not provide data storage.

Replay

  • Kinesis Data Streams supports replay capability
  • Amazon Data Firehose does not support replay capability

Producers & Consumers

  • Kinesis Data Streams & Amazon Data Firehose support multiple producer options including SDK, KPL, Kinesis Agent, IoT, etc.
  • Kinesis Data Streams supports multiple consumer options including SDK, KCL, and Lambda, and can write data to multiple destinations. However, they have to be coded.
    • Supports up to 50 enhanced fan-out consumers per stream with On-demand Advantage mode (up from 20 with On-demand Standard/Provisioned modes) — Launched November 2025
    • Enhanced fan-out provides dedicated 2 MB/s throughput per consumer per shard with ~70 ms latency
  • Amazon Data Firehose consumers are close-ended and support destinations including:
    • Amazon S3
    • Amazon Redshift
    • Amazon OpenSearch Service
    • Amazon OpenSearch Serverless
    • Apache Iceberg Tables (Added October 2024) — Stream data directly into Iceberg format tables in S3
    • Amazon S3 Tables (GA March 2025) — Stream data into S3 Tables with built-in Apache Iceberg support and automatic table maintenance (compaction, snapshot management)
    • Snowflake (with Snowpipe Streaming) — Real-time streaming to Snowflake
    • Splunk
    • Third-party HTTP endpoints (Datadog, Dynatrace, New Relic, MongoDB, Coralogix, Elastic, etc.)
  • Amazon Data Firehose also supports database CDC replication (Preview, November 2024) — captures change data from MySQL and PostgreSQL databases and replicates directly to Apache Iceberg tables in S3, enabling near real-time data lake updates without custom ETL code.

Key Differences Summary

Feature Kinesis Data Streams Amazon Data Firehose
Capacity Mode Provisioned, On-Demand Standard, or On-Demand Advantage Fully managed (automatic)
Latency Real-time (~70-200 ms) Near real-time (60s default, ~5s with zero buffering)
Max Record Size 10 MiB (since Oct 2025) 1 MiB
Data Retention 24 hours to 365 days No storage
Replay ✅ Supported ❌ Not supported
Max Throughput Up to 10 GB/s (On-demand Advantage with warm throughput) Automatic scaling (gigabytes/second)
Enhanced Fan-Out Consumers Up to 50 (Advantage) / 20 (Standard/Provisioned) N/A — pre-defined destinations
Consumers Custom (SDK, KCL, Lambda) Pre-defined (S3, Redshift, OpenSearch, Iceberg, S3 Tables, Snowflake, etc.)
Use Case Custom processing, real-time analytics ETL, loading to data stores, CDC replication

KCL 1.x End of Support

⚠️ Important: Kinesis Client Library (KCL) 1.x reached end-of-support on January 30, 2026. AWS strongly recommends migrating KCL applications to KCL 2.x or later. KCL 1.x entered maintenance mode on April 17, 2025 with only critical bug fixes and security updates.

Migration: See Migrate consumers from KCL 1.x to KCL 2.x

AWS Certification Exam Practice Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  1. Your organization needs to ingest a big data stream into its data lake on Amazon S3. The data may stream in at a rate of hundreds of megabytes per second. What AWS service will accomplish the goal with the least amount of management?
    1. Amazon Data Firehose
    2. Amazon Kinesis Data Streams
    3. Amazon CloudFront
    4. Amazon SQS
  2. Your organization is looking for a solution that can help the business with streaming data several services will require access to read and process the same stream concurrently. What AWS service meets the business requirements?
    1. Amazon Data Firehose
    2. Amazon Kinesis Data Streams
    3. Amazon CloudFront
    4. Amazon SQS
  3. Your application generates a 1 KB JSON payload that needs to be queued and delivered to EC2 instances for applications. At the end of the day, the application needs to replay the data for the past 24 hours. In the near future, you also need the ability for other multiple EC2 applications to consume the same stream concurrently. What is the best solution for this?
    1. Kinesis Data Streams
    2. Amazon Data Firehose
    3. SNS
    4. SQS
  4. A company needs to stream data to Amazon S3 with the lowest possible latency (under 10 seconds). Which Kinesis service and configuration should they use? (Assume December 2023 or later)
    1. Kinesis Data Streams with Lambda consumer
    2. Amazon Data Firehose with zero buffering enabled
    3. Amazon Data Firehose with 60-second buffer
    4. Kinesis Data Streams with KCL consumer
  5. A company wants to avoid manual shard management for their Kinesis Data Streams and needs to handle instant traffic surges up to 10 GB/s. Which capacity mode should they use?
    1. Provisioned mode with Auto Scaling
    2. On-Demand Standard mode
    3. On-Demand Advantage mode with warm throughput
    4. Enhanced fan-out mode
  6. A data analytics team needs to stream real-time data into Apache Iceberg tables in S3 for analytics. Which AWS service supports this natively? (Assume October 2024 or later)
    1. Kinesis Data Streams
    2. Amazon Data Firehose
    3. AWS Glue Streaming
    4. Amazon MSK
  7. A company streams data using Kinesis Data Streams with 30 independent consumer applications needing dedicated throughput. Which configuration supports this?
    1. On-Demand Standard mode with enhanced fan-out
    2. Provisioned mode with enhanced fan-out
    3. On-Demand Advantage mode with enhanced fan-out
    4. Create separate streams for each consumer
  8. A company needs to stream IoT sensor data that occasionally includes 5 MiB payloads. Which streaming service supports this record size natively? (Assume October 2025 or later)
    1. Amazon Data Firehose
    2. Kinesis Data Streams
    3. Amazon SQS
    4. Amazon SNS
  9. A company wants to stream real-time data into Amazon S3 Tables with built-in Apache Iceberg support and automatic table maintenance. Which service should they use?
    1. Kinesis Data Streams with custom Lambda
    2. Amazon Data Firehose
    3. AWS Glue ETL job
    4. Amazon EMR Streaming

References

Amazon Data Firehose – Delivery & Transformation

Kinesis Data Firehose
~ Last updated on : ~ jayendrapatil ~ 8 Comments

Amazon Data Firehose (formerly Kinesis Data Firehose)

📢 Service Renamed (February 2024): Amazon Kinesis Data Firehose has been renamed to Amazon Data Firehose. The functionality remains the same. Existing applications, API endpoints, and IAM policies continue to work without changes.

  • Amazon Data Firehose is a fully managed service for delivering real-time streaming data to data stores and analytics tools.
  • Amazon Data Firehose automatically scales to match the throughput of the data (gigabytes per second or more) and requires no ongoing administration or need to write applications or manage resources.
  • is a data transfer solution for delivering real-time streaming data to destinations such as S3, Redshift, OpenSearch Service, OpenSearch Serverless, Apache Iceberg Tables, Amazon S3 Tables, Snowflake, Splunk, and third-party HTTP endpoints.
  • is NOT Real Time, but Near Real Time as it supports batching and buffers streaming data to a certain size (Buffer Size in MBs) or for a certain period of time (Buffer Interval in seconds) before delivering it to destinations.
    • Zero Buffering (December 2023): Firehose now supports zero buffering, delivering data within ~5 seconds with no buffering delay for real-time use cases.
  • supports data compression, minimizing the amount of storage used at the destination. It currently supports GZIP, ZIP, and SNAPPY compression formats. Only GZIP is supported if the data is further loaded to Redshift.
  • supports Apache Parquet and ORC format conversion — converts incoming JSON data to columnar formats optimized for analytics with Athena, Redshift Spectrum, and EMR before storing in S3.
  • supports data at rest encryption using KMS after the data is delivered to the S3 bucket.
  • supports 20+ data sources including Amazon Kinesis Data Streams, Amazon MSK (and MSK Serverless), Direct PUT API, Kinesis Agent, CloudWatch Logs, CloudWatch Events, AWS IoT Core, Amazon SNS, AWS WAF web ACL logs, and Amazon VPC Flow Logs.
  • supports out of box data transformation as well as custom transformation using the Lambda function to transform incoming source data and deliver the transformed data to destinations.
  • supports Dynamic Partitioning — groups streaming data by static or dynamically defined keys (e.g., customer_id, region) and delivers into key-unique S3 prefixes for optimized analytics.
  • supports source record backup with custom data transformation with Lambda, where Data Firehose will deliver the un-transformed incoming data to a separate S3 bucket.
  • uses at least once semantics for data delivery. In rare circumstances such as request timeout upon data delivery attempt, delivery retry by Firehose could introduce duplicates if the previous request eventually goes through.
  • supports Interface VPC Interface Endpoint (AWS Private Link) to keep traffic between the VPC and Data Firehose from leaving the Amazon network.

Amazon Data Firehose

Amazon Data Firehose Key Concepts

  • Data Firehose delivery stream
    • Underlying entity of Data Firehose, where the data is sent
  • Record
    • Data sent by data producer to a Data Firehose delivery stream
    • Maximum size of a record (before Base64-encoding) is 1024 KB.
    • With Amazon MSK as source, maximum record size is 10 MB (6 MB if Lambda transformation is enabled).
  • Data producer
    • Producers send records to Data Firehose delivery streams.
  • Buffer size and buffer interval
    • Data Firehose buffers incoming streaming data to a certain size or for a certain time period before delivering it to destinations
    • Buffer size and buffer interval can be configured while creating the delivery stream
    • Buffer size is in MBs and ranges from 1MB to 128MB for the S3 destination and 1MB to 100MB for the OpenSearch Service destination.
    • Buffer interval is in seconds and ranges from 0 secs (zero buffering) to 900 secs
    • Zero Buffering (December 2023): Set buffer interval to 0 seconds to deliver data within ~5 seconds with no buffering delay
    • Firehose raises buffer size dynamically to catch up and make sure that all data is delivered to the destination, if data delivery to the destination is falling behind data writing to the delivery stream
    • Buffer size is applied before compression.
  • Source
    • Data Firehose supports 20+ data sources:
      • Amazon Kinesis Data Streams — read directly from a KDS stream
      • Amazon MSK / MSK Serverless — consume from Kafka topics
      • Direct PUT — via Firehose API, SDK, or Kinesis Agent
      • AWS Services — CloudWatch Logs, CloudWatch Events, AWS IoT Core, Amazon SNS, AWS WAF web ACL logs, Amazon VPC Flow Logs, and others
  • Destination
    • A destination is the data store where the data will be delivered.
    • supports the following destinations:
      • Amazon S3 — with optional dynamic partitioning and format conversion
      • Amazon Redshift — via intermediate S3 COPY
      • Amazon OpenSearch Service
      • Amazon OpenSearch Serverless (added November 2022)
      • Apache Iceberg Tables in S3 (GA September 2024) — stream into Iceberg format tables with ACID transactions
      • Amazon S3 Tables (GA March 2025) — purpose-built managed Iceberg tables with automatic optimization
      • Snowflake — real-time streaming via Snowpipe Streaming
      • Splunk
      • Third-party HTTP endpoints — Datadog, Dynatrace, New Relic, MongoDB, Coralogix, Elastic, and others

Zero Buffering (December 2023)

  • Amazon Data Firehose now supports zero buffering for real-time data delivery
  • Delivers data within ~5 seconds with no buffering delay
  • Available for destinations: S3, OpenSearch Service, Redshift, and third-party HTTP endpoints
  • Enables real-time use cases that previously required Kinesis Data Streams
  • Trade-off: More frequent deliveries may result in more small files and higher costs

Dynamic Partitioning

  • Dynamically partition streaming data before delivery to S3 using static or dynamically defined keys (e.g., customer_id, transaction_id, region)
  • Firehose groups data by these keys and delivers into key-unique S3 prefixes
  • Enables high-performance, cost-efficient analytics with Athena, EMR, and Redshift Spectrum
  • Supports inline parsing (JQ expressions) to extract keys from JSON records without Lambda
  • Can be combined with data transformation (Lambda) for complex routing logic
  • Available only for S3 destination

Format Conversion (Parquet and ORC)

  • Firehose can convert incoming JSON data to columnar formats (Apache Parquet or Apache ORC) before storing in S3
  • Columnar formats are optimized for analytics cost and performance with Athena, Redshift Spectrum, and EMR
  • Uses AWS Glue Data Catalog schema for conversion
  • Reduces storage costs and improves query performance compared to raw JSON

Apache Iceberg Tables Support (GA September 2024)

  • Amazon Data Firehose can stream data directly into Apache Iceberg tables in S3
  • Iceberg brings SQL table reliability and ACID transactions to S3 data lakes
  • Supports automatic schema management, partitioning, and compaction
  • Compatible with Athena, EMR, Redshift, Spark, Flink, and other analytics engines
  • Simplifies data lake ingestion without custom ETL code
  • Content-based routing: Route records from a single stream to different Iceberg tables based on record content
  • Row-level operations: Apply update or delete operations for data correction and right-to-forget scenarios
  • Use cases: Real-time data lake ingestion, streaming analytics, CDC to data lake

Amazon S3 Tables Support (GA March 2025)

  • Amazon Data Firehose can deliver streaming data directly into Amazon S3 Tables — a purpose-built, managed Apache Iceberg table store
  • S3 Tables provide storage optimized for analytics workloads with built-in Apache Iceberg support
  • Delivers up to 3x faster query performance and 10x higher transactions per second compared to self-managed Iceberg tables in general purpose S3 buckets
  • Automatic continuous table optimization (compaction, snapshot management) without additional infrastructure
  • Supports content-based routing to different S3 Tables and row-level update/delete operations
  • Integrated with AWS Glue Data Catalog multi-catalog hierarchy (May 2025) — no resource links needed between default catalog and S3TablesCatalog
  • Compatible with Athena, EMR, Redshift, SageMaker Lakehouse, and other analytics engines
  • Use cases: Real-time data lake analytics, IoT data ingestion, streaming to data lakehouse

Amazon Data Firehose vs Kinesis Data Streams

Kinesis Data Streams vs. Amazon Data Firehose

AWS Certification Exam Practice Questions

  1. A user is designing a new service that receives location updates from 3600 rental cars every hour. The cars location needs to be uploaded to an Amazon S3 bucket. Each location must also be checked for distance from the original rental location. Which services will process the updates and automatically scale?
    1. Amazon EC2 and Amazon EBS
    2. Amazon Data Firehose and Amazon S3
    3. Amazon ECS and Amazon RDS
    4. Amazon S3 events and AWS Lambda
  2. You need to perform ad-hoc SQL queries on massive amounts of well-structured data. Additional data comes in constantly at a high velocity, and you don’t want to have to manage the infrastructure processing it if possible. Which solution should you use?
    1. Data Firehose and RDS
    2. EMR running Apache Spark
    3. Data Firehose and Redshift
    4. EMR using Hive
  3. Your organization needs to ingest a big data stream into their data lake on Amazon S3. The data may stream in at a rate of hundreds of megabytes per second. What AWS service will accomplish the goal with the least amount of management?
    1. Amazon Data Firehose
    2. Amazon Kinesis Data Streams
    3. Amazon CloudFront
    4. Amazon SQS
  4. A startup company is building an application to track the high scores for a popular video game. Their Solution Architect is tasked with designing a solution to allow real-time processing of scores from millions of players worldwide. Which AWS service should the Architect use to provide reliable data ingestion from the video game into the datastore?
    1. AWS Data Pipeline
    2. Amazon Data Firehose
    3. Amazon DynamoDB Streams
    4. Amazon OpenSearch Service
  5. A company has an infrastructure that consists of machines which keep sending log information every 5 minutes. The number of these machines can run into thousands and it is required to ensure that the data can be analyzed at a later stage. Which of the following would help in fulfilling this requirement?
    1. Use Data Firehose with S3 to take the logs and store them in S3 for further processing.
    2. Launch an Elastic Beanstalk application to take the processing job of the logs.
    3. Launch an EC2 instance with enough EBS volumes to consume the logs which can be used for further processing.
    4. Use CloudTrail to store all the logs which can be analyzed at a later stage.
  6. A company needs to stream data to Amazon S3 with the lowest possible latency (under 10 seconds). Which configuration should they use?
    1. Data Firehose with 60-second buffer
    2. Data Firehose with zero buffering enabled
    3. Kinesis Data Streams with Lambda consumer
    4. Direct PUT to S3
  7. A data analytics team needs to stream real-time data into Apache Iceberg tables in S3 for analytics with automatic table optimization and up to 3x faster queries. Which destination should they use?
    1. Apache Iceberg Tables in general-purpose S3 bucket
    2. Amazon S3 Tables
    3. Amazon Redshift with COPY command
    4. AWS Glue Streaming ETL
  8. A company streams millions of events per day from different applications. They need to route events to different analytics tables in S3 based on event type, with ACID transaction support. Which Data Firehose feature enables this?
    1. Dynamic Partitioning to S3 prefixes
    2. Content-based routing to Apache Iceberg Tables
    3. Lambda transformation with multiple outputs
    4. Multiple delivery streams
  9. A company wants to consume streaming data from an Amazon MSK cluster and load it into S3 in Parquet format without managing consumers or infrastructure. Which solution requires the LEAST effort?
    1. Write a custom Kafka consumer with Spark
    2. Use Amazon MSK Connect with S3 Sink Connector
    3. Amazon Data Firehose with MSK as source and Parquet format conversion
    4. AWS Glue Streaming ETL job

References

Amazon QuickSight

~ Last updated on : ~ jayendrapatil

Amazon QuickSight

🔄 Major Update (October 2025): Amazon QuickSight has evolved into Amazon Quick Suite, expanding from a BI tool to a unified AI-powered workspace. QuickSight’s core BI capabilities continue as Quick Sight within the Quick Suite umbrella. Quick Suite adds AI agents, deep research, workflow automation, and a shared knowledge base. Existing dashboards, datasets, APIs, and security controls remain unchanged. See Quick Suite section below for details.
  • QuickSight is a very fast, easy-to-use, cloud-powered business analytics service that makes it easy to build visualizations, perform ad-hoc analysis, and quickly get business insights from their data, anytime, on any device.
  • enables organizations to scale their business analytics capabilities to hundreds of thousands of users, and delivers fast and responsive query performance by using SPICE – a robust in-memory engine.
  • supports various datasources including
    • Excel files and flat files like CSV, TSV, CLF, ELF
    • On-premises databases like PostgreSQL, SQL Server and MySQL
    • SaaS applications like Salesforce
    • AWS data sources such as Redshift, RDS, Aurora, Athena, and S3
    • Google BigQuery (with Direct Query support for near real-time querying)
    • Snowflake and Starburst (with OAuth support for role-based access control)
  • supports various functions to format and transform the data.
    • alias data fields and change data types.
    • subset the data using built in filters and perform database join operations using drag and drop.
    • create calculated fields using mathematical operations and built-in functions such conditional statements, string, numerical and date functions
  • supports assorted visualizations that facilitate different analytical approaches:
    • Comparison and distribution – Bar charts (several assorted variants)
    • Changes over time – Line graphs, Area line charts
    • Correlation – Scatter plots, Heat maps
    • Aggregation – Pie graphs, Tree maps
    • Tabular – Pivot tables
    • Geospatial – Layer maps for custom shape file visualization
    • Custom – Highcharts visual for specialized charts (sunburst, network graphs, 3D charts)
  • comes with a built-in suggestion engine that provides suggested visualizations based on the properties of the underlying datasets
  • supports Stories, that provide guided tours through specific views of an analysis. They are used to convey key points, a thought process, or the evolution of an analysis for collaboration.
  • supports Pixel-Perfect Reports (Paginated Reports) for creating highly formatted, printable business documents with exact page layout control, scheduled email delivery, and export to PDF/CSV/Excel.

QuickSight Architecture

Super-fast, Parallel, In-memory Calculation Engine – SPICE

  • QuickSight is built with “SPICE” – a Super-fast, Parallel, In-memory Calculation Engine
  • SPICE uses a combination of columnar storage, in-memory technologies enabled through the latest hardware innovations and machine code generation to run interactive queries on large datasets and get rapid responses.
  • SPICE supports rich data discovery and business analytics capabilities to help customers derive valuable insights from their data without worrying about provisioning or managing infrastructure.
  • SPICE supports rich calculations to help derive valuable insights from the analysis without worrying about provisioning or managing infrastructure.
  • Data in SPICE is persisted until it is explicitly deleted by the user.
  • QuickSight can also be configured to keep the data in SPICE up-to-date as the data in the underlying sources change.
  • SPICE automatically replicates data for high availability and enables QuickSight to scale to hundreds of thousands of users who can all simultaneously perform fast interactive analysis across a wide variety of AWS data sources.
  • SPICE supports increased join sizes up to 20 GB (previously 1 GB) for secondary tables during data ingestion, enabling complex cross-source join tasks.
  • SPICE supports Customer Managed Keys (CMK) for encrypting datasets, allowing revocation of access with one click and auditable access logs.
  • SPICE supports Auto Purchase API to programmatically enable automatic capacity purchases when consumed capacity exceeds provisioned capacity.

Amazon Q in QuickSight – Generative BI

  • Amazon Q in QuickSight (GA April 2024) brings generative AI-powered business intelligence capabilities to the entire organization using natural language.
  • Amazon Q in QuickSight replaces and enhances the previous QuickSight Q NLQ feature, combining LLMs from Amazon Bedrock with QuickSight’s proven models.
  • Key capabilities include:
    • Multi-visual Data Q&A – ask questions of data not answered in existing dashboards and get instant visual answers
    • Executive Summaries – generate natural language summaries of dashboards to quickly discover key trends and insights
    • Data Stories – automatically build documents and slides explaining data with natural language prompts
    • Dashboard Authoring – build dashboards using simple prompts to describe visuals, calculations, and refinements
    • Pixel-Perfect Report Generation – create reports using natural language descriptions
    • Scenario Analysis (re:Invent 2024) – agentic data analysis to solve complex business questions 10x faster than spreadsheets
    • Unstructured Insights – unified insights from structured databases plus unstructured data (documents, websites, emails, images) via Amazon Q Business integration
  • Available by default to all Enterprise Edition users for data Q&A capabilities.
  • Available in US East (N. Virginia), US West (Oregon), Europe (Frankfurt, Ireland, London), Asia Pacific (Mumbai, Sydney, Tokyo), Canada (Central), and South America (São Paulo).

QuickSight User Roles

  • QuickSight Author ($24/month) – can connect to data sources, create visuals, analyze data, create interactive dashboards, and publish them with other users.
  • QuickSight Author Pro ($40/month, reduced from $50) – all Author capabilities plus:
    • Build dashboards with Amazon Q using natural language
    • Build and share Q Topics to enable Q&A on specific data
    • Amazon Q Business Pro user entitlement included
    • Access to Quick Suite Enterprise capabilities
  • QuickSight Reader ($3/month fixed) – consumes interactive dashboards, can log in via SAML/AD/QuickSight credentials, filter data, drill down, export as CSV, and ask questions of data with Amazon Q.
    • Readers do not have any allocated SPICE capacity.
  • QuickSight Reader Pro ($20/month) – all Reader capabilities plus:
    • Create data stories with Amazon Q
    • Generate executive dashboard summaries with Amazon Q
    • Amazon Q Business Pro user entitlement included
    • Access to Quick Suite Professional capabilities
  • Admin Pro – full generative BI capabilities plus account management permissions; billed as Author Pro.

QuickSight Security

  • QuickSight supports multi-factor authentication (MFA) for the AWS account via the AWS Management console.
  • For VPC with public connectivity, QuickSight’s IP address range can be added to the database instances’ security group rules to enable traffic flow into the VPC and database instances.
  • QuickSight supports Row-level security (RLS)
    • RLS enables dataset owners to control access to data at row granularity based on permissions associated with the user interacting with the data.
    • With RLS, QuickSight users only need to manage a single set of data and apply appropriate row-level dataset rules to it.
    • All associated dashboards and analyses will enforce these rules, simplifying dataset management and removing the need to maintain multiple datasets for users with different data access privileges.
  • QuickSight supports Column-level security (CLS)
    • CLS enables dataset owners to restrict access to specific columns in a dataset.
    • Users without access to restricted columns cannot create, view, or edit visuals that use those fields.
    • Combined with RLS, CLS provides fine-grained data governance over who can access what data within visualizations.
  • QuickSight supports Private VPC (Virtual Private Cloud) Access, which uses an Elastic Network Interface (ENI) for secure, private communication with data sources in a VPC. It also allows the use of AWS Direct Connect to create a secure, private link with the on-premises resources.
  • QuickSight supports AWS PrivateLink to securely connect to data sources through VPC endpoints, keeping network traffic within the AWS infrastructure.
  • QuickSight supports users defined via IAM, email signup, or AWS IAM Identity Center (with custom permissions for fine-grained capability control using directory groups).

QuickSight Editions

  • Standard Edition – available through the CreateAccountSubscription API for individual users and small groups. New Quick Suite features are not available in the Standard edition.
  • Enterprise Edition – offers enhanced functionality which includes:
    • QuickSight Readers
    • Amazon Q in QuickSight (Generative BI capabilities)
    • Connectivity to data sources in Private VPC
    • Row-level security and Column-level security
    • Hourly refresh of SPICE data
    • Encryption at Rest (with CMK support)
    • AD connectivity and group-based management of assets for AD accounts
    • AWS IAM Identity Center integration
    • Pixel-Perfect Reports
    • Embedded analytics
    • Brand customization

Amazon Quick Suite (October 2025)

  • Amazon QuickSight evolved into Amazon Quick Suite in October 2025, expanding from a BI tool to a unified AI-powered workspace.
  • Quick Suite represents an expansion of BI where AI agents work alongside users to answer complex questions, conduct deep research, and automate routine tasks.
  • The core BI functionality is now called Quick Sight within the Quick Suite umbrella.
  • Quick Suite introduces the following AI-powered capabilities:
    • Quick Research – delivers comprehensive, cited insights from both enterprise and public data sources in minutes
    • Quick Flows – helps users create and share workflow automations using natural language
    • Quick Automate – handles complex, multi-step business processes
    • Quick Index – a shared knowledge base of all company documents and data
    • Quick Chat – natural language interface to access all Quick Suite capabilities
    • Quick Spaces – customizable AI agents for specific teams (e.g., troubleshooting assistants, policy guides)
  • Existing QuickSight environments (dashboards, datasets, analyses, security, permissions, API integrations) remain unchanged—no data migration is required.
  • Compliance certifications (SOC, HIPAA, ISO 27001, GDPR, FedRAMP) remain valid.
  • AWS does not use customer content to train models.
  • Available in US East (N. Virginia), US West (Oregon), Europe (Dublin), and Asia Pacific (Sydney) for new capabilities; all regions see the new UI/branding.

QuickSight Embedding

  • QuickSight supports embedded analytics, allowing developers to embed interactive dashboards, visuals, Q&A experiences, and reports into applications and websites.
  • Supports 1-click embedding, SDK-based embedding, and API-based embedding.
  • Amazon Q in QuickSight generative BI capabilities (multi-visual Q&A, stories, executive summaries) can be embedded in applications.
  • Supports sharing views of embedded dashboards, allowing readers to share unique links with their specific filter/bookmark state.
  • Brand customization features allow overriding default Quick Suite styling for embedded scenarios.

AWS Certification Exam Practice Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  1. You are using QuickSight to identify demand trends over multiple months for your top five product lines. Which type of visualization do you choose?
    1. Scatter Plot
    2. Pie Chart
    3. Pivot Table
    4. Line Chart
  2. You need to provide customers with rich visualizations that allow you to easily connect multiple disparate data sources in S3, Redshift, and several CSV files. Which tool should you use that requires the least setup?
    1. Hue on EMR
    2. Redshift
    3. QuickSight
    4. Elasticsearch
  3. A company needs to restrict access to specific columns in a QuickSight dataset so that only certain users can view salary data. Which feature should they use?
    1. Row-level Security (RLS)
    2. Column-level Security (CLS)
    3. IAM policies
    4. VPC Security Groups
  4. A business analyst wants to ask natural language questions about sales data directly from a QuickSight dashboard without writing queries. Which feature enables this?
    1. QuickSight SPICE
    2. Amazon Q in QuickSight
    3. QuickSight Parameters
    4. QuickSight Calculated Fields
  5. A company wants to generate weekly pixel-perfect PDF reports from QuickSight dashboards and distribute them via email to stakeholders who don’t have QuickSight access. Which QuickSight feature should they use?
    1. QuickSight Stories
    2. QuickSight Embedded Analytics
    3. QuickSight Pixel-Perfect Reports with scheduled email delivery
    4. QuickSight SPICE exports
  6. An organization uses Amazon QuickSight and wants to ensure SPICE datasets are encrypted using their own encryption keys with the ability to revoke access. Which feature should they implement?
    1. AWS KMS default encryption
    2. S3 server-side encryption
    3. SPICE Customer Managed Keys (CMK)
    4. QuickSight VPC encryption
  7. A company recently rebranded Amazon QuickSight and wants to understand the relationship between QuickSight and Quick Suite. Which statement is correct? (Select TWO)
    1. Amazon Quick Suite is the evolution of Amazon QuickSight that adds AI agents, research, and workflow automation capabilities
    2. Quick Suite replaces QuickSight and requires data migration
    3. Existing QuickSight dashboards, APIs, and security controls continue to work unchanged within Quick Suite
    4. Quick Suite requires a new subscription separate from QuickSight
    5. Quick Suite is only available in the Standard Edition

References

 

AWS DMS – Database Migration Service Overview

DMS Migration
~ Last updated on : ~ jayendrapatil

AWS Database Migration Service – DMS

  • AWS Database Migration Service enables quick and secure data migration with minimal to zero downtime.
  • Database Migration Service helps migration to AWS with virtually no downtime. The source database remains fully operational during the migration, minimizing downtime to applications that rely on the database.
  • AWS DMS can migrate
    • relational databases, data warehouses, NoSQL databases, and other types of data stores
    • data to and from the most widely used commercial and open-source databases.
  • DMS supports homogeneous migrations such as Oracle to Oracle, as well as heterogeneous migrations (using SCT/DMS Schema Conversion) between different database platforms, such as Oracle or Microsoft SQL Server to Aurora.
  • AWS DMS Schema Conversion (formerly AWS Schema Conversion Tool) helps in heterogeneous database migrations by automatically converting the source database schema and a majority of the database code objects, including views, stored procedures, and functions, to a format compatible with the target database.
  • DMS enables both one-time migration and continuous data replication with high availability and consolidates databases into a petabyte-scale data warehouse by streaming data to Redshift and S3.
  • DMS continually monitors source and target databases, network connectivity, and the replication instance.
  • DMS automatically manages all of the infrastructure that supports the migration server, including hardware and software, software patching, and error reporting.
  • DMS is highly resilient and self–healing. If the primary replication server fails for any reason, a backup replication server can take over with little or no interruption of service.
  • In case of interruption, DMS automatically restarts the process and continues the migration from where it was halted.
  • AWS DMS supports the Multi-AZ option to provide high availability for database migration and continuous data replication by enabling redundant replication instances.
  • AWS DMS ensures that the data migration is secure. Data at rest is encrypted with AWS KMS encryption. During migration, SSL can be used to encrypt the in-flight data as it travels from source to target.
  • DMS integrates with AWS Secrets Manager for centralized storage and management of database credentials and other sensitive information.
  • DMS supports data masking capabilities that can conceal sensitive data by modifying the contents of specified columns before loading them into the target database.

DMS Serverless

  • AWS DMS Serverless (GA since June 2023) removes the need to provision, monitor, and scale replication instances manually.
  • DMS Serverless automatically provisions, scales, and manages migration resources to the optimal capacity needed to meet demand.
  • Eliminates infrastructure tasks like capacity estimation, provisioning, cost-optimization, and managing replication engine versions and patching.
  • Supports several use cases including continuous data replication, database consolidation, and both heterogeneous and homogeneous migrations.
  • DMS Serverless supports S3 as a source endpoint for migrating data in CSV or Parquet format to any supported target endpoint (GA Feb 2025).
  • DMS Serverless provides automatic storage scaling (April 2025), removing the previous 100 GB default storage capacity limit for high transaction volumes.
  • You only pay for the data migration capacity you use—no need to overprovision for peak demand.

Database Migration Service Components

DMS Migration

DMS Replication Instance

  • A DMS replication instance performs the actual data migration between the source and the target.
  • DMS replication instance is a managed EC2 instance that hosts one or more replication tasks.
  • The replication instance also caches the transaction logs during the migration.
  • CPU and memory capacity of the replication instance influences the overall time required for the migration.
  • DMS can provide high availability and failover support using a Multi-AZ deployment.
    • In a Multi-AZ deployment, DMS automatically provisions and maintains a standby replica of the replication instance in a different AZ
    • Primary replication instance is synchronously replicated to the standby replica.
    • If the primary replication instance fails or becomes unresponsive, the standby resumes any running tasks with minimal interruption.
    • Because the primary is constantly replicating its state to the standby, Multi-AZ deployment does incur some performance overhead.

Endpoints

  • AWS DMS uses an endpoint to access the source or target data store.
  • DMS supports 20+ databases and data warehouses as source and target endpoints.

Replication tasks

  • DMS replication task helps move a set of data from the source endpoint to the target endpoint.
  • Replication task required Replication instance, source, and target endpoints
  • Replication task supports following migration type options
    • Full load (Migrate existing data) – Migrate the data from the source to the target database as a one-time migration.
    • CDC only (Replicate data changes only) – Replicate only changes, while using native export tools for performing bulk data load.
    • Full load + CDC (Migrate existing data and replicate ongoing changes) – Performs a full data load while capturing changes on the source. After the full load is complete, captured changes are applied to the target. Once the changes reach a steady state, the applications can be switched over to the target.
  • LOB mode options
    • Don’t include LOB columns – LOB columns are excluded
    • Full LOB mode – Migrate complete LOBs regardless of size. AWS DMS migrates LOBs piecewise in chunks controlled by the Max LOB Size parameter. This mode is slower than using limited LOB mode.
    • Limited LOB mode – Truncate LOBs to the value specified by the Max LOB Size parameter. This mode is faster than using full LOB mode.
  • Data validation – validates data between source and target after migration.
  • Data Resync (May 2025) – automatically corrects data inconsistencies identified during validation by retrieving current values from the source and applying them to the target.

DMS Homogeneous Data Migrations

  • DMS Homogeneous Data Migrations use built-in native database tooling to provide simple and performant like-to-like migrations with minimal downtime.
  • Homogeneous data migrations are serverless — DMS automatically provisions the resources required for migration without needing to manage replication instances.
  • Supports migrating all data types, secondary objects (functions, stored procedures, triggers, indexes), and partitions.
  • Supported source databases include:
    • PostgreSQL (version 10.4–14.x)
    • MySQL (version 5.7 and higher)
    • MariaDB (version 10.2 and higher)
    • MongoDB (version 4.x and higher)
  • Supported targets include Amazon RDS, Amazon Aurora, and Amazon DocumentDB.
  • Available via AWS Console, CLI, SDK, and API (October 2024).
  • Supports a “1-Click Move to Managed” feature (November 2024) for auto-migration of EC2 self-managed databases to Amazon RDS or Aurora.
  • Available in 29 AWS Regions (August 2024).

DMS Schema Conversion

  • AWS DMS Schema Conversion (DMS SC) is a fully managed schema conversion capability integrated directly into the DMS console.
  • Previously, you had to download the standalone AWS Schema Conversion Tool (AWS SCT). Now, DMS SC provides a fully managed experience from within the DMS console.
  • DMS SC converts source database schema and a majority of database code objects, including views, stored procedures, and functions, to a format compatible with the target database.
  • DMS SC supports heterogeneous migrations by automatically assessing, converting, and migrating database and code objects.
  • DMS SC clearly marks any objects that cannot be automatically converted so that they can be manually converted.
  • Provides automated assessment and conversion reports that analyze the entire source database with detailed conversion estimates for every schema object.
  • Generative AI-Assisted Schema Conversion (December 2024):
    • Uses large language models (LLMs) hosted on Amazon Bedrock combined with traditional rule-based conversion
    • Automatically converts up to 90% of complex database objects including stored procedures, triggers, and functions
    • Supports conversions from Oracle, SQL Server, MySQL, PostgreSQL, and Sybase ASE to Amazon Aurora PostgreSQL and RDS for PostgreSQL
    • Available in 9+ additional regions (March 2026)
  • Amazon Q Developer can scan Java application source code and convert embedded SQL from Oracle to Aurora PostgreSQL or RDS for PostgreSQL.
  • The standalone AWS SCT tool is still available for download but AWS recommends using DMS Schema Conversion for new projects.

AWS SCT DMS Heterogeneous Migration

Database Migration Service Best Practices

  • DMS Performance
    • In full load, multiple tables are loaded in parallel and it is recommended to drop primary key indexes, secondary indexes, referential integrity constraints, and data manipulation language (DML) triggers.
    • For a full load + CDC task, it is recommended to add secondary indexes before the CDC phase. Because AWS DMS uses logical replication, secondary indexes that support DML operations should be in-place to prevent full table scans.
    • Replication task can be paused before the CDC phase to build indexes, create triggers, and create referential integrity constraints
    • Use multiple tasks for a single migration to improve performance
    • Disable backups and Multi-AZ on the target until ready to cut over.
    • For homogeneous migrations, use DMS Homogeneous Data Migrations (native tools) for best performance.
    • Consider DMS Serverless to eliminate capacity estimation overhead and allow automatic scaling.
  • Migration LOBs
    • DMS migrates LOBS in a two step process
      • creates a new row in the target table and populates the row with all data except the associated LOB value.
      • Update the row in the target table with the LOB data.
    • All LOB columns on the target table must be nullable
    • Limited LOB mode
      • default for all migration tasks
      • migrates all LOB values up to a user-specified size limit, default 32K
      • LOB values larger than the size limit must be manually migrated. typically provides the best performance.
      • Ensure that the Max LOB size parameter setting is set to the largest LOB size for all the tables.
    • Full LOB mode
      • migrates all LOB data in the tables, regardless of size.
      • provides the convenience of moving all LOB data in the tables, but the process can have a significant impact on performance.
  • Migrating Large Tables
    • Break the migration into more than one task.
    • Using row filtering, use a key or a partition key to create multiple tasks
  • Convert schema
    • Use DMS Schema Conversion (DMS SC) or AWS SCT to convert the source objects, table, indexes, views, triggers, and other system objects into the target DDL format
    • DMS doesn’t perform schema or code conversion
  • Replication
    • Enable Multi-AZ for ongoing replication (for high availability and failover support)
  • DMS can read/write from/to encrypted DBs
  • Data Validation
    • Enable data validation to compare source and target data after migration
    • Use Data Resync to automatically fix any identified inconsistencies

DMS Fleet Advisor

⚠️ DMS Fleet Advisor – End of Support (May 20, 2026)

AWS DMS Fleet Advisor will reach End of Support on May 20, 2026.

No new customers accepted since May 20, 2025. Existing customers can continue using Fleet Advisor until May 20, 2026.

Migration Recommendation: Migrate database assessment projects to AWS Migration Evaluator before May 20, 2026.

  • AWS DMS Fleet Advisor is a free, fully managed capability of AWS DMS that automates migration planning and helps you migrate database and analytics fleets to the cloud at scale with minimal effort.
  • DMS Fleet Advisor is intended for users looking to migrate a large number of database and analytics servers to AWS.
  • AWS DMS Fleet Advisor helps discover and analyze the OLTP and OLAP database workloads and allows building a customized migration plan by determining the complexity of migrating the source databases to target services in AWS.

DMS Security

  • DMS implements a comprehensive security framework using:
    • IAM policies – Define specific permissions for users, groups, and roles for controlled access
    • SSL/TLS encryption – Secures endpoint connections and data in transit
    • AWS KMS – Encryption of data at rest with customer managed keys
    • AWS Secrets Manager – Centralized storage and management of credentials
    • VPC isolation – Replication instances created within a VPC for secure resource isolation
    • Data masking – Conceal sensitive data by modifying column contents before loading into target
  • Monitoring and visibility through Amazon SNS, CloudWatch, CloudTrail, and database logs.

AWS Certification Exam Practice Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  1. Which AWS service would simplify the migration of a database to AWS?
    1. AWS Storage Gateway
    2. AWS Database Migration Service (AWS DMS)
    3. Amazon Elastic Compute Cloud (Amazon EC2)
    4. Amazon AppStream 2.0
  2. A company wants to migrate an on-premises Oracle database to Amazon Aurora PostgreSQL with minimal downtime. Which combination of services should be used? (Select TWO)
    1. AWS DMS for data migration
    2. AWS DataSync
    3. AWS DMS Schema Conversion for schema conversion
    4. AWS Transfer Family
    5. Amazon Kinesis Data Streams
  3. A company needs to migrate a self-managed MySQL database on EC2 to Amazon Aurora MySQL with the least operational overhead. Which DMS feature should they use?
    1. DMS Replication Instance with Full Load + CDC
    2. DMS Homogeneous Data Migrations
    3. AWS Schema Conversion Tool
    4. DMS Fleet Advisor
  4. A solutions architect needs to set up a database migration that automatically scales resources based on workload demand without manual capacity planning. Which option should be used?
    1. DMS Multi-AZ replication instance
    2. DMS Serverless
    3. DMS Fleet Advisor
    4. DMS Replication Instance with t3.large
  5. After completing a DMS full load migration, a company discovers data inconsistencies between the source and target databases. Which DMS feature can automatically resolve these discrepancies?
    1. DMS Data Validation only
    2. DMS CDC task
    3. DMS Data Resync
    4. DMS Full Load restart

References

AWS SQS Standard Queue

~ Last updated on : ~ jayendrapatil ~ 1 Comment

AWS SQS Standard Queue

  • SQS offers standard as the default queue type.
  • Standard queues support at-least-once message delivery. However, occasionally (because of the highly distributed architecture that allows nearly unlimited throughput), more than one copy of a message might be delivered out of order.
  • Standard queues support a nearly unlimited number of API calls per second, per API action (SendMessage, ReceiveMessage, or DeleteMessage).
  • Standard queues provide best-effort ordering which ensures that messages are generally delivered in the same order as they’re sent.
  • SQS processes over 100 million messages per second at peak times, making it a fundamental building block for microservices, distributed systems, and serverless applications.
  • All SQS queues are encrypted at rest by default using SQS-owned encryption keys (SSE-SQS) since September 2022.

SQS Standard Queue Features

Redundant infrastructure

  • offers reliable and scalable hosted queues for storing messages
  • is engineered to always be available and deliver messages
  • provides the ability to store messages in a fail safe queue
  • highly concurrent access to messages

At-Least-Once delivery

  • ensures delivery of each message at least once
  • stores copies of the messages on multiple servers for redundancy and high availability
  • might deliver duplicate copy of messages, if the servers storing a copy of a message is unavailable when you receive or delete the message and the copy of the message is not deleted on that unavailable server
  • Applications should be designed to be idempotent with the ability to handle duplicate messages and not be adversely affected if it processes the same message more than once

Message Attributes

  • SQS messages can contain up to 10 metadata attributes.
  • take the form of name-type-value triples
  • can be used to separate the body of a message from the metadata that describes it.
  • helps process and store information with greater speed and efficiency because the applications don’t have to inspect an entire message before understanding how to process it

Message Sampling

  • behavior of retrieving messages from the queue depends on whether short (standard) polling, the default behavior, or long polling is used
  • With short polling,
    • samples only a subset of the servers (based on a weighted random distribution) and returns messages from just those servers.
    • A receive request might not return all the messages in the queue. But a subsequent receive request would return the message
  • With Long polling,
    • request persists for the time specified and returns as soon as the message is available thereby reducing costs and time the message has to dwell in the queue
    • long polling doesn’t return a response until a message arrives in the message queue, or the long poll times out.
    • makes it inexpensive to retrieve messages from the SQS queue as soon as the messages are available.
    • might help reduce the cost of using SQS, as the number of empty receives are reduced

Batching

  • SQS allows send, receive and delete batching, which helps club up to 10 messages in a single batch while charging price for a single message
  • helps lower cost and also increases the throughput

Configurable settings per queue

  • All queues don’t have to be alike

Order

  • makes a best effort to preserve order in messages does not guarantee first in, first out delivery of messages
  • can be handled by placing sequencing information within the message and performing the ordering on the client side

Loose coupling

  • removes tight coupling between components
  • provides the ability to move data between distributed components of the applications that perform different tasks without losing messages or requiring each component to be always available

Multiple writers and readers

  • supports multiple readers and writers interacting with the same queue as the same time
  • locks the message during processing, using Visibility Timeout, preventing it to be processed by any other consumer

Variable message size

  • supports message in any format up to 256KB of text.
  • messages larger than 256 KB can be managed using the S3 or DynamoDB, with SQS holding a pointer to the S3 object

Access Control

  • Access can be controlled for who can produce and consume messages to each queue
  • Supports Attribute-Based Access Control (ABAC) using queue tags for flexible and scalable permissions management
  • ABAC allows defining fine-grained permissions based on resource tags reflecting organizational structures (e.g., department, environment, team)

Delay Queues

  • Delay queue allows the user to set a default delay on a queue such that delivery of all messages enqueued is postponed for that time duration

Dead Letter Queues

  • Dead letter queue is a queue for messages that were not able to be processed after a maximum number of attempts
  • useful to isolate messages that can’t be processed for later analysis.
  • DLQ Redrive allows moving messages from a DLQ back to the source queue or a custom destination queue for reprocessing
  • DLQ Redrive can be managed programmatically using APIs: StartMessageMoveTask, ListMessageMoveTasks, and CancelMessageMoveTask
  • DLQ Redrive is supported for both standard and FIFO queues

Server-Side Encryption

  • All SQS queues are encrypted at rest by default using SQS-owned encryption keys (SSE-SQS) with AES-256 encryption
  • Optionally, queues can use AWS KMS keys (SSE-KMS) for encryption with customer-managed keys
  • For SSE-KMS queues, only kms:GenerateDataKey permission is required for SendMessage (kms:Decrypt no longer needed for sending)
  • kms:Decrypt permission is still required for ReceiveMessage
  • SSE-SQS encryption is provided at no additional charge

PCI Compliance

  • supports the processing, storage, and transmission of credit card data by a merchant or service provider, and has been validated as being PCI-DSS (Payment Card Industry – Data Security Standard) compliant

SQS Fair Queues (Standard Queues)

  • Fair Queues is a feature of standard queues that automatically mitigates noisy-neighbor impact in multi-tenant queues (launched July 2025)
  • Addresses scenarios where one tenant creates a message backlog that increases dwell time for all other tenants
  • SQS detects noisy neighbors by monitoring message distribution among tenants during processing (in-flight state)
  • When a tenant has disproportionately large number of in-flight messages, SQS identifies it as a noisy neighbor and prioritizes message delivery for other tenants
  • Does not limit the consumption rate per tenant — allows consumers to receive messages from noisy neighbor tenants when there is spare consumer capacity
  • Supports virtually unlimited throughput with no limits on the number of tenants

How to Use Fair Queues

  • Message producers set a MessageGroupId on outgoing messages as a tenant identifier
  • Fairness is applied automatically to all standard queues for messages with the MessageGroupId property
  • Does NOT require any consumer code changes
  • No impact on API latency and no throughput limitations
  • Note: MessageGroupId on standard queues (for fair queues) does NOT enforce message ordering like it does on FIFO queues — it is used only as a tenant identifier

When to Use Fair Queues

  • The queue is multi-tenant (carries messages for multiple logical entities)
  • The queue is high-throughput (at low throughput, bursts rarely create backlogs)
  • Dwell time is part of the application’s quality of service

Fair Queues CloudWatch Metrics

  • Provides additional CloudWatch metrics for monitoring noisy-neighbor mitigation
  • ApproximateNumberOfMessagesVisibleInQuietGroups — shows backlog for non-noisy tenants
  • Allows comparing quiet group metrics with standard queue-level metrics to identify impact isolation

SQS Performance and Networking

JSON Protocol

  • SQS supports the AWS JSON protocol for API requests (available since November 2023)
  • Reduces end-to-end message processing latency by up to 23% compared to the legacy query protocol
  • Reduces application client-side CPU and memory usage
  • Enabled automatically by updating to supported AWS SDK versions

Dual-Stack (IPv4 and IPv6) Endpoints

  • SQS supports dual-stack endpoints allowing queues to be accessed via both IPv4 and IPv6 (launched April 2025)
  • Enables applications on IPv6-only networks to interact with SQS
  • Also available on VPC endpoints (July 2025) for gradual IPv4 to IPv6 transitions
  • Dual-stack endpoint format: sqs.{region}.amazonaws.com

CloudTrail Integration

  • CloudTrail integration is available for all Amazon SQS APIs (January 2025)
  • Provides comprehensive audit logging for all SQS operations

SQS Integration with AWS Lambda

  • SQS queues can trigger Lambda functions using Event Source Mappings (ESM)
  • Lambda polls the queue and invokes the function with batches of messages
  • Provisioned Mode for SQS ESM (launched November 2025):
    • Provides dedicated event polling resources for the SQS event source mapping
    • Offers 3x faster scaling — up to 1000 event pollers per concurrent executions per minute
    • Provides up to 16x higher concurrency — up to 20,000 concurrent processing capacity
    • Allows defining minimum and maximum limits for provisioned event pollers
    • Handles sudden traffic spikes more effectively through responsive autoscaling
  • The Lambda function and the SQS queue must be in the same AWS Region (can be in different accounts)

SQS Security Features

  • Encryption at Rest: All queues encrypted by default with SSE-SQS (AES-256); optional SSE-KMS for customer-managed keys
  • Encryption in Transit: All API requests use HTTPS (TLS)
  • IAM Policies: Identity-based and resource-based policies for access control
  • ABAC: Attribute-based access control using queue tags
  • VPC Endpoints: Access SQS from within a VPC without traversing the internet; supports VPC endpoint policies
  • SQSUnlockQueuePolicy: AWS-managed policy to unlock a queue with a misconfigured policy that denies all principals (November 2024)
  • AWS X-Ray: Troubleshoot messages passing through queues using trace headers

SQS Standard Queues vs SQS FIFO Queues

SQS Standard vs FIFO Queues

AWS Certification Exam Practice Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  1. A company has a multi-tenant SaaS application where multiple customers send messages to a shared SQS standard queue. During peak hours, one customer generates significantly more messages than others, causing increased message dwell time for all customers. Which SQS feature should the architect recommend to mitigate this issue?
    1. Use FIFO queues with message group IDs
    2. Enable fair queues with MessageGroupId as tenant identifier
    3. Create separate queues for each customer
    4. Increase the visibility timeout

    Answer: b – SQS Fair Queues automatically mitigate noisy-neighbor impact by monitoring message distribution and prioritizing delivery for quiet tenants when a noisy neighbor is detected.

  2. A developer needs to reduce SQS message processing latency without changing the application architecture. Which approach provides up to 23% latency reduction?
    1. Switch from standard to FIFO queue
    2. Enable long polling with maximum wait time
    3. Update the AWS SDK to use the JSON protocol for SQS
    4. Increase the batch size to 10 messages

    Answer: c – The AWS JSON protocol for SQS reduces end-to-end message processing latency by up to 23% and reduces client-side CPU and memory usage.

  3. A team is using Lambda to process messages from an SQS queue. They experience delays during sudden traffic spikes because Lambda’s event pollers take time to scale up. Which feature addresses this?
    1. Increase Lambda reserved concurrency
    2. Enable provisioned mode for the SQS event source mapping
    3. Reduce the batch size
    4. Use multiple Lambda functions polling the same queue

    Answer: b – Provisioned mode for SQS ESM provides dedicated polling resources with 3x faster scaling and up to 16x higher concurrency to handle sudden traffic spikes.

  4. A solutions architect is configuring an SQS queue with SSE-KMS encryption. Which IAM permissions are required for a producer to send messages to the queue?
    1. kms:Encrypt and kms:Decrypt
    2. kms:GenerateDataKey only
    3. kms:GenerateDataKey and kms:Decrypt
    4. kms:Encrypt only

    Answer: b – Since July 2024, SQS no longer requires kms:Decrypt for SendMessage. Only kms:GenerateDataKey is needed for sending; kms:Decrypt is still required for ReceiveMessage.

  5. An application architect needs to programmatically move messages from a dead-letter queue back to the source queue for reprocessing. Which SQS API should be used?
    1. MoveMessage
    2. RedriveMessage
    3. StartMessageMoveTask
    4. SendMessage with the source queue URL

    Answer: c – The StartMessageMoveTask API starts an asynchronous task to move messages from a DLQ to a specified destination queue. ListMessageMoveTasks and CancelMessageMoveTask are also available.

  6. Which statement about SQS fair queues MessageGroupId on standard queues is correct?
    1. It enforces strict FIFO ordering within the group
    2. It is used only as a tenant identifier and does not enforce message ordering
    3. It limits throughput to 300 messages per second per group
    4. It requires consumer code changes to process grouped messages

    Answer: b – On standard queues, MessageGroupId is used only as a tenant identifier for fair queues and does not enforce message ordering (unlike FIFO queues).

References

AWS Data Analytics Services Cheat Sheet

AWS Data Analytics Services
~ Last updated on : ~ jayendrapatil ~ 7 Comments

AWS Data Analytics Services Cheat Sheet

📋 Last Updated: June 2026. This post has been updated to reflect service renamings (Kinesis Data Firehose → Amazon Data Firehose, Kinesis Data Analytics → Amazon Managed Service for Apache Flink, Elasticsearch → OpenSearch Service, QuickSight → Quick Suite), deprecations (AWS Data Pipeline, Kinesis Data Analytics for SQL), and major new features (Zero-ETL integrations, MSK Express brokers, Glue 5.0, SageMaker Lakehouse).

AWS Data Analytics Services

Kinesis Data Streams – KDS

  • enables real-time processing of streaming data at a massive scale
  • provides ordering of records per shard
  • provides an ability to read and/or replay records in the same order
  • allows multiple applications to consume the same data
  • data is replicated across three data centers within a region
  • data is preserved for 24 hours, by default, and can be extended to 365 days
  • data inserted in Kinesis, it can’t be deleted (immutability) but only expires
  • streams can be scaled using multiple shards, based on the partition key
  • each shard provides the capacity of 1MB/sec data input and 2MB/sec data output with 1000 PUT requests per second
  • supports two capacity modes:
    • Provisioned mode – you manage the number of shards
    • On-demand mode – automatically scales to accommodate up to 10 GB/s write and 20 GB/s read throughput per stream
  • On-demand Advantage mode (launched Nov 2025) – enables on-demand streams to handle instant throughput increases via warm throughput capability, with up to 10GB/s or 10 million events/second, eliminating over-provisioning needs and offering 60%+ cost savings for consistent workloads
  • supports record sizes up to 10 MiB (increased from 1 MiB in Oct 2025)
  • supports up to 50 enhanced fan-out consumers per stream (increased from 20 in Nov 2025)
  • Kinesis vs SQS
    • real-time processing of streaming big data vs reliable, highly scalable hosted queue for storing messages
    • ordered records, as well as the ability to read and/or replay records in the same order vs no guarantee on data ordering (with the standard queues before the FIFO queue feature was released)
    • data storage up to 24 hours, extended to 365 days vs 1 minute to extended to 14 days but cleared if deleted by the consumer.
    • supports multiple consumers vs a single consumer at a time and requires multiple queues to deliver messages to multiple consumers.
  • Kinesis Producer
    • API
      • PutRecord and PutRecords are synchronous
      • PutRecords uses batching and increases throughput
      • might experience ProvisionedThroughputExceeded Exceptions, when sending more data. Use retries with backoff, resharding, or change partition key.
    • KPL
      • producer supports synchronous or asynchronous use cases
      • supports inbuilt batching and retry mechanism
      • ⚠️ KPL 0.x reached end-of-support on January 30, 2026. Migrate to KPL 1.x.
    • Kinesis Agent can help monitor log files and send them to KDS
    • supports third-party libraries like Spark, Flume, Kafka connect, etc.
  • Kinesis Consumers
    • Kinesis SDK
      • Records are polled by consumers from a shard
    • Kinesis Client Library (KCL)
      • Read records from Kinesis produced with the KPL (de-aggregation)
      • supports the checkpointing feature to keep track of the application’s state and resume progress using the DynamoDB table.
      • if application receives provisioned-throughput exceptions, increase the provisioned throughput for the DynamoDB table
      • ⚠️ KCL 1.x reached end-of-support on January 30, 2026. Migrate to KCL 2.x.
    • Kinesis Connector Library – can be replaced using Firehose or Lambda
    • Third-party libraries: Spark, Log4J Appenders, Flume, Kafka Connect…
    • Amazon Data Firehose, AWS Lambda
    • Kinesis Consumer Enhanced Fan-Out
      • supports Multiple Consumer applications for the same Stream
      • provides Low Latency ~70ms
      • Higher costs
      • now supports up to 50 consumers per stream
  • Kinesis Security
    • allows access/authorization control using IAM policies
    • supports Encryption in flight using HTTPS endpoints
    • supports data encryption at rest using either server-side encryption with KMS or using client-side encryption before pushing the data to data streams.
    • supports VPC Endpoints to access within VPC

Amazon Data Firehose

(Previously known as Amazon Kinesis Data Firehose, renamed February 2024)

  • data transfer solution for delivering near real-time streaming data to destinations such as S3, Redshift, OpenSearch Service, Splunk, Snowflake, and other 3rd-party analytics services.
  • is a fully managed service that automatically scales to match the throughput of your data and requires no ongoing administration
  • is Near Real Time (min. 60 secs) as it buffers incoming streaming data to a certain size or for a certain period of time before delivering it
  • supports batching, compression, and encryption of the data before loading it, minimizing the amount of storage used at the destination and increasing security
  • supports data compression, minimizing the amount of storage used at the destination. It currently supports GZIP, ZIP, and SNAPPY compression formats. Only GZIP is supported if the data is further loaded to Redshift.
  • supports out of box data transformation as well as custom transformation using Lambda function to transform incoming source data and deliver the transformed data to destinations
  • uses at least once semantics for data delivery.
  • supports multiple producers as datasource, which include Kinesis data stream, KPL, Kinesis Agent, or the Data Firehose API using the AWS SDK, CloudWatch Logs, CloudWatch Events, or AWS IoT
  • does NOT support consumers like Spark and KCL
  • supports interface VPC endpoint to keep traffic between the VPC and Data Firehose from leaving the Amazon network.
  • Apache Iceberg Tables destination (launched 2024) – delivers streaming data directly into Apache Iceberg format tables in S3 and S3 Tables, supporting record routing to different Iceberg tables, CDC replication from databases, schema evolution, and ACID transactions.
  • Database CDC replication (Preview 2024) – supports continuous replication of database changes from MySQL and PostgreSQL directly into Apache Iceberg Tables in S3.

Kinesis Data Streams vs Amazon Data Firehose

Amazon Managed Service for Apache Flink

(Previously known as Amazon Kinesis Data Analytics, renamed August 2023)

⚠️ Kinesis Data Analytics for SQL was discontinued on January 27, 2026. Migrate to Amazon Managed Service for Apache Flink or Apache Flink Studio for real-time stream processing.

  • helps analyze streaming data, gain actionable insights, and respond to the business and customer needs in real time.
  • is a fully managed and serverless service for building and running real-time streaming applications using Apache Flink.
  • reduces the complexity of building, managing, and integrating streaming applications with other AWS services.
  • supports Apache Flink applications written in Java, Scala, Python, and SQL.
  • provides automatic scaling, high availability, and exactly-once processing semantics.
  • integrates with Kinesis Data Streams, Amazon MSK, and Amazon S3 as data sources and sinks.

Managed Streaming for Kafka – MSK

  • Managed Streaming for Kafka- MSK is an AWS streaming data service that manages Apache Kafka infrastructure and operations.
  • makes it easy for developers and DevOps managers to run Kafka applications and Kafka Connect connectors on AWS, without the need to become experts in operating Kafka.
  • operates, maintains, and scales Kafka clusters, provides enterprise-grade security features out of the box, and has built-in AWS integrations that accelerate development of streaming data applications.
  • always runs within a VPC managed by the MSK and is available to your own selected VPC, subnet, and security group when the cluster is setup.
  • IP addresses from the VPC are attached to the MSK resources through elastic network interfaces (ENIs), and all network traffic stays within the AWS network and is not accessible to the internet by default.
  • integrates with CloudWatch for monitoring, metrics, and logging.
  • MSK Serverless is a cluster type for MSK that makes it easy for you to run Apache Kafka clusters without having to manage compute and storage capacity.
  • MSK Express Brokers (GA November 2024) – a new broker type for MSK Provisioned designed to deliver:
    • up to 3x more throughput per broker (500 MBps ingress, 1000 MBps egress on m7g instances)
    • up to 20x faster scaling
    • 90% faster recovery from failures
    • up to 5x more partitions per broker
    • virtually unlimited storage with instant storage scaling
    • supports Intelligent Rebalancing for 180x faster operation performance
  • supports EBS server-side encryption using KMS to encrypt storage.
  • supports encryption in transit enabled via TLS for inter-broker communication.
  • For provisioned clusters, you have three options:
    • IAM Access Control for both AuthN/Z (recommended),
    • TLS certificate authentication (CA) for AuthN and access control lists for AuthZ
    • SASL/SCRAM for AuthN and access control lists for AuthZ.
  • For serverless clusters, IAM Access Control can be used for both authentication and authorization.

Redshift

  • Redshift is a fast, fully managed data warehouse
  • provides simple and cost-effective solutions to analyze all the data using standard SQL and the existing Business Intelligence (BI) tools.
  • manages the work needed to set up, operate, and scale a data warehouse, from provisioning the infrastructure capacity to automating ongoing administrative tasks such as backups, and patching.
  • automatically monitors your nodes and drives to help you recover from failures.
  • only supported Single-AZ deployments. However, now supports Multi-AZ deployments.
  • replicates all the data within the data warehouse cluster when it is loaded and also continuously backs up your data to S3.
  • attempts to maintain at least three copies of your data (the original and replica on the compute nodes and a backup in S3).
  • supports cross-region snapshot replication to another region for disaster recovery
  • Redshift supports four distribution styles; AUTO, EVEN, KEY, or ALL.
    • KEY distribution uses a single column as distribution key (DISTKEY) and helps place matching values on the same node slice
    • Even distribution distributes the rows across the slices in a round-robin fashion, regardless of the values in any particular column
    • ALL distribution replicates whole table in every compute node.
    • AUTO distribution lets Redshift assigns an optimal distribution style based on the size of the table data
  • Redshift supports Compound and Interleaved sort keys
    • Compound key
      • is made up of all of the columns listed in the sort key definition, in the order they are listed and is more efficient when query predicates use a prefix, or query’s filter applies conditions, such as filters and joins, which is a subset of the sort key columns in order.
    • Interleaved sort key
      • gives equal weight to each column in the sort key, so query predicates can use any subset of the columns that make up the sort key, in any order.
      • Not ideal for monotonically increasing attributes
  • Import/Export Data
    • UNLOAD helps copy data from Redshift table to S3
    • COPY command
      • helps copy data from S3 to Redshift
      • also supports EMR, DynamoDB, remote hosts using SSH
      • parallelized and efficient
      • can decrypt data as it is loaded from S3
      • DON’T use multiple concurrent COPY commands to load one table from multiple files as Redshift is forced to perform a serialized load, which is much slower.
      • supports data decryption when loading data, if data encrypted
      • supports decompressing data, if data is compressed.
    • Split the Load Data into Multiple Files
    • Load the data in sort key order to avoid needing to vacuum.
    • Use a Manifest File
      • provides Data consistency, to avoid S3 eventual consistency issues
      • helps specify different S3 locations in a more efficient way that with the use of S3 prefixes.
  • Zero-ETL Integrations (2024-2025)
    • enable near real-time analytics by connecting operational databases and applications to Redshift without building data pipelines
    • supports integrations from Aurora (MySQL/PostgreSQL), DynamoDB, RDS, and third-party applications (Salesforce, SAP, Zendesk)
    • works with both Redshift Serverless workgroups and provisioned clusters using RA3 instance types
    • includes SQL features: QUERY_ALL_STATES, TRUNCATECOLUMNS, and ACCEPTINVCHARS for zero-ETL data handling
    • integrates with Amazon SageMaker Lakehouse for unified analytics and AI/ML
  • Redshift Distribution Style determines how data is distributed across compute nodes and helps minimize the impact of the redistribution step by locating the data where it needs to be before the query is executed.
  • Redshift Enhanced VPC routing forces all COPY and UNLOAD traffic between the cluster and the data repositories through the VPC.
  • Workload management (WLM) enables users to flexibly manage priorities within workloads so that short, fast-running queries won’t get stuck in queues behind long-running queries.
  • Redshift Spectrum helps query and retrieve structured and semistructured data from files in S3 without having to load the data into Redshift tables.
    • Redshift Spectrum external tables are read-only. You can’t COPY or INSERT to an external table.
  • Federated Query feature allows querying and analyzing data across operational databases, data warehouses, and data lakes.
  • Short query acceleration (SQA) prioritizes selected short-running queries ahead of longer-running queries.
  • Redshift Serverless is a serverless option of Redshift that makes it more efficient to run and scale analytics in seconds without the need to set up and manage data warehouse infrastructure.

EMR

  • is a web service that utilizes a hosted Hadoop framework running on the web-scale infrastructure of EC2 and S3
  • launches all nodes for a given cluster in the same Availability Zone, which improves performance as it provides a higher data access rate.
  • seamlessly supports Reserved, On-Demand, and Spot Instances
  • consists of Master/Primary Node for management and Slave nodes, which consist of Core nodes holding data and providing compute and Task nodes for performing tasks only.
  • is fault tolerant for slave node failures and continues job execution if a slave node goes down
  • supports Persistent and Transient cluster types
    • Persistent EMR clusters continue to run after the data processing job is complete
    • Transient EMR clusters shut down when the job or the steps (series of jobs) are complete
  • supports EMRFS which allows S3 to be used as a durable HA data storage
  • EMR Serverless helps run big data frameworks such as Apache Spark and Apache Hive without configuring, managing, and scaling clusters.
    • now supports Spark Connect (2026) for interactive PySpark development from local environments, IDEs, and SageMaker Unified Studio Notebooks
    • eliminates local storage provisioning, reducing costs by up to 20%
  • Apache Spark 4.0 support (GA 2026) – includes VARIANT data type, state-management improvements, and Spark Connect, with EMR optimized runtime running workloads up to 4.5x faster than open-source Spark
  • EMR Studio is an IDE that helps data scientists and data engineers to develop, visualize, and debug data engineering and data science applications written in R, Python, Scala, and PySpark.
  • EMR Notebooks provide a managed environment, based on Jupyter Notebook, that helps prepare and visualize data, collaborate with peers, build applications, and perform interactive analysis using EMR clusters.

Glue

  • AWS Glue is a fully managed, ETL service that automates the time-consuming steps of data preparation for analytics.
  • is serverless and supports a pay-as-you-go model.
  • handles provisioning, configuration, and scaling of the resources required to run the ETL jobs on a fully managed, scale-out Apache Spark environment.
  • helps setup, orchestrate, and monitor complex data flows.
  • supports custom Scala or Python code and import custom libraries and Jar files into the AWS Glue ETL jobs to access data sources not natively supported by AWS Glue.
  • supports server side encryption for data at rest and SSL for data in motion.
  • provides development endpoints to edit, debug, and test the code it generates.
  • AWS Glue natively supports data stored in RDS, Redshift, DynamoDB, S3, MySQL, Oracle, Microsoft SQL Server, and PostgreSQL databases in the VPC running on EC2 and Data streams from MSK, Kinesis Data Streams, and Apache Kafka.
  • Glue ETL engine to Extract, Transform, and Load data that can automatically generate Scala or Python code.
  • AWS Glue 5.0/5.1 (2024-2026):
    • provides performance-optimized Apache Spark 3.5 runtime for batch and stream processing
    • native support for open table formats: Apache Iceberg, Delta Lake, and Apache Hudi
    • Spark-native fine-grained access control (FGAC) integration with AWS Lake Formation
    • faster job start times and automatic partition pruning
    • Glue 5.1 adds support for Apache Iceberg format version 3.0, deletion vectors, and row lineage tracking
    • new worker types: G.12X, G.16X general compute, and R.1X/R.2X/R.4X/R.8X memory-optimized
  • Glue Materialized Views (2025) – Apache Iceberg-based materialized views for transforming data and accelerating query performance
  • supports generative AI assistance for data integration tasks
  • Glue Data Catalog is a central repository and persistent metadata store to store structural and operational metadata for all the data assets.
  • Glue Crawlers scan various data stores to automatically infer schemas and partition structures to populate the Data Catalog with corresponding table definitions and statistics.
  • Glue Job Bookmark tracks data that has already been processed during a previous run of an ETL job by persisting state information from the job run.
  • AWS Glue Streaming ETL enables performing ETL operations on streaming data using continuously-running jobs.
  • Glue provides flexible scheduler that handles dependency resolution, job monitoring, and retries.
  • Glue Studio offers a graphical interface for authoring AWS Glue jobs to process data allowing you to define the flow of the data sources, transformations, and targets in the visual interface and generating Apache Spark code on your behalf.
  • Glue Data Quality helps reduces manual data quality effort by automatically measuring and monitoring the quality of data in data lakes and pipelines.
  • Glue DataBrew is a visual data preparation tool that makes it easy for data analysts and data scientists to prepare, visualize, clean, and normalize terabytes, and even petabytes of data directly from your data lake, data warehouses, and databases, including S3, Redshift, Aurora, and RDS.
  • ⚠️ AWS Glue for Ray will no longer accept new customers starting April 30, 2026. Existing customers can continue using the service. Explore Amazon EKS for similar capabilities.

Lake Formation

  • AWS Lake Formation helps create secure data lakes, making data available for wide-ranging analytics.
  • is an integrated data lake service that helps to discover, ingest, clean, catalog, transform, and secure data and make it available for analysis and ML.
  • automatically manages access to the registered data in S3 through services including AWS Glue, Athena, Redshift, QuickSight, and EMR using Zeppelin notebooks with Apache Spark to ensure compliance with your defined policies.
  • helps configure and manage your data lake without manually integrating multiple underlying AWS services.
  • uses a shared infrastructure with AWS Glue, including console controls, ETL code creation and job monitoring, blueprints to create workflows for data ingest, the same data catalog, and a serverless architecture.
  • can manage data ingestion through AWS Glue. Data is automatically classified, and relevant data definitions, schema, and metadata are stored in the central Glue Data Catalog. Once the data is in the S3 data lake, access policies, including table-and-column-level access controls can be defined, and encryption for data at rest enforced.
  • integrates with IAM so authenticated users and roles can be automatically mapped to data protection policies that are stored in the data catalog. The IAM integration also supports Microsoft Active Directory or LDAP to federate into IAM using SAML.
  • helps centralize data access policy controls. Users and roles can be defined to control access, down to the table and column level.
  • supports fine-grained access control (FGAC) including row-level and cell-level security, and tag-based access control (LF-Tags) for scalable permission management.
  • supports private endpoints in the VPC and records all activity in AWS CloudTrail for network isolation and auditability.
  • ⚠️ Lake Formation’s Governed Tables feature was deprecated in February 2025. Use Apache Iceberg tables with Lake Formation for transactional data lake capabilities.

Amazon Quick Suite (formerly QuickSight)

(Amazon QuickSight evolved to Amazon Quick Suite on October 9, 2025, expanding from a single BI product to a comprehensive analytics and AI platform)

  • is a cloud-powered business analytics service that integrates BI capabilities with AI agents for business insights, deep research, and automation in one unified experience.
  • delivers fast and responsive query performance by using a robust in-memory engine (SPICE).
    • “SPICE” stands for a Super-fast, Parallel, In-memory Calculation Engine
    • can also be configured to keep the data in SPICE up-to-date as the data in the underlying sources change.
    • automatically replicates data for high availability and enables Quick Suite to scale to support users to perform simultaneous fast interactive analysis across a wide variety of AWS data sources.
  • Amazon Q in QuickSight (GA April 2024) – generative BI capabilities powered by Amazon Bedrock:
    • multi-visual data Q&A for asking questions of data not in dashboards
    • executive summaries for quick trend and insight discovery
    • automated stories – documents and slides explaining data
    • natural language generation for pixel-perfect reports
    • available to all Enterprise Edition users without separate Q add-on
  • supports
    • Excel files and flat files like CSV, TSV, CLF, ELF
    • on-premises databases like PostgreSQL, SQL Server and MySQL
    • SaaS applications like Salesforce
    • and AWS data sources such as Redshift, RDS, Aurora, Athena, and S3
  • supports various functions to format and transform the data.
  • supports assorted visualizations that facilitate different analytical approaches:
    • Comparison and distribution – Bar charts (several assorted variants)
    • Changes over time – Line graphs, Area line charts
    • Correlation – Scatter plots, Heat maps
    • Aggregation – Pie graphs, Tree maps
    • Tabular – Pivot tables
  • Amazon Quick Sight (a capability within Quick Suite) now offers visual data preparation experience for advanced data transformations without code.

Data Pipeline

⚠️ AWS Data Pipeline – No Longer Available to New Customers (July 25, 2024)

AWS Data Pipeline is in maintenance mode and is no longer available to new customers. Console access was removed on April 30, 2023. Existing customers can continue to use the service via CLI and API.

Migration Options:

  • Amazon MWAA (Managed Workflows for Apache Airflow) – for complex workflow orchestration
  • AWS Step Functions – for serverless workflow orchestration
  • AWS Glue – for ETL-focused data movement pipelines
  • Amazon EventBridge – for event-driven scheduling
  • orchestration service that helps define data-driven workflows to automate and schedule regular data movement and data processing activities
  • integrates with on-premises and cloud-based storage systems
  • allows scheduling, retry, and failure logic for the workflows

Amazon OpenSearch Service

(Previously known as Amazon Elasticsearch Service, renamed September 8, 2021)

  • Amazon OpenSearch Service is a managed service that makes it easy to deploy, operate, and scale OpenSearch clusters in the AWS Cloud.
  • OpenSearch Service provides
    • real-time, distributed search and analytics engine
    • ability to provision all the resources for OpenSearch cluster and launches the cluster
    • easy to use cluster scaling options. Scaling OpenSearch Service domain by adding or modifying instances, and storage volumes is an online operation that does not require any downtime.
    • provides self-healing clusters, which automatically detects and replaces failed nodes, reducing the overhead associated with self-managed infrastructures
    • domain snapshots to back up and restore domains and replicate domains across AZs
    • enhanced security with IAM, Network, Domain access policies, and fine-grained access control
    • storage volumes for the data using EBS volumes
    • ability to span cluster nodes across multiple AZs in the same region, known as zone awareness, for high availability and redundancy. OpenSearch Service automatically distributes the primary and replica shards across instances in different AZs.
    • dedicated master nodes to improve cluster stability
    • data visualization using OpenSearch Dashboards (formerly Kibana)
    • integration with CloudWatch for monitoring domain metrics
    • integration with CloudTrail for auditing configuration API calls to domains
    • integration with S3, Kinesis, and DynamoDB for loading streaming data
    • ability to handle structured and Unstructured data
    • supports encryption at rest through KMS, node-to-node encryption over TLS, and the ability to require clients to communicate with HTTPS
  • Amazon OpenSearch Serverless
    • automatically scales without managing infrastructure
    • NextGen architecture (2026) – decoupled compute from storage, provisions in seconds, scales to zero when idle, up to 20x faster autoscaling, and up to 60% lower cost than provisioned clusters
    • two collection architectures: Classic (original) and NextGen (default for new collections)
  • Vector Database Capabilities
    • stores vector embeddings from LLMs for semantic/similarity search
    • supports hybrid search combining vector, lexical, and agentic retrieval
    • GPU-accelerated vector indexes for billion-scale databases (2025)
    • auto-optimized vector indexes for search quality/speed/cost tradeoffs
    • integrates with Amazon Bedrock for RAG and agentic AI applications
  • Zero-ETL integrations – direct data access from other AWS services without pipeline management
  • Extended Support – Standard Support ends Nov 7, 2025 for legacy Elasticsearch versions up to 6.7, ES 7.1-7.8, and OpenSearch 1.0-1.2

Athena

  • Amazon Athena is a serverless, interactive analytics service built on open-source frameworks, supporting open-table and file formats.
  • provides a simplified, flexible way to analyze petabytes of data in an S3 data lake and 30 data sources, including on-premises data sources or other cloud systems using SQL or Python without loading the data.
  • is built on open-source Trino and Presto engines and Apache Spark frameworks, with no provisioning or configuration effort required.
  • is highly available and runs queries using compute resources across multiple facilities, automatically routing queries appropriately if a particular facility is unreachable
  • can process unstructured, semi-structured, and structured datasets.
  • integrates with QuickSight for visualizing the data or creating dashboards.
  • supports various standard data formats, including CSV, TSV, JSON, ORC, Avro, and Parquet.
  • supports compressed data in Snappy, Zlib, LZO, and GZIP formats. You can improve performance and reduce costs by compressing, partitioning, and using columnar formats.
  • can handle complex analysis, including large joins, window functions, and arrays
  • uses a managed Glue Data Catalog to store information and schemas about the databases and tables that you create for the data stored in S3
  • uses schema-on-read technology, which means that the table definitions are applied to the data in S3 when queries are being applied. There’s no data loading or transformation required. Table definitions and schema can be deleted without impacting the underlying data stored in S3.
  • supports fine-grained access control with AWS Lake Formation which allows for centrally managing permissions and access control for data catalog resources in the S3 data lake.
  • integrates with Amazon SageMaker Lakehouse for governed federated queries across data sources

Amazon SageMaker Lakehouse

(Launched at re:Invent 2024, GA March 2025)

  • part of the next generation of Amazon SageMaker – a unified platform for data, analytics, and AI
  • unifies data across S3 data lakes (including S3 Tables), Redshift data warehouses, and operational databases
  • supports zero-ETL integrations from Aurora, DynamoDB, RDS, and third-party applications (Salesforce, SAP, Zendesk) for near real-time data access
  • enables querying, analyzing, and joining data using Redshift, Athena, EMR, and AWS Glue
  • provides unified access through Amazon SageMaker Unified Studio – a single development experience for data engineers, data scientists, and analysts
  • supports Apache Iceberg open table format for interoperability
  • integrates with Lake Formation for fine-grained access control and governance