AWS Storage Options – SQS & Redshift

SQS

• is a fully managed message queuing service that provides a reliable, highly scalable, hosted queue for temporary storage and delivery of messages up to 1 MiB in size (increased from 256 KB in August 2025).
• supports a virtually unlimited number of queues and supports two queue types:
  • Standard queues – unordered, at-least-once delivery with nearly unlimited throughput.
  • FIFO queues – exactly-once processing with strict message ordering, supporting up to 70,000 messages per second with high throughput mode.

Ideal Usage Patterns

• is ideally suited to any scenario where multiple application components must communicate and coordinate their work in a loosely coupled manner particularly producer consumer scenarios.
• can be used to coordinate a multi-step processing pipeline, where each message is associated with a task that must be processed.
• enables the number of worker instances to scale up or down, and also enable the processing power of each single worker instance to scale up or down, to suit the total workload, without any application changes.
• ideal for multi-tenant workloads using fair queues (launched July 2025) to mitigate noisy neighbor impact and ensure consistent processing across tenants.
• supports event-driven architectures with AWS Lambda event source mapping, including provisioned mode for 3x faster scaling and 16x higher concurrency.

Anti-Patterns

• Binary or Large Messages
  • SQS supports messages up to 1 MiB. If the application requires binary or messages exceeding this limit, it is best to use the Amazon SQS Extended Client Library with Amazon S3 to store the payload and SQS to store the pointer.
• Long Term storage
  • SQS stores messages for max 14 days and if application requires storage period longer than 14 days, Amazon S3 or other storage options should be preferred.
• High-speed message queuing or very short tasks
  • If the application requires a very high-speed message send and receive response from a single producer or consumer, use of Amazon DynamoDB or a message-queuing system hosted on Amazon EC2 may be more appropriate.

Performance

• is a distributed queuing system that is optimized for horizontal scalability, not for single-threaded sending or receiving speeds.
• Standard queues support nearly unlimited throughput (thousands of transactions per second per API action).
• FIFO queues support up to 3,000 messages per second with batching by default, or up to 70,000 messages per second (700,000 with batching) in high throughput mode in select regions.
• FIFO queues support up to 120,000 in-flight messages (increased from 20,000 in November 2024).
• Higher receive performance can be achieved by requesting multiple messages (up to 10) in a single call.
• Fair queues (July 2025) automatically reorder messages to maintain consistent dwell time across tenants, preventing noisy neighbors from impacting processing latency.

Durability & Availability

• are highly durable but temporary.
• stores all messages redundantly across multiple servers and data centers.
• Message retention time is configurable on a per-queue basis, from a minimum of one minute to a maximum of 14 days.
• Messages are retained in a queue until they are explicitly deleted, or until they are automatically deleted upon expiration of the retention time.
• supports dead-letter queues (DLQ) for isolating messages that fail processing, with DLQ redrive capability to move messages back to the source queue or a custom destination for reprocessing.

Cost Model

• pricing is based on
  • number of requests (per million requests)
  • the amount of data transferred out (priced per GB per month)
  • First 1 million requests per month are free (Free Tier)

Scalability & Elasticity

• is both highly elastic and massively scalable.
• is designed to enable a virtually unlimited number of computers to read and write a virtually unlimited number of messages at any time.
• supports virtually unlimited numbers of queues and messages per queue for any user.
• supports dual-stack (IPv4 and IPv6) endpoints for flexible network access.

Key Features (Recent Updates)

• Message payload size increased to 1 MiB (August 2025) – supports larger messages for both standard and FIFO queues without needing the Extended Client Library.
• Fair queues (July 2025) – automatically mitigates noisy neighbor impact in multi-tenant standard queues by reordering messages to maintain consistent dwell time across tenants.
• FIFO high throughput mode – up to 70,000 TPS per API action (November 2023), enabling 700,000 messages per second with batching.
• FIFO in-flight limit increase (November 2024) – increased from 20,000 to 120,000 in-flight messages per FIFO queue.
• Lambda provisioned mode for SQS (January 2025) – dedicated polling resources providing 3x faster scaling and 16x higher concurrency for event source mapping.
• Dead-letter queue redrive – move failed messages from DLQ back to source queue or a custom destination for both standard and FIFO queues.
• Simplified KMS permissions – SendMessage no longer requires kms:Decrypt permission; only kms:GenerateDataKey is needed.
• Temporary queues – application-managed virtual queues for request-response patterns that reduce cost and development time.

Amazon Redshift

• is a fast, fully-managed, petabyte-scale data warehouse service that makes it simple and cost-effective to efficiently analyze all data using existing business intelligence tools.
• is optimized for datasets that range from a few hundred gigabytes to a petabyte or more.
• manages the work needed to set up, operate, and scale a data warehouse, from provisioning the infrastructure capacity to automating ongoing administrative tasks such as backups and patching.
• offers two deployment models: Provisioned clusters (RA3 and new RG instances) and Redshift Serverless (pay-per-use with automatic scaling).

Ideal Usage Pattern

• is ideal for analyzing large datasets using existing business intelligence tools.
• Common use cases include
  • Analyze global sales data for multiple products
  • Store historical stock trade data
  • Analyze ad impressions and clicks
  • Aggregate gaming data
  • Analyze social trends
  • Measure clinical quality, operation efficiency, and financial performance in the health care space
  • Near real-time analytics using zero-ETL integrations from Aurora, DynamoDB, RDS, and SaaS applications
  • Data lakehouse analytics querying data in S3 data lakes using Redshift Spectrum
  • Generative AI applications using Amazon Bedrock integration for sentiment analysis, text generation, and summarization directly on warehouse data

Anti-Pattern

• OLTP workloads
  • Redshift is a column-oriented database and more suited for data warehousing and analytics. If application involves online transaction processing, Amazon RDS or Aurora would be a better choice.
• Blob data
  • For Blob storage, Amazon S3 would be a better choice with metadata in other storage as RDS or DynamoDB.

Performance

• Amazon Redshift allows very high query performance on datasets ranging in size from hundreds of gigabytes to a petabyte or more.
• It uses columnar storage, data compression, and zone maps to reduce the amount of I/O needed to perform queries.
• It has a massively parallel processing (MPP) architecture that parallelizes and distributes SQL operations to take advantage of all available resources.
• Underlying hardware is designed for high performance data processing that uses local attached storage to maximize throughput.
• New RG instances (GA May 2026) powered by AWS Graviton deliver up to 2.4x faster performance than RA3 at 30% lower price per vCPU.
• AI-driven scaling and optimization in Redshift Serverless automatically provisions and scales capacity for demanding workloads.
• Query performance improvements (March 2026) speed up new queries in BI dashboards and ETL workloads by up to 7x.
• Concurrency scaling automatically adds additional cluster capacity to handle burst read and write workloads, with support for data ingestion (COPY queries in Parquet/ORC from S3).

Durability & Availability

• Amazon Redshift stores three copies of your data—all data written to a node in your cluster is automatically replicated to other nodes within the cluster, and all data is continuously backed up to Amazon S3.
• Snapshots are automated, incremental, and continuous and stored for a user-defined period (1-35 days).
• Manual snapshots can be created and are retained until explicitly deleted.
• Amazon Redshift continuously monitors the health of the cluster and automatically re-replicates data from failed drives and replaces nodes as necessary.
• Multi-AZ deployments (GA for RA3 clusters) run your data warehouse across two Availability Zones simultaneously, providing continued operation during AZ failure scenarios.

Cost Model

• Provisioned clusters pricing:
  • Compute node hours – total hours run across all compute nodes (RA3 or RG instances)
  • Redshift Managed Storage (RMS) – billed per GB/month, separate from compute (RA3/RG only)
  • Backup storage – for automated and manual snapshots beyond the free tier
  • Data transfer – standard AWS data transfer charges apply
  • Concurrency scaling – free for 1 hour per day per cluster, then per-second billing
  • Spectrum – per TB of data scanned in S3
• Redshift Serverless pricing:
  • Compute – per RPU-hour (Redshift Processing Unit), billed per second with no charge when idle
  • Storage – per GB/month for managed storage
• Reserved Instance pricing available for provisioned clusters (1-year or 3-year terms) for significant discounts.

Scalability & Elasticity

• Provisioned clusters – Elastic resize allows adding or removing nodes within minutes. Classic resize available for node type changes.
• Redshift Serverless – automatically scales compute capacity up and down based on workload demands with no cluster management required.
• Data sharing allows securely sharing live, transactionally consistent data across Redshift clusters (cross-account, cross-region) without copying data.
• Multi-warehouse writes through data sharing (GA November 2024) enable using different warehouses of different types and sizes for ETL workloads.

Key Features (Recent Updates)

• RG Instances (GA May 2026) – New Graviton-powered instance family delivering 2.4x faster performance than RA3 at 30% lower price per vCPU.
• DC2 End of Life (April 24, 2026) – Migrate to RA3, RG, or Serverless. New DC2 cluster creation blocked since May 15, 2025.
• Redshift Serverless – Pay-per-use model with automatic scaling, AI-driven optimization, and per-second billing with no charge when idle.
• Zero-ETL integrations – Near real-time data replication from Aurora, DynamoDB, RDS, and self-managed databases to Redshift without building ETL pipelines. Also supports SaaS sources (Salesforce, SAP, Zendesk).
• Multi-AZ deployments – Run RA3 provisioned clusters across two Availability Zones for high availability.
• Amazon Bedrock integration (October 2024) – Run generative AI tasks (text generation, sentiment analysis, summarization, classification) directly on Redshift data using foundation models via SQL.
• Amazon Q generative SQL – Generate SQL from natural language prompts in the Redshift Query Editor.
• Data sharing – Share live data across clusters, accounts, and regions without data movement. Supports multi-warehouse writes for ETL.
• Redshift Spectrum – Query exabytes of data in S3 without loading it into Redshift, enabling data lakehouse architectures.
• Concurrency scaling for ingestion (2026) – Automatically scales for COPY queries in Parquet/ORC formats from S3 during traffic spikes.
• 7x query performance improvement (March 2026) – Faster response for BI dashboards, ETL pipelines, and near real-time analytics.