Amazon SQS is a highly available distributed queue system
offers a reliable, highly-scalable, hosted queue for storing messages in transit between computers
provides fault tolerant, loosely coupled, flexibility of distributed components of applications to send & receive without requiring each component to be concurrently available
helps build distributed application with decoupled components
requires no administrative overhead and little configuration
supports the HTTP over SSL (HTTPS) and Transport Layer Security (TLS) protocols for security
SQS does not preserve ordering of the messages (Update, with latest introduction of FIFO queues it does)
SQS provide at-least-once delivery of the messages (Update, with latest FIFO queue it provides exactly-once processing)
A queue is a temporary repository for messages awaiting for processing and acts as a buffer between the component producer and the consumer
SQS Features & Key Points
offers reliable and scalable hosted queues for storing messages
is engineered to always be available and deliver messages
provides ability to store messages in a fail safe queue
highly concurrent access to messages
ensures delivery of each message at least once
stores copies of the messages on multiple servers for redundancy and high availability.
might deliver duplicate copy of messages, if the servers storing a copy of a message is unavailable when you receive or delete the message and the copy of the message will not be deleted on that unavailable server
Applications should be designed to be idempotent with the ability to handle duplicate messages and not be adversely affected if it processes the same message more than once
SQS message can contain up to 10 metadata attributes.
take the form of name-type-value triples
can be used to separate the body of a message from the metadata that describes it.
helps process and store information with greater speed and efficiency because the applications don’t have to inspect an entire message before understanding how to process it
behavior of retrieving messages from the queue depends on whether short (standard) polling, the default behavior, or long polling is used
With short polling,
SQS samples only a subset of the servers (based on a weighted random distribution) and returns messages from just those servers.
A receive request might not return all the messages in the queue. But a subsequent receive request would return the message
With Long polling,
request persists for the time specified and returns as soon as the message is available thereby reducing costs and time the message has to dwell in the queue
SQS allows send, receive and delete batching which helps club up to 10 messages in a single batch while charging price for a single message
helps lower cost and also increases the throughput
Configurable settings per queue
All queues don’t have to be alike
makes a best effort to preserve order in messages does not guarantee first in, first out delivery of messages
UPDATE @ link – SQS now offers FIFO queues which maintain order and Exactly-Once Processing
can be handled by placing sequencing information within the message and performing the ordering on the client side
removes tight coupling between components
provides the ability to move data between distributed components of the applications that perform different tasks without losing messages or requiring each component to be always available
Multiple writers and readers
supports multiple readers and writers interacting with the same queue as the same time
locks the message during processing, using Visibility Timeout, preventing it to be processed by any other consumer
Variable message size
supports message in any format up to 256KB of text.
messages larger than 256 KB can be managed using the SQS or DynamoDB with SQS storing pointer
Access can be controlled for who can produce and consume messages to each queue
delay queue allows the user to set a default delay on a queue such that delivery of all messages enqueued is postponed for that time duration
Dead Letter Queues
Dead letter queue is a queue for messages that were not able to be processed after a maximum number of attempts
supports the processing, storage, and transmission of credit card data by a merchant or service provider, and has been validated as being PCI-DSS (Payment Card Industry – Data Security Standard) compliant
SQS Use Cases
Decouple components of a distributed application that may not all process the same amount of work simultaneously.
Buffer and Batch Operations
Add scalability and reliability to the architecture and smooth out temporary volume spikes without losing messages or increasing latency
Move slow operations off of interactive request paths by enqueueing the request.
Combine Amazon SQS with Amazon SNS to send identical copies of a message to multiple queues in parallel for simultaneous processing.
SQS queues can be used to determine the load on an application, and combined with Auto Scaling, the EC2 instances can be scaled in or out, depending on the volume of traffic
How SQS Queues Works
SQS allows queues to be created, deleted and messages can be sent and received from it
SQS queue retains messages for four days, by default.
Queues can configured to retain messages for 1 minute to 14 days after the message has been sent.
SQS can delete a queue without notification if one of the following actions hasn’t been performed on it for 30 consecutive days.
SQS allows the deletion of the queue with messages in it
Queue and Message Identifiers
Queue is identified by a unique queue name within the same AWS account
SQS assigns each queue with a Queue URL identifier for e.g. http://sqs.us-east-1.amazonaws.com/123456789012/queue2
Queue URL is needed to perform any operation on the Queue
Message IDs are useful for identifying messages,
Each message receives a system-assigned message ID that SQS returns to with the SendMessage response.
To delete a message, the message’s receipt handle instead of the message ID is needed
Message ID can be of is 100 characters max
When a message is received from a queue, a receipt handle is returned with the message which is associated with the act of receiving the message rather then the message itself
Receipt handle is required, not the message id, to delete a message or to change the message visibility
If a message is received more than once, each time its received, a different receipt handle is assigned and the latest should be used always
SQS does not delete the message once it is received by a consumer,
because the system is distributed, there’s no guarantee that the consumer will actually receive the message (it’s possible the connection could break or the component could fail before receiving the message)
Consumer should explicitly delete the message from the Queue once it is received and successfully processed
As the message is still available on the Queue, other consumers would be able to receive and process and this needs to be prevented
SQS handles the above behavior using Visibility timeout.
SQS blocks the visibility of the message for the Visibility timeout period, which is the time during which SQS prevents other consuming components from receiving and processing that message
Consumer should delete the message within the Visibility timeout. If the consumer fails to delete the message before the visibility timeout expires, the message is visible again for other consumers.
Visibility timeout considerations
clock starts ticking once SQS returns the message
should be large enough to take into account the processing time for each of the message
default Visibility timeout for each Queue is 30 seconds and can be changed at the Queue level
when receiving messages, a special visibility timeout for the returned messages can be set without changing the overall queue timeout using the receipt handle
can be extended by the consumer, if the consumer thinks it won’t be able to process the message within the current visibility timeout period. SQS restarts the timeout period using the new value
a message’s Visibility timeout extension applies only to that particular receipt of the message and does not affect the timeout for the queue or later receipts of the message
SQS has an 120,000 limit for the number of inflight messages per queue i.e. message received but not yet deleted and any further messages would receive an error after reaching the limit
Component 1 sends Message A to a queue, and the message is redundantly distributed across the SQS servers.
When Component 2 is ready to process a message, it retrieves messages from the queue, and Message A is returned. While Message A is being processed, it remains in the queue but is not returned to subsequent receive requests for the duration of the visibility timeout.
Component 2 deletes Message A from the queue to avoid the message being received and processed again once the visibility timeout expires.
SQS Design Patterns
Priority Queue Pattern
Use SQS to prepare multiple queues for the individual priority levels.
Place those processes to be executed immediately (job requests) in the high priority queue.
Prepare numbers of batch servers, for processing the job requests of the queues, depending on the priority levels.
Queues have a message “Delayed Send” function, which can be used to delay the time for starting a process.
SQS Job Observer Pattern
Enqueue job requests as SQS messages.
Have the batch server dequeue and process messages from SQS.
Set up Auto Scaling to automatically increase or decrease the number of batch servers, using the number of SQS messages, with CloudWatch, as the trigger to do so.
AWS Certification Exam Practice Questions
Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
Open to further feedback, discussion and correction.
Which AWS service can help design architecture to persist in-flight transactions?
Elastic IP Address
A company has a workflow that sends video files from their on-premise system to AWS for transcoding. They use EC2 worker instances that pull transcoding jobs from SQS. Why is SQS an appropriate service for this scenario?
SQS guarantees the order of the messages.
SQS synchronously provides transcoding output.
SQS checks the health of the worker instances.
SQS helps to facilitate horizontal scaling of encoding tasks
Which statement best describes an Amazon SQS use case?
Automate the process of sending an email notification to administrators when the CPU utilization reaches 70% on production servers (Amazon EC2 instances) (CloudWatch + SNS)
Create a video transcoding website where multiple components need to communicate with each other, but can’t all process the same amount of work simultaneously (SQS provides loose coupling)
Coordinate work across distributed web services to process employee’s expense reports (SWF – Steps in order and might need manual steps)
Distribute static web content to end users with low latency across multiple countries (CloudFront + S3)
Your application provides data transformation services. Files containing data to be transformed are first uploaded to Amazon S3 and then transformed by a fleet of spot EC2 instances. Files submitted by your premium customers must be transformed with the highest priority. How should you implement such a system?
Use a DynamoDB table with an attribute defining the priority level. Transformation instances will scan the table for tasks, sorting the results by priority level.
Use Route 53 latency based-routing to send high priority tasks to the closest transformation instances.
Use two SQS queues, one for high priority messages, and the other for default priority. Transformation instances first poll the high priority queue; if there is no message, they poll the default priority queue
Use a single SQS queue. Each message contains the priority level. Transformation instances poll high-priority messages first.
Your company plans to host a large donation website on Amazon Web Services (AWS). You anticipate a large and undetermined amount of traffic that will create many database writes. To be certain that you do not drop any writes to a database hosted on AWS. Which service should you use?
Amazon RDS with provisioned IOPS up to the anticipated peak write throughput.
Amazon Simple Queue Service (SQS) for capturing the writes and draining the queue to write to the database
Amazon ElastiCache to store the writes until the writes are committed to the database.
Amazon DynamoDB with provisioned write throughput up to the anticipated peak write throughput.
A customer has a 10 GB AWS Direct Connect connection to an AWS region where they have a web application hosted on Amazon Elastic Computer Cloud (EC2). The application has dependencies on an on-premises mainframe database that uses a BASE (Basic Available. Sort stale Eventual consistency) rather than an ACID (Atomicity. Consistency isolation. Durability) consistency model. The application is exhibiting undesirable behavior because the database is not able to handle the volume of writes. How can you reduce the load on your on-premises database resources in the most cost-effective way?
Use an Amazon Elastic Map Reduce (EMR) S3DistCp as a synchronization mechanism between the onpremises database and a Hadoop cluster on AWS.
Modify the application to write to an Amazon SQS queue and develop a worker process to flush the queue to the on-premises database
Modify the application to use DynamoDB to feed an EMR cluster which uses a map function to write to the on-premises database.
Provision an RDS read-replica database on AWS to handle the writes and synchronize the two databases using Data Pipeline.
An organization has created a Queue named “modularqueue” with SQS. The organization is not performing any operations such as SendMessage, ReceiveMessage, DeleteMessage, GetQueueAttributes, SetQueueAttributes, AddPermission, and RemovePermission on the queue. What can happen in this scenario?
AWS SQS sends notification after 15 days for inactivity on queue
AWS SQS can delete queue after 30 days without notification
AWS SQS marks queue inactive after 30 days
AWS SQS notifies the user after 2 weeks and deletes the queue after 3 weeks.
A user is using the AWS SQS to decouple the services. Which of the below mentioned operations is not supported by SQS?
A user has created a queue named “awsmodule” with SQS. One of the consumers of queue is down for 3 days and then becomes available. Will that component receive message from queue?
Yes, since SQS by default stores message for 4 days
No, since SQS by default stores message for 1 day only
No, since SQS sends message to consumers who are available that time
Yes, since SQS will not delete message until it is delivered to all consumers
A user has created a queue named “queue2” in US-East region with AWS SQS. The user’s AWS account ID is 123456789012. If the user wants to perform some action on this queue, which of the below Queue URL should he use?
A user has created a queue named “myqueue” with SQS. There are four messages published to queue, which are not received by the consumer yet. If the user tries to delete the queue, what will happen?
A user can never delete a queue manually. AWS deletes it after 30 days of inactivity on queue
It will delete the queue
It will initiate the delete but wait for four days before deleting until all messages are deleted automatically.
I t will ask user to delete the messages first
A user has developed an application, which is required to send the data to a NoSQL database. The user wants to decouple the data sending such that the application keeps processing and sending data but does not wait for an acknowledgement of DB. Which of the below mentioned applications helps in this scenario?
AWS Simple Notification Service
AWS Simple Workflow
AWS Simple Queue Service
AWS Simple Query Service
You are building an online store on AWS that uses SQS to process your customer orders. Your backend system needs those messages in the same sequence the customer orders have been put in. How can you achieve that?
It is not possible to do this with SQS
You can use sequencing information on each message
You can do this with SQS but you also need to use SWF
Messages will arrive in the same order by default
A user has created a photo editing software and hosted it on EC2. The software accepts requests from the user about the photo format and resolution and sends a message to S3 to enhance the picture accordingly. Which of the below mentioned AWS services will help make a scalable software with the AWS infrastructure in this scenario?