Amazon SQS Features

• Visibility timeout defines the period where SQS blocks the visibility of the message and prevents other consuming components from receiving and processing that message.
• Dead-letter queues – DLQ helps source queues (Standard and FIFO) target messages that can’t be processed (consumed) successfully.
• DLQ Redrive policy specifies the source queue, the dead-letter queue, and the conditions under which messages are moved from the former to the latter if the consumer of the source queue fails to process a message a specified number of times.
• Short and Long polling control how the queues would be polled and Long polling help reduce empty responses.

Queue and Message Identifiers

Queue URLs

• Queue is identified by a unique queue name within the same AWS account
• Each queue is assigned with a Queue URL identifier for e.g. http://sqs.us-east-1.amazonaws.com/123456789012/queue2
• Queue URL is needed to perform any operation on the Queue.

Message ID

• Message IDs are useful for identifying messages
• Each message receives a system-assigned message ID that is returned with the SendMessage response.
• To delete a message, the message’s receipt handle instead of the message ID is needed
• Message ID can be of is 100 characters max

Receipt Handle

• When a message is received from a queue, a receipt handle is returned with the message which is associated with the act of receiving the message rather than the message itself.
• Receipt handle is required, not the message id, to delete a message or to change the message visibility.
• If a message is received more than once, each time it is received, a different receipt handle is assigned and the latest should be used always.

Message Deduplication ID

• Message Deduplication ID is used for the deduplication of sent messages.
• Message Deduplication ID is applicable for FIFO queues.
• If a message with a particular message deduplication ID is sent successfully, any messages sent with the same message deduplication ID are accepted successfully but aren’t delivered during the 5-minute deduplication interval.

Message Group ID

• Message Group ID specifies that a message belongs to a specific message group.
• Message Group ID is applicable for FIFO queues.
• Messages that belong to the same message group are always processed one by one, in a strict order relative to the message group.
• However, messages that belong to different message groups might be processed out of order.

Visibility timeout

• SQS does not delete the message once it is received by a consumer, because the system is distributed, there’s no guarantee that the consumer will actually receive the message (it’s possible the connection could break or the component could fail before receiving the message)
• The consumer should explicitly delete the message from the Queue once it is received and successfully processed.
• As the message is still available in the Queue, other consumers would be able to receive and process and this needs to be prevented.
• SQS handles the above behavior using Visibility timeout.
• SQS blocks the visibility of the message for the Visibility timeout period, which is the time during which SQS prevents other consuming components from receiving and processing that message.
• Consumer should delete the message within the Visibility timeout. If the consumer fails to delete the message before the visibility timeout expires, the message is visible again to other consumers.
• Once Visible the message is available for other consumers to consume and can lead to duplicate messages.
• Visibility timeout considerations
  • Clock starts ticking once SQS returns the message
  • should be large enough to take into account the processing time for each message
  • default Visibility timeout for each Queue is 30 seconds and can be changed at the Queue level
  • when receiving messages, a special visibility timeout for the returned messages can be set without changing the overall queue timeout using the receipt handle
  • can be extended by the consumer, using ChangeMessageVisibility , if the consumer thinks it won’t be able to process the message within the current visibility timeout period. SQS restarts the timeout period using the new value.
  • a message’s Visibility timeout extension applies only to that particular receipt of the message and does not affect the timeout for the queue or later receipts of the message
• SQS has a 120,000 limit for the number of inflight messages per queue i.e. messages received but not yet deleted and any further messages would receive an error after reaching the limit

Message Lifecycle

1. Component 1 sends Message A to a queue, and the message is redundantly distributed across the SQS servers.
2. When Component 2 is ready to process a message, it retrieves messages from the queue, and Message A is returned. While Message A is being processed, it remains in the queue but is not returned to subsequent receive requests for the duration of the visibility timeout.
3. Component 2 deletes Message A from the queue to avoid the message being received and processed again once the visibility timeout expires.

SQS Dead Letter Queues – DLQ

• SQS supports dead-letter queues (DLQ), which other queues (source queues – Standard and FIFO) can target for messages that can’t be processed (consumed) successfully.
• Dead-letter queues are useful for debugging the application or messaging system because DLQ help isolates unconsumed messages to determine why their processing doesn’t succeed.
• DLQ redrive policy
  • specifies the source queue, the dead-letter queue, and the conditions under which SQS moves messages from the former to the latter if the consumer of the source queue fails to process a message a specified number of times.
  • specifies which source queues can access the dead-letter queue.
  • also helps move the messages back to the source queue.
• SQS does not create the dead-letter queue automatically. DLQ must first be created before being used.
• DLQ for the source queue should be of the same type i.e. Dead-letter queue of a FIFO queue must also be a FIFO queue. Similarly, the dead-letter queue of a standard queue must also be a standard queue.
• DLQ should be in the same account and region as the source queue.

SQS Delay Queues

• Delay queues help postpone the delivery of new messages to consumers for a number of seconds
• Messages sent to the delay queue remain invisible to consumers for the duration of the delay period.
• Minimum delay is 0 seconds (default) and the Maximum is 15 minutes.
• Delay queues are similar to visibility timeouts as both features make messages unavailable to consumers for a specific period of time.
• The difference between the two is that, for delay queues, a message is hidden when it is first added to the queue, whereas for visibility timeouts a message is hidden only after it is consumed from the queue.

Short and Long polling

SQS provides short polling and long polling to receive messages from a queue.

Short Polling

• ReceiveMessage request queries only a subset of the servers (based on a weighted random distribution) to find messages that are available to include in the response.
• SQS sends the response right away, even if the query found no messages.
• By default, queues use short polling.

Long Polling

• ReceiveMessage request queries all of the servers for messages.
• SQS sends a response after it collects at least one available message, up to the maximum number of messages specified in the request.
• SQS sends an empty response only if the polling wait time expires.
• Wait time greater than 0 triggers long polling with a max of 20 secs.
• Long polling helps
  • reduce the cost of using SQS by eliminating the number of empty responses (when there are no messages available for a ReceiveMessage request)
  • reduce false empty responses (when messages are available but aren’t included in a response).
  • Return messages as soon as they become available.

AWS Certification Exam Practice Questions

• Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
• AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
• AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
• Open to further feedback, discussion and correction.

1. How does Amazon SQS allow multiple readers to access the same message queue without losing messages or processing them many times?
   1. By identifying a user by his unique id
   2. By using unique cryptography
   3. Amazon SQS queue has a configurable visibility timeout
   4. Multiple readers can’t access the same message queue
2. If a message is retrieved from a queue in Amazon SQS, how long is the message inaccessible to other users by default?
   1. 0 seconds
   2. 1 hour
   3. 1 day
   4. forever
   5. 30 seconds
3. When a Simple Queue Service message triggers a task that takes 5 minutes to complete, which process below will result in successful processing of the message and remove it from the queue while minimizing the chances of duplicate processing?
   1. Retrieve the message with an increased visibility timeout, process the message, delete the message from the queue
   2. Retrieve the message with an increased visibility timeout, delete the message from the queue, process the message
   3. Retrieve the message with increased DelaySeconds, process the message, delete the message from the queue
   4. Retrieve the message with increased DelaySeconds, delete the message from the queue, process the message
4. You need to process long-running jobs once and only once. How might you do this?
   1. Use an SNS queue and set the visibility timeout to long enough for jobs to process.
   2. Use an SQS queue and set the reprocessing timeout to long enough for jobs to process.
   3. Use an SQS queue and set the visibility timeout to long enough for jobs to process.
   4. Use an SNS queue and set the reprocessing timeout to long enough for jobs to process.
5. You are getting a lot of empty receive requests when using Amazon SQS. This is making a lot of unnecessary network load on your instances. What can you do to reduce this load?
   1. Subscribe your queue to an SNS topic instead.
   2. Use as long of a poll as possible, instead of short polls. 
   3. Alter your visibility timeout to be shorter.
   4. Use sqsd on your EC2 instances.
6. Company B provides an online image recognition service and utilizes SQS to decouple system components for scalability. The SQS consumers poll the imaging queue as often as possible to keep end-to-end throughput as high as possible. However, Company B is realizing that polling in tight loops is burning CPU cycles and increasing costs with empty responses. How can Company B reduce the number of empty responses?
   1. Set the imaging queue visibility Timeout attribute to 20 seconds
   2. Set the Imaging queue ReceiveMessageWaitTimeSeconds attribute to 20 seconds (Long polling. Refer link)
   3. Set the imaging queue MessageRetentionPeriod attribute to 20 seconds
   4. Set the DelaySeconds parameter of a message to 20 seconds