SQS
- extremely scalable queue service and potentially handles millions of messages
- helps build fault tolerant, distributed loosely coupled applications
- stores copies of the messages on multiple servers for redundancy and high availability
- guarantees At-Least-Once Delivery, but does not guarantee Exact One Time Delivery which might result in duplicate messages (Not true anymore with the introduction of FIFO queues)
- does not maintain or guarantee message order, and if needed sequencing information needs to be added to the message itself (Not true anymore with the introduction of FIFO queues)
- supports multiple readers and writers interacting with the same queue as the same time
- holds message for 4 days, by default, and can be changed from 1 min – 14 days after which the message is deleted
- message needs to be explicitly deleted by the consumer once processed
- allows send, receive and delete batching which helps club up to 10 messages in a single batch while charging price for a single message
- handles visibility of the message to multiple consumers using Visibility Timeout, where the message once read by a consumer is not visible to the other consumers till the timeout occurs
- can handle load and performance requirements by scaling the worker instances as the demand changes (Job Observer pattern)
- message sample allowing short and long polling
- returns immediately vs waits for fixed time for e.g. 20 secs
- might not return all messages as it samples a subset of servers vs returns all available messages
- repetitive vs helps save cost with long connection
- supports delay queues to make messages available after a certain delay, can you used to differentiate from priority queues
- supports dead letter queues, to redirect messages which failed to process after certain attempts instead of being processed repeatedly
- Design Patterns
- Job Observer Pattern can help coordinate number of EC2 instances with number of job requests (Queue Size) automatically thus Improving cost effectiveness and performance
- Priority Queue Pattern can be used to setup different queues with different handling either by delayed queues or low scaling capacity for handling messages in lower priority queues
SNS
- delivery or sending of messages to subscribing endpoints or clients
- publisher-subscriber model
- Producers and Consumers communicate asynchronously with subscribers by producing and sending a message to a topic
- supports Email (plain or JSON), HTTP/HTTPS, SMS, SQS
- supports Mobile Push Notifications to push notifications directly to mobile devices with services like Amazon Device Messaging (ADM), Apple Push Notification Service (APNS), Google Cloud Messaging (GCM) etc. supported
- order is not guaranteed and No recall available
- integrated with Lambda to invoke functions on notifications
- for Email notifications, use SNS or SES directly, SQS does not work
SWF
- orchestration service to coordinate work across distributed components
- helps define tasks, stores, assigns tasks to workers, define logic, tracks and monitors the task and maintains workflow state in a durable fashion
- helps define tasks which can be executed on AWS cloud or on-premises
- helps coordinating tasks across the application which involves managing intertask dependencies, scheduling, and concurrency in accordance with the logical flow of the application
- supports built-in retries, timeouts and logging
- supports manual tasks
- Characteristics
- deliver exactly once
- uses long polling, which reduces number of polls without results
- Visibility of task state via API
- Timers, signals, markers, child workflows
- supports versioning
- keeps workflow history for a user-specified time
- AWS SWF vs AWS SQS
- task-oriented vs message-oriented
- track of all tasks and events vs needs custom handling
SES
- highly scalable and cost-effective email service
- uses content filtering technologies to scan outgoing emails to check standards and email content for spam and malware
- supports full fledged emails to be sent as compared to SNS where only the message is sent in Email
- ideal for sending bulk emails at scale
- guarantees first hop
- eliminates the need to support custom software or applications to do heavy lifting of email transport
Please include API Gateway
Thanks, will check on the same.
Worth mentioning about ReceiveMessageWaitTimeSeconds – w.r.t long polling – saving CPU cycles for a EC2 consumer.
Thanks Bhuvan, will add the same.
Jayendra it would be great if you include EFS as a section.
holds message for 4 days, by default, and can be changed from 1 min – 14 days after which the message is deleted
message needs to be explicitly deleted by the consumer once processed
Hi Jayendra,
Can you clarify on the above? Are the messages deleted automatically after 14 days or needs to be explicitly deleted by the consumer after processing the job. Thanks.
The messages expire i.e. are automatically deleted after 14 days.
For SQS you say…
supports multiple readers and writers interacting with the same queue as the same time
Does this not conflict with your SQS comparison with Kinesis Streams on that page? There it says 1 queue for 1 consumer for SQS vs many applications can read from Kinesis Stream.
Appreciate if you can clarify.
Hi Grif, the statement is “Kinesis supports multiple consumers while SQS allows the messages to be delivered to only one consumer at a time”.
SQS is a queue where the message can be consumed by a single consumer at a time, after which it should be deleted.
Kinesis is more like a messaging system for e.g. Kafka, which holds the message and multiple consumers can read the same message.