AWS Storage Options – Whitepaper – Certification

May 29, 2016 ~ Last updated on : October 15, 2018 ~ jayendrapatil ~ 58 Comments

Storage Options Whitepaper

AWS Storage Options is one of the most important Whitepaper for AWS Solution Architect Professional Certification exam and covers a brief summary of each AWS storage options, their ideal usage patterns, anti-patterns, performance, durability and availability, scalability etc.

Overview

AWS offers multiple cloud-based storage options. Each has a unique combination of performance, durability, availability, cost, and interface, as well as other characteristics such as scalability and elasticity
All storage options are ideally suited for some uses cases and there are certain Anti-Patterns which should be taken in account while making a storage choice

AWS Various Storage Options

AWS Certification Exam Practice Questions

Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).

AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.

AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated

Open to further feedback, discussion and correction.

You are developing a highly available web application using stateless web servers. Which services are suitable for storing session state data? Choose 3 answers.
1. Elastic Load Balancing
2. Amazon Relational Database Service (RDS)
3. Amazon CloudWatch
4. Amazon ElastiCache
5. Amazon DynamoDB
6. AWS Storage Gateway
Your firm has uploaded a large amount of aerial image data to S3. In the past, in your on-premises environment, you used a dedicated group of servers to oaten process this data and used Rabbit MQ, an open source messaging system, to get job information to the servers. Once processed the data would go to tape and be shipped offsite. Your manager told you to stay with the current design, and leverage AWS archival storage and messaging services to minimize cost. Which is correct? [PROFESSIONAL]
1. Use SQS for passing job messages, use Cloud Watch alarms to terminate EC2 worker instances when they become idle. Once data is processed, change the storage class of the S3 objects to Reduced Redundancy Storage.
2. Setup Auto-Scaled workers triggered by queue depth that use spot instances to process messages in SQS. Once data is processed, change the storage class of the S3 objects to Reduced Redundancy Storage.
3. Setup Auto-Scaled workers triggered by queue depth that use spot instances to process messages in SQS. Once data is processed, change the storage class of the S3 objects to Glacier.
4. Use SNS to pass job messages use Cloud Watch alarms to terminate spot worker instances when they become idle. Once data is processed, change the storage class of the S3 object to Glacier.
You are developing a new mobile application and are considering storing user preferences in AWS, which would provide a more uniform cross-device experience to users using multiple mobile devices to access the application. The preference data for each user is estimated to be 50KB in size. Additionally 5 million customers are expected to use the application on a regular basis. The solution needs to be cost-effective, highly available, scalable and secure, how would you design a solution to meet the above requirements? [PROFESSIONAL]
1. Setup an RDS MySQL instance in 2 availability zones to store the user preference data. Deploy a public facing application on a server in front of the database to manage security and access credentials
2. Setup a DynamoDB table with an item for each user having the necessary attributes to hold the user preferences. The mobile application will query the user preferences directly from the DynamoDB table. Utilize STS. Web Identity Federation, and DynamoDB Fine Grained Access Control to authenticate and authorize access
3. Setup an RDS MySQL instance with multiple read replicas in 2 availability zones to store the user preference data .The mobile application will query the user preferences from the read replicas. Leverage the MySQL user management and access privilege system to manage security and access credentials.
4. Store the user preference data in S3 Setup a DynamoDB table with an item for each user and an item attribute pointing to the user’ S3 object. The mobile application will retrieve the S3 URL from DynamoDB and then access the S3 object directly utilize STS, Web identity Federation, and S3 ACLs to authenticate and authorize access.
A company is building a voting system for a popular TV show, viewers would watch the performances then visit the show’s website to vote for their favorite performer. It is expected that in a short period of time after the show has finished the site will receive millions of visitors. The visitors will first login to the site using their Amazon.com credentials and then submit their vote. After the voting is completed the page will display the vote totals. The company needs to build the site such that can handle the rapid influx of traffic while maintaining good performance but also wants to keep costs to a minimum. Which of the design patterns below should they use? [PROFESSIONAL]
1. Use CloudFront and an Elastic Load balancer in front of an auto-scaled set of web servers, the web servers will first can the Login With Amazon service to authenticate the user then process the users vote and store the result into a multi-AZ Relational Database Service instance.
2. Use CloudFront and the static website hosting feature of S3 with the Javascript SDK to call the Login With Amazon service to authenticate the user, use IAM Roles to gain permissions to a DynamoDB table to store the users vote.
3. Use CloudFront and an Elastic Load Balancer in front of an auto-scaled set of web servers, the web servers will first call the Login with Amazon service to authenticate the user, the web servers will process the users vote and store the result into a DynamoDB table using IAM Roles for EC2 instances to gain permissions to the DynamoDB table.
4. Use CloudFront and an Elastic Load Balancer in front of an auto-scaled set of web servers, the web servers will first call the Login. With Amazon service to authenticate the user, the web servers would process the users vote and store the result into an SQS queue using IAM Roles for EC2 Instances to gain permissions to the SQS queue. A set of application servers will then retrieve the items from the queue and store the result into a DynamoDB table
A large real-estate brokerage is exploring the option to adding a cost-effective location-based alert to their existing mobile application. The application backend infrastructure currently runs on AWS. Users who opt in to this service will receive alerts on their mobile device regarding real-estate offers in proximity to their location. For the alerts to be relevant delivery time needs to be in the low minute count. The existing mobile app has 5 million users across the US. Which one of the following architectural suggestions would you make to the customer? [PROFESSIONAL]
1. Mobile application will submit its location to a web service endpoint utilizing Elastic Load Balancing and EC2 instances. DynamoDB will be used to store and retrieve relevant offers. EC2 instances will communicate with mobile earners/device providers to push alerts back to mobile application. —
2. Use AWS Direct Connect or VPN to establish connectivity with mobile carriers EC2 instances will receive the mobile applications location through carrier connection: RDS will be used to store and relevant offers. EC2 instances will communicate with mobile carriers to push alerts back to the mobile application
3. Mobile application will send device location using SQS. EC2 instances will retrieve the relevant offers from DynamoDB. AWS Mobile Push will be used to send offers to the mobile application
4. Mobile application will send device location using AWS Mobile Push. EC2 instances will retrieve the relevant offers from DynamoDB. EC2 instances will communicate with mobile carriers/device providers to push alerts back to the mobile application.
You are running a news website in the eu-west-1 region that updates every 15 minutes. The website has a worldwide audience and it uses an Auto Scaling group behind an Elastic Load Balancer and an Amazon RDS database. Static content resides on Amazon S3, and is distributed through Amazon CloudFront. Your Auto Scaling group is set to trigger a scale up event at 60% CPU utilization; you use an Amazon RDS extra-large DB instance with 10.000 Provisioned IOPS its CPU utilization is around 80%. While freeable memory is in the 2 GB range. Web analytics reports show that the average load time of your web pages is around 1.5 to 2 seconds, but your SEO consultant wants to bring down the average load time to under 0.5 seconds. How would you improve page load times for your users? (Choose 3 answers) [PROFESSIONAL]
1. Lower the scale up trigger of your Auto Scaling group to 30% so it scales more aggressively.
2. Add an Amazon ElastiCache caching layer to your application for storing sessions and frequent DB queries
3. Configure Amazon CloudFront dynamic content support to enable caching of re-usable content from your site
4. Switch Amazon RDS database to the high memory extra-large Instance type
5. Set up a second installation in another region, and use the Amazon Route 53 latency-based routing feature to select the right region.
A read only news reporting site with a combined web and application tier and a database tier that receives large and unpredictable traffic demands must be able to respond to these traffic fluctuations automatically. What AWS services should be used meet these requirements? [PROFESSIONAL]
1. Stateless instances for the web and application tier synchronized using ElastiCache Memcached in an autoscaling group monitored with CloudWatch. And RDS with read replicas.
2. Stateful instances for the web and application tier in an autoscaling group monitored with CloudWatch and RDS with read replicas
3. Stateful instances for the web and application tier in an autoscaling group monitored with CloudWatch. And multi-AZ RDS
4. Stateless instances for the web and application tier synchronized using ElastiCache Memcached in an autoscaling group monitored with CloudWatch and multi-AZ RDS
You have a periodic Image analysis application that gets some files as input, analyzes them and for each file writes some data in output to a ten file. The number of files in input per day is high and concentrated in a few hours of the day. Currently you have a server on EC2 with a large EBS volume that hosts the input data and the results it takes almost 20 hours per day to complete the process. What services could be used to reduce the elaboration time and improve the availability of the solution? [PROFESSIONAL]
1. S3 to store I/O files. SQS to distribute elaboration commands to a group of hosts working in parallel. Auto scaling to dynamically size the group of hosts depending on the length of the SQS queue
2. EBS with Provisioned IOPS (PIOPS) to store I/O files. SNS to distribute elaboration commands to a group of hosts working in parallel Auto Scaling to dynamically size the group of hosts depending on the number of SNS notifications
3. S3 to store I/O files, SNS to distribute evaporation commands to a group of hosts working in parallel. Auto scaling to dynamically size the group of hosts depending on the number of SNS notifications
4. EBS with Provisioned IOPS (PIOPS) to store I/O files SOS to distribute elaboration commands to a group of hosts working in parallel Auto Scaling to dynamically size the group to hosts depending on the length of the SQS queue.
A 3-tier e-commerce web application is current deployed on-premises and will be migrated to AWS for greater scalability and elasticity. The web server currently shares read-only data using a network distributed file system The app server tier uses a clustering mechanism for discovery and shared session state that depends on IP multicast The database tier uses shared-storage clustering to provide database fail over capability, and uses several read slaves for scaling. Data on all servers and the distributed file system directory is backed up weekly to off-site tapes. Which AWS storage and database architecture meets the requirements of the application? [PROFESSIONAL]
1. Web servers store read-only data in S3, and copy from S3 to root volume at boot time. App servers share state using a combination of DynamoDB and IP unicast. Database use RDS with multi-AZ deployment and one or more Read Replicas. Backup web and app servers backed up weekly via AMIs, database backed up via DB snapshots.
2. Web servers store read-only data in S3, and copy from S3 to root volume at boot time. App servers share state using a combination of DynamoDB and IP unicast. Database use RDS with multi-AZ deployment and one or more Read replicas. Backup web servers app servers, and database backed up weekly to Glacier using snapshots (Snapshots to Glacier don’t work directly with EBS snapshots)
3. Web servers store read-only data in S3 and copy from S3 to root volume at boot time. App servers share state using a combination of DynamoDB and IP unicast. Database use RDS with multi-AZ deployment. Backup web and app servers backed up weekly via AMIs. Database backed up via DB snapshots (Need Read replicas for scalability and elasticity)
4. Web servers, store read-only data in an EC2 NFS server, mount to each web server at boot time App servers share state using a combination of DynamoDB and IP multicast Database use RDS with multi-AZ deployment and one or more Read Replicas Backup web and app servers backed up weekly via AMIs database backed up via DB snapshots (IP multicast not available in AWS)
Our company is getting ready to do a major public announcement of a social media site on AWS. The website is running on EC2 instances deployed across multiple Availability Zones with a Multi-AZ RDS MySQL Extra Large DB Instance. The site performs a high number of small reads and writes per second and relies on an eventual consistency model. After comprehensive tests you discover that there is read contention on RDS MySQL. Which are the best approaches to meet these requirements? (Choose 2 answers) [PROFESSIONAL]
1. Deploy ElasticCache in-memory cache running in each availability zone
2. Implement sharding to distribute load to multiple RDS MySQL instances (Would distributed read write both, focus is on read contention)
3. Increase the RDS MySQL Instance size and Implement provisioned IOPS (Would distributed read write both, focus is on read contention)
4. Add an RDS MySQL read replica in each availability zone
Run 2-tier app with the following: an ELB, three web app server on EC2, and 1 MySQL RDS db. With grown load, db queries take longer and longer and slow down the overall response time for user request. What Options could speed up performance? (Choose 3) [PROFESSIONAL]
1. Create an RDS read-replica and redirect half of the database read request to it
2. Cache database queries in amazon ElastiCache
3. Setup RDS in multi-availability zone mode.
4. Shard the database and distribute loads between shards.
5. Use amazon CloudFront to cache database queries.
You have a web application leveraging an Elastic Load Balancer (ELB) In front of the web servers deployed using an Auto Scaling Group Your database is running on Relational Database Service (RDS) The application serves out technical articles and responses to them in general there are more views of an article than there are responses to the article. On occasion, an article on the site becomes extremely popular resulting in significant traffic Increases that causes the site to go down. What could you do to help alleviate the pressure on the infrastructure while maintaining availability during these events? Choose 3 answers [PROFESSIONAL]
1. Leverage CloudFront for the delivery of the articles.
2. Add RDS read-replicas for the read traffic going to your relational database
3. Leverage Elastic Cache for caching the most frequently used data.
4. Use SQS to queue up the requests for the technical posts and deliver them out of the queue (does not process and would not be real time)
5. Use Route53 health checks to fail over to an S3 bucket for an error page (more of an error handling then availability)
Your website is serving on-demand training videos to your workforce. Videos are uploaded monthly in high resolution MP4 format. Your workforce is distributed globally often on the move and using company-provided tablets that require the HTTP Live Streaming (HLS) protocol to watch a video. Your company has no video transcoding expertise and it required you might need to pay for a consultant. How do you implement the most cost-efficient architecture without compromising high availability and quality of video delivery? [PROFESSIONAL]
1. Elastic Transcoder to transcode original high-resolution MP4 videos to HLS. S3 to host videos with lifecycle Management to archive original flies to Glacier after a few days. CloudFront to serve HLS transcoded videos from S3 (Elastic Transcoder for High quality, S3 to host videos cheaply, Glacier for archives and CloudFront for high availability)
2. A video transcoding pipeline running on EC2 using SQS to distribute tasks and Auto Scaling to adjust the number or nodes depending on the length of the queue S3 to host videos with Lifecycle Management to archive all files to Glacier after a few days CloudFront to serve HLS transcoding videos from Glacier
3. Elastic Transcoder to transcode original high-resolution MP4 videos to HLS EBS volumes to host videos and EBS snapshots to incrementally backup original rues after a few days. CloudFront to serve HLS transcoded videos from EC2.
4. A video transcoding pipeline running on EC2 using SQS to distribute tasks and Auto Scaling to adjust the number of nodes depending on the length of the queue. EBS volumes to host videos and EBS snapshots to incrementally backup original files after a few days. CloudFront to serve HLS transcoded videos from EC2
To meet regulatory requirements, a pharmaceuticals company needs to archive data after a drug trial test is concluded. Each drug trial test may generate up to several thousands of files, with compressed file sizes ranging from 1 byte to 100MB. Once archived, data rarely needs to be restored, and on the rare occasion when restoration is needed, the company has 24 hours to restore specific files that match certain metadata. Searches must be possible by numeric file ID, drug name, participant names, date ranges, and other metadata. Which is the most cost-effective architectural approach that can meet the requirements? [PROFESSIONAL]
1. Store individual files in Amazon Glacier, using the file ID as the archive name. When restoring data, query the Amazon Glacier vault for files matching the search criteria. (Individual files are expensive and does not allow searching by participant names etc)
2. Store individual files in Amazon S3, and store search metadata in an Amazon Relational Database Service (RDS) multi-AZ database. Create a lifecycle rule to move the data to Amazon Glacier after a certain number of days. When restoring data, query the Amazon RDS database for files matching the search criteria, and move the files matching the search criteria back to S3 Standard class. (As the data is not needed can be stored to Glacier directly and the data need not be moved back to S3 standard)
3. Store individual files in Amazon Glacier, and store the search metadata in an Amazon RDS multi-AZ database. When restoring data, query the Amazon RDS database for files matching the search criteria, and retrieve the archive name that matches the file ID returned from the database query. (Individual files and Multi-AZ is expensive)
4. First, compress and then concatenate all files for a completed drug trial test into a single Amazon Glacier archive. Store the associated byte ranges for the compressed files along with other search metadata in an Amazon RDS database with regular snapshotting. When restoring data, query the database for files that match the search criteria, and create restored files from the retrieved byte ranges.
5. Store individual compressed files and search metadata in Amazon Simple Storage Service (S3). Create a lifecycle rule to move the data to Amazon Glacier, after a certain number of days. When restoring data, query the Amazon S3 bucket for files matching the search criteria, and retrieve the file to S3 reduced redundancy in order to move it back to S3 Standard class. (Once the data is moved from S3 to Glacier the metadata is lost, as Glacier does not have metadata and must be maintained externally)
A document storage company is deploying their application to AWS and changing their business model to support both free tier and premium tier users. The premium tier users will be allowed to store up to 200GB of data and free tier customers will be allowed to store only 5GB. The customer expects that billions of files will be stored. All users need to be alerted when approaching 75 percent quota utilization and again at 90 percent quota use. To support the free tier and premium tier users, how should they architect their application? [PROFESSIONAL]
1. The company should utilize an amazon simple work flow service activity worker that updates the users data counter in amazon dynamo DB. The activity worker will use simple email service to send an email if the counter increases above the appropriate thresholds.
2. The company should deploy an amazon relational data base service relational database with a store objects table that has a row for each stored object along with size of each object. The upload server will query the aggregate consumption of the user in questions by first determining the files store by the user, and then querying the stored objects table for respective file sizes) and send an email via amazon simple email service if the thresholds are breached.
3. The company should write both the content length and the username of the files owner as S3 metadata for the object. They should then create a file watcher to iterate over each object and aggregate the size for each user and send a notification via amazon simple queue service to an emailing service if the storage threshold is exceeded.
4. The company should create two separated amazon simple storage service buckets one for data storage for free tier users and another for data storage for premium tier users. An amazon simple workflow service activity worker will query all objects for a given user based on the bucket the data is stored
Your company has been contracted to develop and operate a website that tracks NBA basketball statistics. Statistical data to derive reports like “best game-winning shots from the regular season” and more frequently built reports like “top shots of the game” need to be stored durably for repeated lookup. Leveraging social media techniques, NBA fans submit and vote on new report types from the existing data set so the system needs to accommodate variability in data queries and new static reports must be generated and posted daily. Initial research in the design phase indicates that there will be over 3 million report queries on game day by end users and other applications that use this application as a data source. It is expected that this system will gain in popularity over time and reach peaks of 10-15 million report queries of the system on game days. Select the answer that will allow your application to best meet these requirements while minimizing costs. [PROFESSIONAL]
1. Launch a multi-AZ MySQL Amazon Relational Database Service (RDS) Read Replica connected to your multi AZ master database and generate reports by querying the Read Replica. Perform a daily table cleanup.
2. Implement a multi-AZ MySQL RDS deployment and have the application generate reports from Amazon ElastiCache for in-memory performance results. Utilize the default expire parameter for items in the cache.
3. Generate reports from a multi-AZ MySQL Amazon RDS deployment and have an offline task put reports in Amazon Simple Storage Service (S3) and use CloudFront to cache the content. Use a TTL to expire objects daily. (Offline task with S3 storage and CloudFront cache)
4. Query a multi-AZ MySQL RDS instance and store the results in a DynamoDB table. Generate reports from the DynamoDB table. Remove stale tables daily.

References

AWS_Storage_Options_Whitepaper

AWS Storage Options – SQS & Redshift

May 28, 2016 ~ Last updated on : November 15, 2016 ~ jayendrapatil ~ 13 Comments

SQS

is a temporary data repository for messages and provides a reliable, highly scalable, hosted message queuing service for temporary storage and delivery of short (up to 256 KB) text-based data messages.
supports a virtually unlimited number of queues and supports unordered, at-least-once delivery of messages.

Ideal Usage patterns

is ideally suited to any scenario where multiple application components must communicate and coordinate their work in a loosely coupled manner particularly producer consumer scenarios
can be used to coordinate a multi-step processing pipeline, where each message is associated with a task that must be processed.
enables the number of worker instances to scale up or down, and also enable the processing power of each single worker instance to scale up or down, to suit the total workload, without any application changes.

Anti-Patterns

Binary or Large Messages
- SQS is suited for text messages with maximum size of 64 KB. If the application requires binary or messages exceeding the length, it is best to use Amazon S3 or RDS and use SQS to store the pointer
Long Term storage
- SQS stores messages for max 14 days and if application requires storage period longer than 14 days, Amazon S3 or other storage options should be preferred
High-speed message queuing or very short tasks
- If the application requires a very high-speed message send and receive response from a single producer or consumer, use of Amazon DynamoDB or a message-queuing system hosted on Amazon EC2 may be more appropriate.

Performance

is a distributed queuing system that is optimized for horizontal scalability, not for single-threaded sending or receiving speeds.
A single client can send or receive Amazon SQS messages at a rate of about 5 to 50 messages per second. Higher receive performance can be achieved by requesting multiple messages (up to 10) in a single call.

Durability & Availability

are highly durable but temporary.
stores all messages redundantly across multiple servers and data centers.
Message retention time is configurable on a per-queue basis, from a minimum of one minute to a maximum of 14 days.
Messages are retained in a queue until they are explicitly deleted, or until they are automatically deleted upon expiration of the retention time.

Cost Model

pricing is based on
- number of requests and
- the amount of data transferred in and out (priced per GB per month).

Scalability & Elasticity

is both highly elastic and massively scalable.
is designed to enable a virtually unlimited number of computers to read and write a virtually unlimited number of messages at any time.
supports virtually unlimited numbers of queues and messages per queue for any user.

Amazon Redshift

is a fast, fully-managed, petabyte-scale data warehouse service that makes it simple and cost-effective to efficiently analyze all your data using your existing business intelligence tools.
is optimized for datasets that range from a few hundred gigabytes to a petabyte or more.
manages the work needed to set up, operate, and scale a data warehouse, from provisioning the infrastructure capacity to automating ongoing administrative tasks such as backups and patching.

Ideal Usage Pattern

is ideal for analyzing large datasets using the existing business intelligence tools
Common use cases include
- Analyze global sales data for multiple products
- Store historical stock trade data
- Analyze ad impressions and clicks
- Aggregate gaming data
- Analyze social trends
- Measure clinical quality, operation efficiency, and financial
- performance in the health care space

Anti-Pattern

OLTP workloads
- Redshift is a column-oriented database and more suited for data warehousing and analytics. If application involves online transaction processing, Amazon RDS would be a better choice.
Blob data
- For Blob storage, Amazon S3 would be a better choice with metadata in other storage as RDS or DynamoDB

Performance

Amazon Redshift allows a very high query performance on datasets ranging in size from hundreds of gigabytes to a petabyte or more.
It uses columnar storage, data compression, and zone maps to reduce the amount of I/O needed to perform queries.
It has a massively parallel processing (MPP) architecture that parallelizes and distributes SQL operations to take advantage of all available resources.
Underlying hardware is designed for high performance data processing that uses local attached storage to maximize throughput.

Durability & Availability

Amazon Redshift stores three copies of your data—all data written to a node in your cluster is automatically replicated to other nodes within the cluster, and all data is continuously backed up to Amazon S3.
Snapshots are automated, incremental, and continuous and stored for a user-defined period (1-35 days)
Manual snapshots can be created and are retained until explicitly deleted.
Amazon Redshift also continuously monitors the health of the cluster and automatically re-replicates data from failed drives and replaces nodes as necessary.

Cost Model

has three pricing components:
- data warehouse node hours – total number of hours run across all the compute node
- backup storage – storage cost for automated and manual snapshots
- data transfer
  - There is no data transfer charge for data transferred to or from Amazon Redshift outside of Amazon VPC
  - Data transfer to or from Amazon Redshift in Amazon VPC accrues standard AWS data transfer charges.

Scalability & Elasticity

provides push button scaling and the number of nodes can be easily scaled in the data warehouse cluster as the demand changes.
Redshift places the existing cluster in the read only mode, so the existing queries can continue to run, while is provisions a new cluster with chosen size and copies the data to it. Once the data is copied, it automatically redirects queries to the new cluster

AWS Storage Options – CloudFront & ElastiCache

May 27, 2016 ~ Last updated on : July 14, 2022 ~ jayendrapatil

Amazon CloudFront

is a webservice for content delivery
provides low latency by caching and delivering content from a global network of edge locations located nearest to the user
supports both HTTP to allows static, dynamic content and Real Time Messaging Protocol (RTMP) for streaming of videos
optimized to work as with Amazon services like S3, ELB etc. as well as works seamlessly with any non-AWS origin server

Ideal Usage Patterns

is ideal for distribution of frequently accessed static content, or dynamic content or for streaming audio or video that benefits from edge delivery

Anti-Pattern

Infrequently accessed data
- If the data is infrequently accessed, it would be better to serve the data from the Origin server
Programmatic cache invalidation
- CloudFront supports cache invalidation, however AWS recommends using object versioning rather than programmatic cache invalidation.

Performance

is designed for low latency and high bandwidth delivery of content by redirecting the user to the nearest edge location in terms of latency and caching the content preventing the round trip to the origin server

Durability & Availability

provides high Availability by delivering content from a distributed global network of edge locations. Amazon also constantly monitors the network paths connecting Origin servers to CloudFront
does not provide durable storage, which is more of the responsibility of the underlying Origin server providing the content for e.g. S3

Cost Model

has two pricing components:
- regional data transfer out (per GB) and
- requests (per 10,000)

Scalability & Elasticity

provides seamless scalability & elasticity by automatically responding to the increase or the decrease in the demand

ElastiCache

is a webservice that makes it easy to deploy, operate, and scale a distributed, in-memory cache in the cloud
helps improves performance of the applications by allowing retrieval of data from fast, managed, in-memory caching system
supports Memcached (object caching) & Redis (key value store that supports data structure) open source caching engines

Ideal Usage Patterns

improving application performance by storing critical data in-memory for low latency access
use cases involve usage as a database front end for read heavy applications, improving performance and reducing load on databases, or managing user session data, cache dynamically generated pages, or compute intensive calculations etc.

Anti-Patterns

Persistent Data
- If the application needs fast access to data coupled with strong data durability, Amazon DynamoDB would be a better option

Performance

Although ElastiCache provides low latency access to the data, the performance depends on the caching strategy and the hit ratio at the application level

Durability & Availability

stores transient data or transient copies of durable data, so the data durability is managed by the source
With the Memcached engine
- all ElastiCache nodes in a single cache cluster are provisioned in a single Availability Zone.
- ElastiCache automatically monitors the health of your cache nodes and replaces them in the event of network partitioning, host hardware, or software failure.
- In the event of cache node failure, the cluster remains available, but performance may be reduced due to time needed to repopulate the cache in the new “cold” cache nodes.
- To provide enhanced fault-tolerance for Availability Zone failures or cold-cache effects, you can run redundant cache clusters in different Availability Zones.
With the Redis engine,
- ElastiCache supports replication to up to five read replicas for scaling. To improve availability, you can place read replicas in other Availability Zones.
- ElastiCache monitors the primary node, and if the node becomes unavailable, ElastiCache will repair or replace the primary node if possible, using the same DNS name.
- If the primary cache node recovery fails or its Availability Zone is unavailable, primary node can be failed over to one of the read replicas with an API call.

Cost Model

has a single pricing component:
- pricing is per cache node-hour consumed

Scalability & Elasticity

ElastiCache is highly scalable and elastic.
Cache node can be added or deleted to the cache cluster
Auto Discovery enables automatic discovery of Memcached cache nodes by ElastiCache Clients when the nodes are added to or removed from an ElastiCache cluster.

Storage Options Whitepaper – Storage Gateway – Import/Export – AWS Certification

May 26, 2016 ~ Last updated on : February 9, 2017 ~ jayendrapatil ~ 8 Comments

AWS Storage Options Whitepaper cont.

Provides a brief summary for the Ideal Use cases and Anti-Patterns for Storage Gateway and Import/Export AWS storage options

AWS Storage Gateway

Storage Gateway is a service that connects an on-premises software appliance with cloud-based storage to provide seamless and secure integration between the organization’s on-premises IT environment and AWS’s storage infrastructure.
Storage Gateway enables store data securely to the AWS cloud for scalable and cost-effective storage.
It provides low-latency performance by maintaining frequently accessed data on-premises while securely storing all of your data encrypted in S3.
For disaster recovery scenarios, it can serve as a cloud-hosted solution, together with EC2, that mirrors your entire production environment.
Storage Gateway can be configured as
- Gateway-cached volumes
  - Gateway-cached volumes utilizes S3 for primary data backup, while retaining frequently accessed data locally in a cache.
  - These volumes minimize the need to scale the on-premises storage infrastructure, while still providing applications with low-latency access to their frequently accessed data.
  - Data written to the volumes is stored in S3, with only a cache of recently written and recently read data is stored locally on the on-premises storage hardware.
- Gateway-stored volumes
  - Gateway-stored volumes stores the complete primary data locally, while asynchronously backing up that data to AWS.
  - These volumes provide the on-premises applications with low-latency access to their entire datasets, while providing durable, off-site backups.
  - Data written to the gateway-stored volumes is stored on the on-premises storage hardware, and asynchronously backed up to S3 in the form of EBS snapshots.

Ideal Usage Patterns

AWS Storage Gateway use cases include
- corporate file sharing,
- enabling existing on-premises backup applications to store primary backups on S3,
- disaster recovery, and
- data mirroring to cloud-based compute resources.

Anti-Patterns

Database storage
- For Database backup or storage, EC2 instances using EBS volumes are a natural choice for database storage and workloads.

Performance

As the Storage Gateway VM sits between the application, underlying on-premises storage and S3, the performance experienced will be dependent upon a number of factors, including the speed and configuration of the underlying local disks, the network bandwidth between the iSCSI initiator and gateway VM, the amount of local storage allocated to the gateway VM, and the bandwidth between the gateway VM and S3.
For gateway-cached volumes, to provide low-latency read access to the on-premises applications, it’s important to provide enough local cache storage to store the recently accessed data.
Storage Gateway efficiently uses the Internet bandwidth to speed up the upload of on-premises application data to AWS.
Storage Gateway only uploads incremental changes (data that has changed), which minimizes the amount of data sent over the Internet.
AWS Direct Connect can be used to further increase throughput and reduce the network costs by establishing a dedicated network connection between the on-premises gateway and AWS.

Durability and Availability

AWS Storage Gateway durably stores on-premises application data by uploading it to S3.
S3 stores data in multiple facilities and on multiple devices within each facility.
S3 also performs regular, systematic data integrity checks and is built to be automatically self-healing.

Cost Model

AWS Storage Gateway has four pricing components:
- gateway usage (per gateway per month),
- snapshot storage usage (per GB per month),
- volume storage usage (per GB per month), and
- data transfer out (per GB per month).

Scalability and Elasticity

AWS Storage Gateway stores data in Amazon S3, which has been designed to offer a very high level of scalability and elasticity automatically.

Interfaces

AWS Management Console can be used to download the AWS Storage Gateway VM image, select between a gateway-cached or gateway-stored configuration, activate the on-premises by associating the gateway’s IP Address with your AWS account, select an AWS region, and create AWS Storage Gateway volumes and attach these volumes as iSCSI devices to your on-premises application servers.

AWS Import/Export (Upgraded to Snowball)

AWS Import/Export accelerates moving large amounts of data into and out of AWS using portable storage devices for transport.
AWS transfers the data directly onto and off of storage devices using Amazon’s high-speed internal network and bypassing the Internet and can be much faster and more cost effective than upgrading connectivity.
AWS Import/Export supports importing into several types of AWS storage, including EBS snapshots, S3 buckets, and Glacier vaults and exporting data from S3.

Ideal Usage Patterns

AWS Import/Export is ideal for transferring large amounts of data in and out of the AWS cloud, especially in cases where transferring the data over the Internet would be too slow (a week or more) or too costly.
Common use cases include
- initial data upload to AWS,
- content distribution or regular data interchange to/from your customers or business associates,
- transfer to Amazon S3 or Amazon Glacier for off-site backup and archival storage, and quick retrieval of large backups from Amazon S3 or Amazon Glacier for disaster recovery.

Anti-Patterns

AWS Import/Export may not be the ideal solution for data that is more easily transferred over the Internet in less than one week.

Performance

Each AWS Import/Export station is capable of loading data at over 100 MB per second
Rate of the data load will be bounded by a combination of the read or write speed of the portable storage device and, for Amazon S3 data loads, the average object (file) size.

Durability and Availability

Durability and availability characteristics of the target storage i.e. EBS, S3 or Glacier applies, after the data has been imported

Cost Model

AWS Import/Export has three pricing components: a per-device fee, a data load time charge (per data-loading-hour), and possible return shipping charges (for expedited shipping, or shipping to destinations not local to that AWS Import/Export region).
Storage pricing applies for the destination storage, the standard Amazon EBS snapshot, Amazon S3, and Amazon Glacier request and storage pricing applies.

Scalability and Elasticity

Total amount of data you can load using AWS Import/Export is limited only by the capacity of the devices sent to AWS.
For Amazon S3, individual files will be loaded as objects in Amazon S3, and may range up to 5 terabytes in size.
For Amazon Glacier, individual devices will be loaded as a single archive, and may range up to 4 terabytes in size.
Aggregate total amount of data that can be imported is virtually unlimited.

Interfaces

To upload or download data, AWS Import/Export job for each storage device shipped need to be created and submitted
Jobs can be created using AWS CLI, AWS SDK or native REST API
Each job request requires a manifest file, a YAML-formatted text file that contains a set of key-value pairs that supply the required information—such as your device ID, secret access key, and return address—necessary to complete the job.
Job request is tied to the storage device through a signature file in the root directory (for Amazon S3 import jobs), or by a barcode taped to the device (for Amazon EBS and Amazon Glacier jobs).

AWS Certification Exam Practice Questions

Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).

AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.

AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated

Open to further feedback, discussion and correction.

You are working with a customer who has 10 TB of archival data that they want to migrate to Amazon Glacier. The customer has a 1-Mbps connection to the Internet. Which service or feature provides the fastest method of getting the data into Amazon Glacier?
1. Amazon Glacier multipart upload
2. AWS Storage Gateway
3. VM Import/Export
4. AWS Import/Export

AWS Storage Options – RDS, DynamoDB & Database on EC2

May 25, 2016 ~ Last updated on : March 20, 2017 ~ jayendrapatil ~ 17 Comments

AWS Storage Options Whitepaper with RDS, DynamoDB & Database on EC2 Cont.

Provides a brief summary for the Ideal Use cases, Anti-Patterns and other factors for Amazon RDS, DynamoDB & Databases on EC2 storage options

Amazon RDS

RDS is a web service that provides the capabilities of MySQL, Oracle, MariaDB, Postgres or Microsoft SQL Server relational database as a managed, cloud-based service
RDS eliminates much of the administrative overhead associated with launching, managing, and scaling your own relational database on Amazon EC2 or in another computing environment.

Ideal Usage Patterns

RDS is a great solution for cloud-based fully-managed relational database
RDS is also optimal for new applications with structured data that requires more sophisticated querying and joining capabilities than that provided by Amazon’s NoSQL database offering, DynamoDB.
RDS provides full compatibility with the databases supported and direct access to native database engines, code and libraries and is ideal for existing applications that rely on these databases

Anti-Patterns

Index and query-focused data
- If the applications don’t require advanced features such as joins and complex transactions and is more oriented toward indexing and querying data, DynamoDB would be more appropriate for this needs
Numerous BLOBs
- If the application makes heavy use of files (audio files, videos, images, etc), it is a better choice to use S3 to store the objects instead of database engines Blob feature and use RDS or DynamoDB only to save the metadata
Automated scalability
- RDS provides pushbutton scaling and it only scales up and has limited scale out ability. If fully-automated scaling is needed, DynamoDB may be a better choice.
Complete control
- RDS does not provide admin access and does not enable the full feature set of the database engines.
- So if the application requires complete, OS-level control of the database server with full root or admin login privileges, a self-managed database on EC2 may be a better match.
Other database platforms
- RDS, at this time, provides a MySQL, Oracle, MariaDB, PostgreSQL and SQL Server databases.
- If any other database platform (such as IBM DB2, Informix, or Sybase) is needed, it should be deployed on a self-managed database on an EC2 instance by using a relational database AMI, or by installing database software on an EC2 instance.

Performance

RDS Provisioned IOPS, where the IOPS can be specified when the instance is launched and is guaranteed over the life of the instance, provides a high-performance storage option designed to deliver fast, predictable, and consistent performance for I/O intensive transactional database workload

Durability and Availability

RDS leverages Amazon EBS volumes as its data store
RDS provides database backups, for enhanced durability, which are replicated across multiple AZ’s
- Automated backups
  - If enabled, RDS will automatically perform a full daily backup of your data during the specified backup window, and will also capture DB transaction logs
- User initiated backups
  - User can initiate backups at time and they are not deleted unless deleted explicitly by the user
RDS Multi AZ’s feature enhances both the durability and the availability of the database by synchronously replicating the data between a primary RDS DB instance and a standby instance in another Availability Zone, which prevents data loss,
RDS provides a DNS endpoint and in case of an failure on the primary, it automatically fails over to the standby instance
RDS also allows Read replicas for the supported databases, which are replicated asynchronously

Cost Model

RDS offers a tiered pricing structure, based on the size of the database instance, the deployment type (Single-AZ/Multi-AZ), and the AWS region.
Pricing for RDS is based on several factors: the DB instance hours (per hour), the amount of provisioned database storage (per GB-month and per million I/O requests), additional backup storage (per GB-month), and data transfer in/out (per GB per month)

Scalability and Elasticity

RDS resources can be scaled elastically in several dimensions: database storage size, database storage IOPS rate, database instance compute capacity, and the number of read replicas
RDS supports “pushbutton scaling” of both database storage and compute resources. Additional storage can either be added immediately or during the next maintenance cycle
RDS for MySQL also enables you to scale out beyond the capacity of a single database deployment for read-heavy database workloads by creating one or more read replicas.
Multiple RDS instances can also be configured to leverage database partitioning or sharding to spread the workload over multiple DB instances, achieving even greater database scalability and elasticity.

Interfaces

RDS APIs and the AWS Management Console provide a management interface that allows you to create, delete, modify, and terminate RDS DB instances; to create DB snapshots; and to perform point-in-time restores
There is no AWS data API for Amazon RDS.
Once a database is created, RDS provides a DNS endpoint for the database which can be used to connect to the database.
Endpoint does not change over the lifetime of the instance even during the failover in case of Multi-AZ configuration

Amazon DynamoDB

Amazon DynamoDB is a fast, fully-managed NoSQL database service that makes it simple and cost-effective to store and retrieve any amount of data, and serve any level of request traffic.
DynamoDB being a managed service helps offload the administrative burden of operating and scaling a highly-available distributed database cluster.
DynamoDB helps meet the latency and throughput requirements of highly demanding applications by providing extremely fast and predictable performance with seamless throughput and storage scalability.
DynamoDB provides both eventually-consistent reads (by default), and strongly-consistent reads (optional), as well as implicit item-level transactions for item put, update, delete, conditional operations, and increment/decrement.
Amazon DynamoDB handles the data as below :-
- DynamoDB stores structured data in tables, indexed by primary key, and allows low-latency read and write access to items.
- DynamoDB supports three data types: number, string, and binary, in both scalar and multi-valued sets.
- Tables do not have a fixed schema, so each data item can have a different number of attributes.
- Primary key can either be a single-attribute hash key or a composite hash-range key.
- Local secondary indexes provide additional flexibility for querying against attributes other than the primary key.

Ideal Usage Patterns

DynamoDB is ideal for existing or new applications that need a flexible NoSQL database with low read and write latencies, and the ability to scale storage and throughput up or down as needed without code changes or downtime.
Use cases require a highly available and scalable database because downtime or performance degradation has an immediate negative impact on an organization’s business. for e.g. mobile apps, gaming, digital ad serving, live voting and audience interaction for live events, sensor networks, log ingestion, access control for web-based content, metadata storage for S3 objects, e-commerce shopping carts, and web session management

Anti-Patterns

Structured data with Join and/or Complex Transactions
- If the application uses structured data and required joins, complex transactions or other relationship infrastructure provided by traditional database platforms, it is better to use RDS or Database installed on an EC2 instance
Large Blob data
- If the application uses large blob data for e.g. media, files, videos etc., it is better to use S3 to store the objects and use DynamoDB to store metadata for e.g. name, size, content-type etc
Large Objects with Low I/O rate
- DynamoDB uses SSD drives and is optimized for workloads with a high I/O rate per GB stored. If the applications stores very large amounts of data that are infrequently accessed, S3 might be a better choice
Prewritten application with databases
- For Porting an existing application using databases, RDS or database installed on the EC2 instance would be a better and seamless solution

Performance

SSDs and limited indexing on attributes provides high throughput and low latency and drastically reduces the cost of read and write operations.
Predictable performance can be achieved by defining the provisioned throughput capacity required for a given table.
DynamoDB handles the provisioning of resources to achieve the requested throughput rate, taking away the burden to think about instances, hardware, memory, and other factors that can affect an application’s throughput rate.
Provisioned throughput capacity reservations are elastic and can be increased or decreased on demand.

Durability and Availability

DynamoDB has built-in fault tolerance that automatically and synchronously replicates data across three AZ’s in a region for high availability and to help protect data against individual machine, or even facility failures.

Cost Model

DynamoDB has three pricing components: provisioned throughput capacity (per hour), indexed data storage (per GB per month), data transfer in or out (per GB per month)

Scalability and Elasticity

DynamoDB is both highly-scalable and elastic.
DynamoDB provides unlimited storage capacity, and the service automatically allocates more storage as the demand increases
Data is automatically partitioned and re-partitioned as needed, while the use of SSDs provides predictable low-latency response times at any scale.
DynamoDB is also elastic, in that you can simply “dial-up” or “dial-down” the read and write capacity of a table as your needs change.

Interfaces

DynamoDB provides a low-level REST API, as well as higher-level SDKs in different languages
APIs provide both a management and data interface for Amazon DynamoDB, that enable table management (creating, listing, deleting, and obtaining metadata) and working with attributes (getting, writing, and deleting attributes; query using an index, and full scan).

Databases on EC2

EC2 with EBS volumes allows hosting a self managed relational database
Ready to use, prebuilt AMIs are also available from leading database solutions

Ideal Usage Patterns

Self managed database on EC2 is an ideal scenario for users whose application requires a specific traditional relational database not supported by Amazon RDS for e.g. IBM DB2, Informix, or Sybase
Users or applications that require a maximum level of administrative control and configurability which is not provided by RDS

Anti-Patterns

Index and query-focused data
- If the applications don’t require advanced features such as joins and complex transactions and is more oriented toward indexing and querying data, DynamoDB would be more appropriate for this needs
Numerous BLOBs
- If the application makes heavy use of files (audio files, videos, images, and so on), it is a better choice to use S3 to store the objects instead of database engines Blob feature and use RDS or DynamoDB only to save the metadata
Automated scalability
- Relational databases on EC2 leverages the scalability and elasticity of the underlying AWS platform, but this requires system administrators or DBAs to perform a manual or scripted task. If you need pushbutton scaling or fully-automated scaling, DynamoDB or RDS may be a better choice.
RDS supported database platforms
- If the application using RDS supported database engine and all the features are available, RDS would be a better choice instead of self managed relational database on EC2

Performance

Performance depends on the size of the underlying EC2 instance, the number and configuration of the EBS volumes and the database itself
Performance can be increased by scaling up memory and compute resources by choosing a larger Amazon EC2 instance size.
For database storage, it is usually best to use EBS Provisioned IOPS volumes. To scale up I/O performance, the Provisioned IOPS can be increased, the number of EBS volumes changed, or use software RAID 0 (disk striping) across multiple EBS volumes, which will aggregate total IOPS and bandwidth.

Durability & Availability

As the database on EC2 uses EBS as storage, it has the same durability and availability provided by EBS and can be further enhanced by using EBS snapshots or by using third-party database backup utilities (such as Oracle’s RMAN) to store database backups in Amazon S3

Cost Model

Cost for running a database on EC2 instance is mainly determined by the size and the number of EC2 instance running, the size of the EBS volume used for database storage and any third party licensing cost for the database

Scalability & Elasticity

Users of traditional relational database solutions on Amazon EC2 can take advantage of the scalability and elasticity of the underlying AWS platform by creating AMI and spawning multiple instances

AWS Certification Exam Practice Questions

Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).

AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.

AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated

Open to further feedback, discussion and correction.

Which of the following are use cases for Amazon DynamoDB? Choose 3 answers
1. Storing BLOB data.
2. Managing web sessions
3. Storing JSON documents
4. Storing metadata for Amazon S3 objects
5. Running relational joins and complex updates.
6. Storing large amounts of infrequently accessed data.
A client application requires operating system privileges on a relational database server. What is an appropriate configuration for highly available database architecture?
1. A standalone Amazon EC2 instance
2. Amazon RDS in a Multi-AZ configuration
3. Amazon EC2 instances in a replication configuration utilizing a single Availability Zone
4. Amazon EC2 instances in a replication configuration utilizing two different Availability Zones
You are developing a new mobile application and are considering storing user preferences in AWS, which would provide a more uniform cross-device experience to users using multiple mobile devices to access the application. The preference data for each user is estimated to be 50KB in size. Additionally 5 million customers are expected to use the application on a regular basis. The solution needs to be cost-effective, highly available, scalable and secure, how would you design a solution to meet the above requirements?
1. Setup an RDS MySQL instance in 2 availability zones to store the user preference data. Deploy a public facing application on a server in front of the database to manage security and access credentials
2. Setup a DynamoDB table with an item for each user having the necessary attributes to hold the user preferences. The mobile application will query the user preferences directly from the DynamoDB table. Utilize STS. Web Identity Federation, and DynamoDB Fine Grained Access Control to authenticate and authorize access (DynamoDB provides high availability as it synchronously replicates data across three facilities within an AWS Region and scalability as it is designed to scale its provisioned throughput up or down while still remaining available. Also suitable for storing user preference data)
3. Setup an RDS MySQL instance with multiple read replicas in 2 availability zones to store the user preference data .The mobile application will query the user preferences from the read replicas. Leverage the MySQL user management and access privilege system to manage security and access credentials.
4. Store the user preference data in S3 Setup a DynamoDB table with an item for each user and an item attribute pointing to the user’ S3 object. The mobile application will retrieve the S3 URL from DynamoDB and then access the S3 object directly utilize STS, Web identity Federation, and S3 ACLs to authenticate and authorize access.
A customer is running an application in US-West (Northern California) region and wants to setup disaster recovery failover to the Asian Pacific (Singapore) region. The customer is interested in achieving a low Recovery Point Objective (RPO) for an Amazon RDS multi-AZ MySQL database instance. Which approach is best suited to this need?
1. Synchronous replication
2. Asynchronous replication
3. Route53 health checks
4. Copying of RDS incremental snapshots
You are designing a file -sharing service. This service will have millions of files in it. Revenue for the service will come from fees based on how much storage a user is using. You also want to store metadata on each file, such as title, description and whether the object is public or private. How do you achieve all of these goals in a way that is economical and can scale to millions of users?
1. Store all files in Amazon Simple Storage Service (53). Create a bucket for each user. Store metadata in the filename of each object, and access it with LIST commands against the S3 API.
2. Store all files in Amazon 53. Create Amazon DynamoDB tables for the corresponding key -value pairs on the associated metadata, when objects are uploaded.
3. Create a striped set of 4000 IOPS Elastic Load Balancing volumes to store the data. Use a database running in Amazon Relational Database Service (RDS) to store the metadata.
4. Create a striped set of 4000 IOPS Elastic Load Balancing volumes to store the data. Create Amazon DynamoDB tables for the corresponding key-value pairs on the associated metadata, when objects are uploaded.
Company ABCD has recently launched an online commerce site for bicycles on AWS. They have a “Product” DynamoDB table that stores details for each bicycle, such as, manufacturer, color, price, quantity and size to display in the online store. Due to customer demand, they want to include an image for each bicycle along with the existing details. Which approach below provides the least impact to provisioned throughput on the “Product” table?
1. Serialize the image and store it in multiple DynamoDB tables
2. Create an “Images” DynamoDB table to store the Image with a foreign key constraint to the “Product” table
3. Add an image data type to the “Product” table to store the images in binary format
4. Store the images in Amazon S3 and add an S3 URL pointer to the “Product” table item for each image

AWS Storage Options – S3 & Glacier

May 23, 2016 ~ Last updated on : June 8, 2018 ~ jayendrapatil ~ 30 Comments

Amazon S3

highly-scalable, reliable, and low-latency data storage infrastructure at very low costs.
provides a simple web services interface that can be used to store and retrieve any amount of data, at any time, from within Amazon EC2 or from anywhere on the web.
allows you to write, read, and delete objects containing from 1 byte to 5 terabytes of data each.
number of objects you can store in an Amazon S3 bucket is virtually unlimited.
highly secure, supporting encryption at rest, and providing multiple mechanisms to provide fine-grained control of access to Amazon S3 resources.
highly scalable, allowing concurrent read or write access to Amazon S3 data by many separate clients or application threads.
provides data lifecycle management capabilities, allowing users to define rules to automatically archive Amazon S3 data to Amazon Glacier, or to delete data at end of life.

Ideal Use Cases

Storage & Distribution of static web content and media
- frequently used to host static websites and provides a highly-available and highly-scalable solution for websites with only static content, including HTML files, images, videos, and client-side scripts such as JavaScript
- works well for fast growing websites hosting data intensive, user-generated content, such as video and photo sharing sites as no storage provisioning is required
- content can either be directly served from Amazon S3 since each object in Amazon S3 has a unique HTTP URL address
- can also act as an Origin store for the Content Delivery Network (CDN) such as Amazon CloudFront
- it works particularly well for hosting web content with extremely spiky bandwidth demands because of S3’s elasticity
Data Store for Large Objects
- can be paired with RDS or NoSQL database and used to store large objects for e.g. file or objects, while the associated metadata for e.g. name, tags, comments etc. can be stored in RDS or NoSQL database where it can be indexed and queried providing faster access to relevant data
Data store for computation and large-scale analytics
- commonly used as a data store for computation and large-scale analytics, such as analyzing financial transactions, clickstream analytics, and media transcoding.
- data can be accessed from multiple computing nodes concurrently without being constrained by a single connection because of its horizontal scalability
Backup and Archival of critical data
- used as a highly durable, scalable, and secure solution for backup and archival of critical data, and to provide disaster recovery solutions for business continuity.
- stores objects redundantly on multiple devices across multiple facilities, it provides the highly-durable storage infrastructure needed for these scenarios.
- it’s versioning capability is available to protect critical data from inadvertent deletion

Anti-Patterns

Amazon S3 has following Anti-Patterns where it is not an optimal solution

Dynamic website hosting
- While Amazon S3 is ideal for hosting static websites, dynamic websites requiring server side interaction, scripting or database interaction cannot be hosted and should rather be hosted on Amazon EC2
Backup and archival storage
- Data requiring long term archival storage with infrequent read access can be stored more cost effectively in Amazon Glacier
Structured Data Query
- Amazon S3 doesn’t offer query capabilities, so to read an object the object name and key must be known. Instead pair up S3 with RDS or Dynamo DB to store, index and query metadata about Amazon S3 objects
- NOTE – S3 now provides query capabilities and also Athena can be used
Rapidly Changing Data
- Data that needs to updated frequently might be better served by a storage solution with lower read/write latencies, such as Amazon EBS volumes, RDS or Dynamo DB.
File System
- Amazon S3 uses a flat namespace and isn’t meant to serve as a standalone, POSIX-compliant file system. However, by using delimiters (commonly either the ‘/’ or ‘’ character) you are able construct your keys to emulate the hierarchical folder structure of file system within a given bucket.

Performance

Access to Amazon S3 from within Amazon EC2 in the same region is fast.
Amazon S3 is designed so that server-side latencies are insignificant relative to Internet latencies.
Amazon S3 is also built to scale storage, requests, and users to support a virtually unlimited number of web-scale applications.
If Amazon S3 is accessed using multiple threads, multiple applications, or multiple clients concurrently, total Amazon S3 aggregate throughput will typically scale to rates that far exceed what any single server can generate or consume.

Durability & Availability

Amazon S3 storage provides provides the highest level of data durability and availability, by automatically and synchronously storing your data across both multiple devices and multiple facilities within the selected geographical region
Error correction is built-in, and there are no single points of failure. Amazon S3 is designed to sustain the concurrent loss of data in two facilities, making it very well-suited to serve as the primary data storage for mission-critical data.
Amazon S3 is designed for 99.999999999% (11 nines) durability per object and 99.99% availability over a one-year period.
Amazon S3 data can be protected from unintended deletions or overwrites using Versioning.
Versioning can be enabled with MFA (Multi Factor Authentication) Delete on the bucket, which would require two forms of authentication to delete an object
For Non Critical and Reproducible data for e.g. thumbnails, transcoded media etc., S3 Reduced Redundancy Storage (RRS) can be used, which provides a lower level of durability at a lower storage cost
RRS is designed to provide 99.99% durability per object over a given year. While RRS is less durable than standard Amazon S3, it is still designed to provide 400 times more durability than a typical disk drive

Cost Model

With Amazon S3, you pay only for what you use and there is no minimum fee.
Amazon S3 has three pricing components: storage (per GB per month), data transfer in or out (per GB per month), and requests (per n thousand requests per month).

Scalability & Elasticity

Amazon S3 has been designed to offer a very high level of scalability and elasticity automatically
Amazon S3 supports a virtually unlimited number of files in any bucket
Amazon S3 bucket can store a virtually unlimited number of bytes
Amazon S3 allows you to store any number of objects (files) in a single bucket, and Amazon S3 will automatically manage scaling and distributing redundant copies of your information to other servers in other locations in the same region, all using Amazon’s high-performance infrastructure.

Interfaces

Amazon S3 provides standards-based REST and ~~SOAP web services~~ APIs for both management and data operations.
NOTE – SOAP support over HTTP is deprecated, but it is still available over HTTPS. New Amazon S3 features will not be supported for SOAP. We recommend that you use either the REST API or the AWS SDKs.
Amazon S3 provides easier to use higher level toolkit or SDK in different languages (Java, .NET, PHP, and Ruby) that wraps the underlying APIs
Amazon S3 Command Line Interface (CLI) provides a set of high-level, Linux-like Amazon S3 file commands for common operations, such as ls, cp, mv, sync, etc. They also provide the ability to perform recursive uploads and downloads using a single folder-level Amazon S3 command, and supports parallel transfers.
AWS Management Console provides the ability to easily create and manage Amazon S3 buckets, upload and download objects, and browse the contents of your Amazon S3 buckets using a simple web-based user interface
All interfaces provide the ability to store Amazon S3 objects (files) in uniquely-named buckets (top-level folders), with each object identified by an unique Object key within that bucket.

Glacier

extremely low-cost storage service that provides highly secure, durable, and flexible storage for data backup and archival
can reliably store their data for as little as $0.01 per gigabyte per month.
to offload the administrative burdens of operating and scaling storage to AWS such as capacity planning, hardware provisioning, data replication, hardware failure detection and repair, or time consuming hardware migrations
Data is stored in Amazon Glacier as Archives where an archive can represent a single file or multiple files combined into a single archive
Archives are stored in Vaults for which the access can be controlled through IAM
Retrieving archives from Vaults require initiation of a job and can take anywhere around 3-5 hours
Amazon Glacier integrates seamlessly with Amazon S3 by using S3 data lifecycle management policies to move data from S3 to Glacier
AWS Import/Export can also be used to accelerate moving large amounts of data into Amazon Glacier using portable storage devices for transport

Ideal Usage Patterns

Amazon Glacier is ideally suited for long term archival solution for infrequently accessed data with archiving offsite enterprise information, media assets, research and scientific data, digital preservation and magnetic tape replacement

Anti-Patterns

Amazon Glacier has following Anti-Patterns where it is not an optimal solution

Rapidly changing data
- Data that must be updated very frequently might be better served by a storage solution with lower read/write latencies such as Amazon EBS or a Database
Real time access
- Data stored in Glacier can not be accessed at real time and requires an initiation of a job for object retrieval with retrieval times ranging from 3-5 hours. If immediate access is needed, Amazon S3 is a better choice.

Performance

Amazon Glacier is a low-cost storage service designed to store data that is infrequently accessed and long lived.
Amazon Glacier jobs typically complete in 3 to 5 hours

Durability and Availability

Amazon Glacier redundantly stores data in multiple facilities and on multiple devices within each facility
Amazon Glacier is designed to provide average annual durability of 99.999999999% (11 nines) for an archive
Amazon Glacier synchronously stores your data across multiple facilities before returning SUCCESS on uploading archives.
Amazon Glacier also performs regular, systematic data integrity checks and is built to be automatically self-healing.

Cost Model

Amazon Glacier has three pricing components: storage (per GB per month), data transfer out (per GB per month), and requests (per thousand UPLOAD and RETRIEVAL requests per month).
Amazon Glacier is designed with the expectation that retrievals are infrequent and unusual, and data will be stored for extended periods of time and allows you to retrieve up to 5% of your average monthly storage (pro-rated daily) for free each month. Any additional amount of data retrieved is charged per GB
Amazon Glacier also charges a pro-rated charge (per GB) for items deleted prior to 90 days

Scalability & Elasticity

A single archive is limited to 40 TBs, but there is no limit to the total amount of data you can store in the service.
Amazon Glacier scales to meet your growing and often unpredictable storage requirements whether you’re storing petabytes or gigabytes, Amazon Glacier automatically scales your storage up or down as needed.

Interfaces

Amazon Glacier provides a native, standards-based REST web services interface, as well as Java and .NET SDKs.
AWS Management Console or the Amazon Glacier APIs can be used to create vaults to organize the archives in Amazon Glacier.
Amazon Glacier APIs can be used to upload and retrieve archives, monitor the status of your jobs and also configure your vault to send you a notification via Amazon Simple Notification Service (Amazon SNS) when your jobs complete.
Amazon Glacier can be used as a storage class in Amazon S3 by using object lifecycle management to provide automatic, policy-driven archiving from Amazon S3 to Amazon Glacier.
Amazon S3 api provides a RESTORE operation and the retrieval process takes the same 3-5 hours
On retrieval, a copy of the retrieved object is placed in Amazon S3 RRS storage for a specified retention period; the original archived object remains stored in Amazon Glacier and you are charged for both the storage.
When using Amazon Glacier as a storage class in Amazon S3, use the Amazon S3 APIs, and when using “native” Amazon Glacier, you use the Amazon Glacier APIs
Objects archived to Amazon Glacier via Amazon S3 can only be listed and retrieved via the Amazon S3 APIs or the AWS Management Console—they are not visible as archives in an Amazon Glacier vault.

AWS Certification Exam Practice Questions

Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).

AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.

AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated

Open to further feedback, discussion and correction.

You want to pass queue messages that are 1GB each. How should you achieve this?
1. Use Kinesis as a buffer stream for message bodies. Store the checkpoint id for the placement in the Kinesis Stream in SQS.
2. Use the Amazon SQS Extended Client Library for Java and Amazon S3 as a storage mechanism for message bodies. (Amazon SQS messages with Amazon S3 can be useful for storing and retrieving messages with a message size of up to 2 GB. To manage Amazon SQS messages with Amazon S3, use the Amazon SQS Extended Client Library for Java. Refer link)
3. Use SQS’s support for message partitioning and multi-part uploads on Amazon S3.
4. Use AWS EFS as a shared pool storage medium. Store filesystem pointers to the files on disk in the SQS message bodies.
Company ABCD has recently launched an online commerce site for bicycles on AWS. They have a “Product” DynamoDB table that stores details for each bicycle, such as, manufacturer, color, price, quantity and size to display in the online store. Due to customer demand, they want to include an image for each bicycle along with the existing details. Which approach below provides the least impact to provisioned throughput on the “Product” table?
1. Serialize the image and store it in multiple DynamoDB tables
2. Create an “Images” DynamoDB table to store the Image with a foreign key constraint to the “Product” table
3. Add an image data type to the “Product” table to store the images in binary format
4. Store the images in Amazon S3 and add an S3 URL pointer to the “Product” table item for each image

References

AWS Storage Options

AWS S3 Best Practices

May 20, 2016 ~ Last updated on : October 3, 2021 ~ jayendrapatil ~ 35 Comments

S3 Best Practices

Performance

Multiple Concurrent PUTs/GETs

S3 scales to support very high request rates. If the request rate grows steadily, S3 automatically partitions the buckets as needed to support higher request rates.
S3 can achieve at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per prefix in a bucket.
~~If the typical workload involves only occasional bursts of 100 requests per second and less than 800 requests per second, AWS scales and handle it.~~
If the typical workload involves a request rate for a bucket to more than 300 PUT/LIST/DELETE requests per second or more than 800 GET requests per second, it’s recommended to open a support case to prepare for the workload and avoid any temporary limits on your request rate.
S3 best practice guidelines can be applied only if you are routinely processing 100 or more requests per second
~~Workloads that include a mix of request types~~
- If the request workload is typically a mix of GET, PUT, DELETE, or GET Bucket (list objects), choosing appropriate key names for the objects ensures better performance by providing low-latency access to the S3 index
- ~~This behavior is driven by how S3 stores key names.~~
  - ~~S3 maintains an index of object key names in each AWS region.~~
  - ~~Object keys are stored lexicographically (UTF-8 binary ordering) across multiple partitions in the index i.e. S3 stores key names in alphabetical order.~~
  - ~~Object keys are stored in across multiple partitions in the index and the key name dictates which partition the key is stored in~~
  - Using a sequential prefix, such as timestamp or an alphabetical sequence, increases the likelihood that S3 will target a specific partition for a large number of keys, overwhelming the I/O capacity of the partition.
- ~~Introduce some randomness in the key name prefixes, the key names, and the I/O load, will be distributed across multiple index partitions.~~
- ~~It also ensures scalability regardless of the number of requests sent per second.~~

Transfer Acceleration

S3 Transfer Acceleration enables fast, easy, and secure transfers of files over long distances between the client and an S3 bucket.
Transfer Acceleration takes advantage of CloudFront’s globally distributed edge locations. As the data arrives at an edge location, data is routed to S3 over an optimized network path.

GET-intensive Workloads

CloudFront can be used for performance optimization and can help by
- distributing content with low latency and high data transfer rate.
- caching the content and thereby reducing the number of direct requests to S3
- providing multiple endpoints (Edge locations) for data availability
- available in two flavors as Web distribution or RTMP distribution
To fast data transport over long distances between a client and an S3 bucket, use S3 Transfer Acceleration. Transfer Acceleration uses the globally distributed edge locations in CloudFront to accelerate data transport over geographical distances

PUTs/GETs for Large Objects

AWS allows Parallelizing the PUTs/GETs request to improve the upload and download performance as well as the ability to recover in case it fails
For PUTs, Multipart upload can help improve the uploads by
- performing multiple uploads at the same time and maximizing network bandwidth utilization
- quick recovery from failures, as only the part that failed to upload, needs to be re-uploaded
- ability to pause and resume uploads
- begin an upload before the Object size is known
For GETs, the range HTTP header can help to improve the downloads by
- allowing the object to be retrieved in parts instead of the whole object
- quick recovery from failures, as only the part that failed to download needs to be retried.

List Operations

Object key names are stored lexicographically in S3 indexes, making it hard to sort and manipulate the contents of LIST
S3 maintains a single lexicographically sorted list of indexes
Build and maintain Secondary Index outside of S3 for e.g. DynamoDB or RDS to store, index and query objects metadata rather than performing operations on S3

Security

Use Versioning
- can be used to protect from unintended overwrites and deletions
- allows the ability to retrieve and restore deleted objects or rollback to previous versions
Enable additional security by configuring a bucket to enable MFA (Multi-Factor Authentication) to delete
Versioning does not prevent Bucket deletion and must be backed up as if accidentally or maliciously deleted the data is lost
Use Same Region Replication or Cross Region replication feature to backup data to a different region
When using VPC with S3, use VPC S3 endpoints as
- are horizontally scaled, redundant, and highly available VPC components
- help establish a private connection between VPC and S3 and the traffic never leaves the Amazon network

Refer blog post @ S3 Security Best Practices

Cost

Optimize S3 storage cost by selecting an appropriate storage class for objects
Configure appropriate lifecycle management rules to move objects to different storage classes and expire them

Tracking

Use Event Notifications to be notified for any put or delete request on the S3 objects
Use CloudTrail, which helps capture specific API calls made to S3 from the AWS account and delivers the log files to an S3 bucket
Use CloudWatch to monitor the Amazon S3 buckets, tracking metrics such as object counts and bytes stored, and configure appropriate actions

S3 Monitoring and Auditing Best Practices

Refer blog post @ S3 Monitoring and Auditing Best Practices

AWS Certification Exam Practice Questions

Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).

AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.

AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated

Open to further feedback, discussion and correction.

A media company produces new video files on-premises every day with a total size of around 100GB after compression. All files have a size of 1-2 GB and need to be uploaded to Amazon S3 every night in a fixed time window between 3am and 5am. Current upload takes almost 3 hours, although less than half of the available bandwidth is used. What step(s) would ensure that the file uploads are able to complete in the allotted time window?
1. Increase your network bandwidth to provide faster throughput to S3
2. Upload the files in parallel to S3 using multipart upload
3. Pack all files into a single archive, upload it to S3, then extract the files in AWS
4. Use AWS Import/Export to transfer the video files
You are designing a web application that stores static assets in an Amazon Simple Storage Service (S3) bucket. You expect this bucket to immediately receive over 150 PUT requests per second. What should you do to ensure optimal performance?
1. Use multi-part upload.
2. Add a random prefix to the key names.
3. Amazon S3 will automatically manage performance at this scale.
4. Use a predictable naming scheme, such as sequential numbers or date time sequences, in the key names
You have an application running on an Amazon Elastic Compute Cloud instance, that uploads 5 GB video objects to Amazon Simple Storage Service (S3). Video uploads are taking longer than expected, resulting in poor application performance. Which method will help improve performance of your application?
1. Enable enhanced networking
2. Use Amazon S3 multipart upload
3. Leveraging Amazon CloudFront, use the HTTP POST method to reduce latency.
4. Use Amazon Elastic Block Store Provisioned IOPs and use an Amazon EBS-optimized instance
Which of the following methods gives you protection against accidental loss of data stored in Amazon S3? (Choose 2)
1. Set bucket policies to restrict deletes, and also enable versioning
2. By default, versioning is enabled on a new bucket so you don’t have to worry about it (Not enabled by default)
3. Build a secondary index of your keys to protect the data (improves performance only)
4. Back up your bucket to a bucket owned by another AWS account for redundancy
A startup company hired you to help them build a mobile application that will ultimately store billions of image and videos in Amazon S3. The company is lean on funding, and wants to minimize operational costs, however, they have an aggressive marketing plan, and expect to double their current installation base every six months. Due to the nature of their business, they are expecting sudden and large increases to traffic to and from S3, and need to ensure that it can handle the performance needs of their application. What other information must you gather from this customer in order to determine whether S3 is the right option?
1. You must know how many customers that company has today, because this is critical in understanding what their customer base will be in two years. (No. of customers do not matter)
2. You must find out total number of requests per second at peak usage.
3. You must know the size of the individual objects being written to S3 in order to properly design the key namespace. (Size does not relate to the key namespace design but the count does)
4. In order to build the key namespace correctly, you must understand the total amount of storage needs for each S3 bucket. (S3 provided unlimited storage the key namespace design would depend on the number)
A document storage company is deploying their application to AWS and changing their business model to support both free tier and premium tier users. The premium tier users will be allowed to store up to 200GB of data and free tier customers will be allowed to store only 5GB. The customer expects that billions of files will be stored. All users need to be alerted when approaching 75 percent quota utilization and again at 90 percent quota use. To support the free tier and premium tier users, how should they architect their application?
1. The company should utilize an amazon simple workflow service activity worker that updates the users data counter in amazon dynamo DB. The activity worker will use simple email service to send an email if the counter increases above the appropriate thresholds.
2. The company should deploy an amazon relational data base service relational database with a store objects table that has a row for each stored object along with size of each object. The upload server will query the aggregate consumption of the user in questions (by first determining the files store by the user, and then querying the stored objects table for respective file sizes) and send an email via Amazon Simple Email Service if the thresholds are breached. (Good Approach to use RDS but with so many objects might not be a good option)
3. The company should write both the content length and the username of the files owner as S3 metadata for the object. They should then create a file watcher to iterate over each object and aggregate the size for each user and send a notification via Amazon Simple Queue Service to an emailing service if the storage threshold is exceeded. (List operations on S3 not feasible)
4. The company should create two separated amazon simple storage service buckets one for data storage for free tier users and another for data storage for premium tier users. An amazon simple workflow service activity worker will query all objects for a given user based on the bucket the data is stored in and aggregate storage. The activity worker will notify the user via Amazon Simple Notification Service when necessary (List operations on S3 not feasible as well as SNS does not address email requirement)
Your company host a social media website for storing and sharing documents. the web application allow users to upload large files while resuming and pausing the upload as needed. Currently, files are uploaded to your php front end backed by Elastic Load Balancing and an autoscaling fleet of amazon elastic compute cloud (EC2) instances that scale upon average of bytes received (NetworkIn) After a file has been uploaded. it is copied to amazon simple storage service(S3). Amazon Ec2 instances use an AWS Identity and Access Management (AMI) role that allows Amazon s3 uploads. Over the last six months, your user base and scale have increased significantly, forcing you to increase the auto scaling groups Max parameter a few times. Your CFO is concerned about the rising costs and has asked you to adjust the architecture where needed to better optimize costs. Which architecture change could you introduce to reduce cost and still keep your web application secure and scalable?
1. Replace the Autoscaling launch Configuration to include c3.8xlarge instances; those instances can potentially yield a network throughput of 10gbps. (no info of current size and might increase cost)
2. Re-architect your ingest pattern, have the app authenticate against your identity provider as a broker fetching temporary AWS credentials from AWS Secure token service (GetFederation Token). Securely pass the credentials and s3 endpoint/prefix to your app. Implement client-side logic to directly upload the file to amazon s3 using the given credentials and S3 Prefix. (will not provide the ability to handle pause and restarts)
3. Re-architect your ingest pattern, and move your web application instances into a VPC public subnet. Attach a public IP address for each EC2 instance (using the auto scaling launch configuration settings). Use Amazon Route 53 round robin records set and http health check to DNS load balance the app request this approach will significantly reduce the cost by bypassing elastic load balancing. (ELB is not the bottleneck)
4. Re-architect your ingest pattern, have the app authenticate against your identity provider as a broker fetching temporary AWS credentials from AWS Secure token service (GetFederation Token). Securely pass the credentials and s3 endpoint/prefix to your app. Implement client-side logic that used the S3 multipart upload API to directly upload the file to Amazon s3 using the given credentials and s3 Prefix. (multipart allows one to start uploading directly to S3 before the actual size is known or complete data is downloaded)
If an application is storing hourly log files from thousands of instances from a high traffic web site, which naming scheme would give optimal performance on S3?
1. Sequential
2. instanceID_log-HH-DD-MM-YYYY
3. instanceID_log-YYYY-MM-DD-HH
4. HH-DD-MM-YYYY-log_instanceID (HH will give some randomness to start with instead of instaneId where the first characters would be i-)
5. YYYY-MM-DD-HH-log_instanceID

Reference

S3_Optimizing_Performance

AWS EC2 – Elastic Cloud Compute

May 1, 2016 ~ Last updated on : February 6, 2023 ~ jayendrapatil ~ 9 Comments

Elastic Cloud Compute – EC2

Elastic Compute Cloud – EC2 provides scalable computing capacity in AWS
Elastic Compute Cloud – EC2
- eliminates the need to invest in hardware upfront, so applications can be developed and deployed faster.
- can be used to launch as many or as few virtual servers as you need, configure security and networking, and manage storage.
- enables you to scale up or down to handle changes in requirements or spikes in popularity, reducing the need to forecast traffic.

EC2 features

EC2 instances – Virtual computing environments
Amazon Machine Images (AMIs) – Preconfigured templates for the instances that package the bits needed for a server (including the operating system and additional software)
Instance types – Various configurations of CPU, memory, storage, and networking capacity for the instances
Key Pairs – Secure login information for the instances (AWS stores the public key, and you store the private key in a secure place)
Instance Store Volumes – Storage volumes for temporary data that are deleted when you stop or terminate your instance, known as
EBS Volumes – Persistent storage volumes for the data using Elastic Block Store (EBS)
Regions and Availability Zones – Multiple physical locations for the resources, such as instances and EBS volumes
Security Groups – A firewall that enables you to specify the protocols, ports, and source IP ranges that can reach the instances
Elastic IP addresses – Static IP addresses for dynamic cloud computing
Tags – Metadata can be created and assigned to EC2 resources

Accessing EC2

Amazon EC2 console
- Amazon EC2 console is the web-based user interface that can be accessed from the AWS management console
AWS Command line Interface (CLI)
- Provides commands for a broad set of AWS products, and is supported on Windows, Mac, and Linux.
Amazon EC2 Command Line Interface (CLI) tools
- Provides commands for Amazon EC2, Amazon EBS, and Amazon VPC, and is supported on Windows, Mac, and Linux
AWS Tools for Windows Powershell
- Provides commands for a broad set of AWS products for those who script in the PowerShell environment
AWS Query API
- Query API allows for requests are HTTP or HTTPS requests that use the HTTP verbs GET or POST and a Query parameter named Action
AWS SDK libraries
- AWS provides libraries in various languages which provide basic functions that automate tasks such as cryptographically signing your requests, retrying requests, and handling error responses

Additional Reading

AWS Certification Exam Practice Questions

Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).

AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.

AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated

Open to further feedback, discussion and correction.

What are the Amazon EC2 API tools?
1. They don’t exist. The Amazon EC2 AMI tools, instead, are used to manage permissions.
2. Command-line tools to the Amazon EC2 web service
3. They are a set of graphical tools to manage EC2 instances.
4. They don’t exist. The Amazon API tools are a client interface to Amazon Web Services.
When a user is launching an instance with EC2, which of the below mentioned options is not available during the instance launch console for a key pair?
1. Proceed without the key pair
2. Upload a new key pair
3. Select an existing key pair
4. Create a new key pair

References

AWS_EC2

AWS EC2 Security

May 1, 2016 ~ Last updated on : February 6, 2023 ~ jayendrapatil ~ 17 Comments

AWS EC2 Security

IAM helps control whether users in the organization can perform a task using specific EC2 API actions and whether they can use specific AWS resources.
Use IAM roles to prevent the need to share as well as manage, and rotate the security credentials that the applications use.
Security groups act as a virtual firewall that controls the traffic to the EC2 instances. They can help specify rules that control the inbound traffic that’s allowed to reach the instances and the outbound traffic that’s allowed to leave the instance.
Use AWS Systems Manager Session Manager to connect to the instance as it provides secure and auditable instance management without the need to open inbound ports, maintain bastion hosts, or manage SSH keys.
Use EC2 Instance Connect to connect to your instances using Secure Shell (SSH) without the need to share and manage SSH keys.
Use AWS Systems Manager Run Command to automate common administrative tasks instead of opening inbound SSH ports and managing SSH keys.
Use Systems Manager Patch Manager can be used to automate the process of patching, installing security-related updates for both the operating system and applications.

EC2 Key Pairs

EC2 uses public-key cryptography to encrypt & decrypt login information
Public-key cryptography uses a public key to encrypt a piece of data, such as a password, then the recipient uses the private key to decrypt the data.
Public and private keys are known as a key pair.
To log in to an EC2 instance, a key pair needs to be created and specified when the instance is launched, and the private key can be used to connect to the instance.
Linux instances have no password, and the key pair is used for ssh log in
For Windows instances, the key pair can be used to obtain the administrator password and then log in using RDP
EC2 stores the public key only, and the private key resides with the user. EC2 doesn’t keep a copy of your private key
Public key content (on Linux instances) is placed in an entry within ~/.ssh/authorized_keys at boot time and enables the user to securely access the instance without passwords
Public key specified for an instance when launched is also available through its instance metadata http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key
EC2 Security Best Practice: Store the private keys in a secure place as anyone who possesses the private key can decrypt the login information
Also, if the private key is lost, there is no way to recover the same.
- For instance store, you cannot access the instance
- For EBS-backed Linux instances, access can be regained.
  - EBS-backed instance can be stopped, its root volume detached and attached to another instance as a data volume
  - Modify the authorized_keys file, move the volume back to the original instance, and restart the instance
Key pair associated with the instances can either be
- Generated by EC2
  - Keys that EC2 uses are 2048-bit SSH-2 RSA keys.
- Created separately (using third-party tools) and Imported into EC2
  - EC2 only accepts RSA keys and does not accept DSA keys
  - Supported lengths: 1024, 2048, and 4096
supports five thousand key pairs per region
Deleting a key pair only deletes the public key and does not impact the servers already launched with the key.
Use AWS Systems Manager Session Manager to connect to the instance as it provides secure and auditable instance management without the need to open inbound ports, maintain bastion hosts, or manage SSH keys.

EC2 Security Groups

An EC2 instance, when launched, can be associated with one or more security groups, which acts as a virtual firewall that controls the traffic to that instance
Security groups help specify rules that control the inbound traffic that’s allowed to reach the instances and the outbound traffic that’s allowed to leave the instance
Security groups are associated with network interfaces. Changing an instance’s security groups changes the security groups associated with the primary network interface (eth0)
An ENI can be associated with 5 security groups and with 50 60 rules per security group
Rules for a security group can be modified at any time; the new rules are automatically applied to all instances associated with the security group.
All the rules from all associated security groups are evaluated to decide where to allow traffic to an instance
Security Group features
- For the VPC default security group, it allows all inbound traffic from other instances associated with the default security group
- By default, VPC default security groups or newly created security groups allow all outbound traffic
- Security group rules are always permissive; deny rules can’t be created
- Rules can be added and removed any time.
- Any modification to the rules are automatically applied to the instances associated with the security group after a short period, depending on the connection tracking for the traffic
- Security groups are stateful — if you send a request from your instance, the response traffic for that request is allowed to flow in regardless of inbound security group rules. For VPC security groups, this also means that responses to allowed inbound traffic are allowed to flow out, regardless of outbound rules
- If multiple rules are defined for the same protocol and port, the Most permissive rule is applied for e.g. for multiple rules for tcp and port 22 for specific IP and Everyone, everyone is granted access being the most permissive rule

Connection Tracking

Security groups are Stateful and they use Connection tracking to track information about traffic to and from the instance.
This allows responses to inbound traffic to flow out of the instance regardless of outbound security group rules, and vice versa.
Connection Tracking is maintained only if there is no explicit Outbound rule for an Inbound request (and vice versa)
However, if there is an explicit Outbound rule for an Inbound request, the response traffic is allowed on the basis of the Outbound rule and not on the Tracking information
Any existing flow of traffic, that is tracked, is not interrupted even if the rules for the security groups are changed. To ensure traffic is immediately interrupted, use NACL as they are stateless and therefore do not allow automatic response traffic.
Also, If the instance (host A) initiates traffic to host B and uses a protocol other than TCP, UDP, or ICMP, the instance’s firewall only tracks the IP address and protocol number for the purpose of allowing response traffic from host B. If host B initiates traffic to your instance in a separate request within 600 seconds of the original request or response, your instance accepts it regardless of inbound security group rules, because it’s regarded as response traffic.
can be controlled by modifying the security group’s outbound rules to permit only certain types of outbound traffic or using NACL

IAM with EC2

IAM policy can be defined to allow or deny a user access to the EC2 resources and actions
EC2 partially supports resource-level permissions. For some EC2 API actions, you cannot specify which resource a user is allowed to work with for that action; instead, you have to allow users to work with all resources for that action
IAM allows to control only what actions a user can perform on the EC2 resources but cannot be used to grant access for users to be able to access or login to the instances

EC2 with IAM Role

EC2 instances can be launched with IAM roles so that the applications can securely make API requests from your instances,
IAM roles prevent the need to share as well as manage, rotate the security credentials that the applications use.
IAM role can be added to an existing running EC2 instance.
EC2 uses an instance profile as a container for an IAM role.
- Creation of an IAM role using the console, creates an instance profile automatically and gives it the same name as the role it corresponds to.
- When using the AWS CLI, API, or an AWS SDK to create a role, the role and instance profile needs to be created as separate actions, and they can be given different names.
To launch an instance with an IAM role, the name of its instance profile needs to be specified.
An application on the instance can retrieve the security credentials provided by the role from the instance metadata item http://169.254.169.254/latest/meta-data/iam/security-credentials/role-name.
Security credentials are temporary and are rotated automatically and new credentials are made available at least five minutes prior to the expiration of the old credentials.
Best Practice: Always launch EC2 instance with IAM role instead of hardcoded credentials

EC2 IAM Role S3 Access

EC2 Resiliency

EC2 offers the following features to support your data resiliency:
- Copying AMIs across Regions
- Copying EBS snapshots across Regions
- Automating EBS-backed AMIs using Data Lifecycle Manager
- Automating EBS snapshots using Data Lifecycle Manager
- Maintaining the health and availability of the fleet using EC2 Auto Scaling
- Distributing incoming traffic across multiple instances in a single AZ or multiple AZs using Elastic Load Balancing

AWS Certification Exam Practice Questions

Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).

AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.

AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated

Open to further feedback, discussion and correction.

You launch an Amazon EC2 instance without an assigned AWS identity and Access Management (IAM) role. Later, you decide that the instance should be running with an IAM role. Which action must you take in order to have a running Amazon EC2 instance with an IAM role assigned to it?
1. Create an image of the instance, and register the image with an IAM role assigned and an Amazon EBS volume mapping.
2. Create a new IAM role with the same permissions as an existing IAM role, and assign it to the running instance. (As per AWS latest enhancement, this is possible now)
3. Create an image of the instance, add a new IAM role with the same permissions as the desired IAM role, and deregister the image with the new role assigned.
4. Create an image of the instance, and use this image to launch a new instance with the desired IAM role assigned (This was correct before, as it was not possible to add an IAM role to an existing instance)
What does the following command do with respect to the Amazon EC2 security groups? ec2-revoke RevokeSecurityGroupIngress
1. Removes one or more security groups from a rule.
2. Removes one or more security groups from an Amazon EC2 instance.
3. Removes one or more rules from a security group
4. Removes a security group from our account.
Which of the following cannot be used in Amazon EC2 to control who has access to specific Amazon EC2 instances?
1. Security Groups
2. IAM System
3. SSH keys
4. Windows passwords
You must assign each server to at least _____ security group
1. 3
2. 2
3. 4
4. 1
A company is building software on AWS that requires access to various AWS services. Which configuration should be used to ensure that AWS credentials (i.e., Access Key ID/Secret Access Key combination) are not compromised?
1. Enable Multi-Factor Authentication for your AWS root account.
2. Assign an IAM role to the Amazon EC2 instance
3. Store the AWS Access Key ID/Secret Access Key combination in software comments.
4. Assign an IAM user to the Amazon EC2 Instance.
Which of the following items are required to allow an application deployed on an EC2 instance to write data to a DynamoDB table? Assume that no security keys are allowed to be stored on the EC2 instance. (Choose 2 answers)
1. Create an IAM Role that allows write access to the DynamoDB table
2. Add an IAM Role to a running EC2 instance. (As per AWS latest enhancement, this is possible now)
3. Create an IAM User that allows write access to the DynamoDB table.
4. Add an IAM User to a running EC2 instance.
5. Launch an EC2 Instance with the IAM Role included in the launch configuration (This was correct before, as it was not possible to add an IAM role to an existing instance)
You have an application running on an EC2 Instance, which will allow users to download files from a private S3 bucket using a pre-assigned URL. Before generating the URL the application should verify the existence of the file in S3. How should the application use AWS credentials to access the S3 bucket securely?
1. Use the AWS account access Keys the application retrieves the credentials from the source code of the application.
2. Create a IAM user for the application with permissions that allow list access to the S3 bucket launch the instance as the IAM user and retrieve the IAM user’s credentials from the EC2 instance user data.
3. Create an IAM role for EC2 that allows list access to objects in the S3 bucket. Launch the instance with the role, and retrieve the role’s credentials from the EC2 Instance metadata
4. Create an IAM user for the application with permissions that allow list access to the S3 bucket. The application retrieves the IAM user credentials from a temporary directory with permissions that allow read access only to the application user.
A user has created an application, which will be hosted on EC2. The application makes calls to DynamoDB to fetch certain data. The application is using the DynamoDB SDK to connect with from the EC2 instance. Which of the below mentioned statements is true with respect to the best practice for security in this scenario?
1. The user should attach an IAM role with DynamoDB access to the EC2 instance
2. The user should create an IAM user with DynamoDB access and use its credentials within the application to connect with DynamoDB
3. The user should create an IAM role, which has EC2 access so that it will allow deploying the application
4. The user should create an IAM user with DynamoDB and EC2 access. Attach the user with the application so that it does not use the root account credentials
Your application is leveraging IAM Roles for EC2 for accessing object stored in S3. Which two of the following IAM policies control access to you S3 objects.
1. An IAM trust policy allows the EC2 instance to assume an EC2 instance role.
2. An IAM access policy allows the EC2 role to access S3 objects
3. An IAM bucket policy allows the EC2 role to access S3 objects. (Bucket policy is defined with S3 and not with IAM)
4. An IAM trust policy allows applications running on the EC2 instance to assume as EC2 role (Trust policy allows EC2 instance to assume the role)
5. An IAM trust policy allows applications running on the EC2 instance to access S3 objects. (Applications can access S3 through EC2 assuming the role)
You have an application running on an EC2 Instance, which will allow users to download files from a private S3 bucket using a pre-assigned URL. Before generating the URL the application should verify the existence of the file in S3. How should the application use AWS credentials to access the S3 bucket securely?
1. Use the AWS account access Keys the application retrieves the credentials from the source code of the application.
2. Create a IAM user for the application with permissions that allow list access to the S3 bucket launch the instance as the IAM user and retrieve the IAM user’s credentials from the EC2 instance user data.
3. Create an IAM role for EC2 that allows list access to objects in the S3 bucket. Launch the instance with the role, and retrieve the role’s credentials from the EC2 Instance metadata
4. Create an IAM user for the application with permissions that allow list access to the S3 bucket. The application retrieves the IAM user credentials from a temporary directory with permissions that allow read access only to the application user.

AWS EC2 Instance Lifecycle

April 30, 2016 ~ Last updated on : September 9, 2021 ~ jayendrapatil ~ 18 Comments

EC2 Instance Lifecycle Overview

EC2 instance lifecycle determines how an EC2 instance transitions through different states from the moment it is launched to its termination

EC2 Instance Lifecycle

Instance Launch

Pending
- When the instance is first launched is enters into the pending state
Running
- After the instance is launched, it enters into the running state
- Charges are incurred for each second, with a one-minute minimum, that the instance running is running, even if the instance remains idle

Instance Start & Stop (EBS-backed instances only)

Only an EBS-backed instance can be stopped and started.
Instance store-backed instance cannot be stopped and started.
An instance can be stopped & started in case the instance fails a status check or is not running as expected
Stop
- After the instance is stopped, it enters in stopping state and then to stopped state.
- Charges are only incurred for the EBS storage and not for the instance hourly charge or data transfer.
- While the instance is stopped, its root volume can be treated like any other volume, and modified for e.g. repair file system problems or update software or change the instance type, user data, EBS optimization attributes, etc
- Volume can be detached from the stopped instance, and attached to a running instance, modified, detached from the running instance, and then reattached to the stopped instance. It should be reattached using the storage device name that’s specified as the root device in the block device mapping for the instance.
Start
- When the instance is started, it enters into pending state and then into running
- An instance, when stopped and started, is launched on a new host
- Any data on an instance store volume (not root volume) would be lost while data on the EBS volume persists
EC2 instance retains its private IP address as well as the Elastic IP address.
If the instance has an IPv6 address, it retains its IPv6 address.
However, the public IP address, if assigned instead of the Elastic IP address, would be released
For each transition of an instance from stopped to running, charges per second are incurred when the instance is running, with a minimum of one minute every time the instance is started

Instance Hibernate

Instance hibernation signals the operating system to perform hibernation (suspend-to-disk), which saves the contents from the instance memory (RAM) to the EBS root volume
Instance’s EBS root volume and any attached EBS data volumes are persisted, including the saved contents of the RAM.
Any EC2 instance store volumes remain attached to the instance, but the data on the instance store volumes is lost.
When the instance is restarted, the EBS root volume is restored to its previous state and the RAM contents are reloaded. Previously attached data volumes are reattached and the instance retains its instance ID.
After the instance is hibernated, it enters in stopping state and then to stopped state.
When the instance is restarted
- It enters the pending state and the instance is moved to a new host computer (though in some cases, it remains on the current host).
- EBS root volume is restored to its previous state
- RAM contents are reloaded
- Processes that were previously running on the instance are resumed
- Previously attached data volumes are reattached and the instance retains its instance ID
- Instance retains private IPv4 addresses and any IPv6 addresses
- Instance retains its Elastic IP address
- Instance releases its Public IPv4 address and would get a new one
Hibernation prerequisites
- Supported instance families – C3, C4, C5, M3, M4, M5, R3, R4, R5, & T2
- Instance RAM size – must be less than 150 GB.
- Instance size – not supported for bare metal instances.
- Supported AMIs must be an HVM AMI that supports hibernation
- Root volume type – must be EBS volume and not instance store
- EBS root volume size – must be large enough to store the RAM contents
- EBS root volume MUST be encrypted to ensure the protection of sensitive content that is in memory at the time of hibernation
- Enable hibernation at launch, as changing it is not supported on an existing instance
- Purchasing options – Only On-Demand Instances and Reserved Instances supported
Limitations or Unsupported Actions
- Changing the instance type or size of a hibernated instance
- Creating snapshots or AMIs from hibernated instances or instances for which hibernation is enabled
- the data on any instance store volumes is lost
- can’t hibernate an instance that has more than 150 GB of RAM.
- can’t hibernate an instance that is in an Auto Scaling group or used by ECS. If the instance is in an Auto Scaling group is hibernated, the EC2 Auto Scaling service marks the stopped instance as unhealthy, and may terminate it and launch a replacement instance.
- An instance cannot be hibernated for more than 60 days.

Instance Reboot

Both EBS-backed and Instance store-backed instances can be rebooted
An instance remains on the same host computer and maintains its public DNS name, private IP address
Data on the EBS and Instance store volume is also retained
AWS recommends using EC2 to reboot the instance instead of running the operating system reboot command from the instance as it performs a hard reboot if the instance does not cleanly shut down within four minutes also creates an API record in CloudTrail if enabled.

Instance Retirement

An instance is scheduled to be retired when AWS detects an irreparable failure of the underlying hardware hosting the instance.
When an instance reaches its scheduled retirement date, it is stopped or terminated by AWS.
If the instance root device is an EBS volume, the instance is stopped and can be started again at any time.
If the instance root device is an instance store volume, the instance is terminated, and cannot be used again.

Instance Termination

An instance can be terminated, and it enters into the shutting-down, and then the terminated state
After an instance is terminated, it can’t be connected and no charges are incurred
Instance Shutdown behavior
- Each EBS-backed instance supports the InstanceInitiatedShutdownBehavior attribute which determines whether the instance would be stopped or terminated when a shutdown command is initiated from the instance itself for e.g. shutdown, halt or poweroff command in linux
- Default behavior for the instance to be stopped.
- A shutdown command for an Instance store-backed instance will always terminate the instance
Termination protection
- Termination protection ( DisableApiTermination attribute) can be enabled on the instance to prevent it from being accidentally terminated
- DisableApiTermination from the Console, CLI or API.
- Instance can be terminated through EC2 CLI.
- Termination protection does not work for instances when
  - part of an Autoscaling group
  - launched as Spot instances
  - terminating an instance by initiating shutdown from the instance
Data persistence
- EBS volume has a DeleteOnTermination attribute which determines whether the volumes would be persisted or deleted when an instance they are associated with are terminated
- Data on Instance store volume data does not persist
- Default is to delete the root device volume and preserve any other EBS volumes. i.e.
  - Data on EBS root volumes have the DeleteOnTermination flag set to true and would be deleted by default
  - Additional EBS volumes attached have the DeleteOnTermination flag set to false are not deleted but just detached from the instance

EC2 Instance Lifecycle States and Billing

AWS Certification Exam Practice Questions

Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).

AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.

AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated

Open to further feedback, discussion and correction.

What does Amazon EC2 provide?
1. Virtual servers in the Cloud
2. A platform to run code (Java, PHP, Python), paying on an hourly basis.
3. Computer Clusters in the Cloud.
4. Physical servers, remotely managed by the customer.
A user has enabled termination protection on an EC2 instance. The user has also set Instance initiated shutdown behavior to terminate. When the user shuts down the instance from the OS, what will happen?
1. The OS will shutdown but the instance will not be terminated due to protection
2. It will terminate the instance
3. It will not allow the user to shutdown the instance from the OS
4. It is not possible to set the termination protection when an Instance initiated shutdown is set to Terminate
A user has launched an EC2 instance and deployed a production application in it. The user wants to prohibit any mistakes from the production team to avoid accidental termination. How can the user achieve this?
1. The user can the set DisableApiTermination attribute to avoid accidental termination
2. It is not possible to avoid accidental termination
3. The user can set the Deletion termination flag to avoid accidental termination
4. The user can set the InstanceInitiatedShutdownBehavior flag to avoid accidental termination
You have been doing a lot of testing of your VPC Network by deliberately failing EC2 instances to test whether instances are failing over properly. Your customer who will be paying the AWS bill for all this asks you if he being charged for all these instances. You try to explain to him how the billing works on EC2 instances to the best of your knowledge. What would be an appropriate response to give to the customer in regards to this?
1. Billing commences when Amazon EC2 AMI instance is completely up and billing ends as soon as the instance starts to shutdown.
2. Billing commences when Amazon EC2 initiates the boot sequence of an AMI instance and billing ends when the instance shuts down.
3. Billing only commences only after 1 hour of uptime and billing ends when the instance terminates.
4. Billing commences when Amazon EC2 initiates the boot sequence of an AMI instance and billing ends as soon as the instance starts to shutdown.

References

EC2 User Guide – Instance lifecycle

Storage Options Whitepaper

Overview

Amazon S3 & Amazon Glacier

Amazon Elastic Block Store (EBS) & Instance Store Volumes

Amazon RDS, DynamoDB & Database on EC2

Amazon SQS & Redshift

Amazon CloudFront & Elasticache

Amazon Storage Gateway & Import/Export

AWS Certification Exam Practice Questions

References

SQS

Ideal Usage patterns

Anti-Patterns

Performance

Durability & Availability

Cost Model

Scalability & Elasticity

Amazon Redshift

Ideal Usage Pattern

Anti-Pattern

Performance

Durability & Availability

Cost Model

Scalability & Elasticity

Ideal Usage Patterns

Anti-Pattern

Performance

Durability & Availability

Cost Model

Scalability & Elasticity

Ideal Usage Patterns

Anti-Patterns

Performance

Durability & Availability

Cost Model

Scalability & Elasticity

AWS Storage Options Whitepaper cont.

AWS Storage Gateway

Ideal Usage Patterns

Anti-Patterns

Performance

Durability and Availability

Cost Model

Scalability and Elasticity

Interfaces

AWS Import/Export (Upgraded to Snowball)

Ideal Usage Patterns

Anti-Patterns

Performance

Durability and Availability

Cost Model

Scalability and Elasticity

Interfaces

AWS Certification Exam Practice Questions

AWS Storage Options Whitepaper with RDS, DynamoDB & Database on EC2 Cont.

Amazon RDS

Ideal Usage Patterns

Anti-Patterns

Performance

Durability and Availability

Cost Model

Scalability and Elasticity

Interfaces

Amazon DynamoDB

Ideal Usage Patterns

Anti-Patterns

Performance

Durability and Availability

Cost Model

Scalability and Elasticity

Interfaces

Databases on EC2

Ideal Usage Patterns

Anti-Patterns

Performance

Durability & Availability

Cost Model

Scalability & Elasticity

AWS Certification Exam Practice Questions

Amazon S3