S3 Replication enables automatic, asynchronous copying of objects across S3 buckets in the same or different AWS regions.
S3 Cross-Region Replication – CRR is used to copy objects across S3 buckets in different AWS Regions.
S3 Same-Region Replication – SRR is used to copy objects across S3 buckets in the same AWS Regions.
S3 Replication helps to
Replicate objects while retaining metadata
Replicate objects into different storage classes
Maintain object copies under different ownership
Keep objects stored over multiple AWS Regions
Replicate objects within 15 minutes
S3 can replicate all or a subset of objects with specific key name prefixes
S3 encrypts all data in transit across AWS regions using SSL
Object replicas in the destination bucket are exact replicas of the objects in the source bucket with the same key names and the same metadata.
Objects may be replicated to a single destination bucket or multiple destination buckets.
Cross-Region Replication can be useful for the following scenarios:-
Compliance requirement to have data backed up across regions
Minimize latency to allow users across geography to access objects
Operational reasons compute clusters in two different regions that analyze the same set of objects
Same-Region Replication can be useful for the following scenarios:-
Aggregate logs into a single bucket
Configure live replication between production and test accounts
Abide by data sovereignty laws to store multiple copies
S3 Replication Requirements
source and destination buckets must be versioning-enabled
for CRR, the source and destination buckets must be in different AWS regions.
S3 must have permission to replicate objects from that source bucket to the destination bucket on your behalf.
If the source bucket owner also owns the object, the bucket owner has full permission to replicate the object. If not, the source bucket owner must have permission for the S3 actions s3:GetObjectVersionand s3:GetObjectVersionACLto read the object and object ACL
Setting up cross-region replication in a cross-account scenario (where the source and destination buckets are owned by different AWS accounts), the source bucket owner must have permission to replicate objects in the destination bucket.
if the source bucket has S3 Object Lock enabled, the destination buckets must also have S3 Object Lock enabled.
destination buckets cannot be configured as Requester Pays buckets
S3 Replication – Replicated & Not Replicated
Only new objects created after you add a replication configuration are replicated. S3 does NOT retroactively replicate objects that existed before you added replication configuration.
Objects encrypted using customer provided keys (SSE-C), objects encrypted at rest under an S3 managed key (SSE-S3) or a KMS key stored in AWS Key Management Service (SSE-KMS).
S3 replicates only objects in the source bucket for which the bucket owner has permission to read objects and read ACLs
Any object ACL updates are replicated, although there can be some delay before S3 can bring the two in sync. This applies only to objects created after you add a replication configuration to the bucket.
S3 does NOT replicate objects in the source bucket for which the bucket owner does not have permission.
Updates to bucket-level S3 subresources are NOT replicated, allowing different bucket configurations on the source and destination buckets
Only customer actions are replicated & actions performed by lifecycle configuration are NOT replicated
Replication chaining is NOT allowed, Objects in the source bucket that are replicas, created by another replication, are NOT replicated.
S3 does NOT replicate the delete marker by default. However, you can add delete marker replication to non-tag-based rules to override it.
S3 does NOT replicate deletion by object version ID. This protects data from malicious deletions.
AWS Certification Exam Practice Questions
Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
Open to further feedback, discussion and correction.
S3 Subresources provides support to store, and manage the bucket configuration information.
S3 subresources only exist in the context of a specific bucket or object
S3 subresources are associated with buckets and objects.
S3 Subresources are subordinates to objects; i.e. they do not exist on their own, they are always associated with some other entity, such as an object or a bucket.
S3 supports various options to configure a bucket for e.g., the bucket can be configured for website hosting, configuration added to manage the lifecycle of objects in the bucket, and to log all access to the bucket.
S3 can be used for Static Website hosting with Client-side scripts.
S3 does not support server-side scripting.
S3, in conjunction with Route 53, supports hosting a website at the root domain which can point to the S3 website endpoint
S3 website endpoints do not support HTTPS or access points
For S3 website hosting the content should be made publicly readable which can be provided using a bucket policy or an ACL on an object.
Users can configure the index, and error document as well as configure the conditional routing of an object name
Bucket policy applies only to objects owned by the bucket owner. If the bucket contains objects not owned by the bucket owner, then public READ permission on those objects should be granted using the object ACL.
Requester Pays buckets or DevPay buckets do not allow access through the website endpoint. Any request to such a bucket will receive a 403 -Access Denied response
All browsers implement the Same-Origin policy, for security reasons, where the web page from a domain can only request resources from the same domain.
CORS allows client web applications loaded in one domain access to the restricted resources to be requested from another domain.
With CORS support, S3 allows cross-origin access to S3 resources
CORS configuration rules identify the origins allowed to access the bucket, the operations (HTTP methods) that would be supported for each origin, and other operation-specific information.
S3 Access Logs
S3 Access Logs enable tracking access requests to an S3 bucket.
S3 Access logs are disabled by default.
Each access log record provides details about a single access request, such as the requester, bucket name, request time, request action, response status, and error code, etc.
Access log information can be useful in security and access audits and also help learn about the customer base and understand the S3 bill.
S3 periodically collects access log records, consolidates the records in log files, and then uploads log files to a target bucket as log objects.
Logging can be enabled on multiple source buckets with the same target bucket which will have access logs for all those source buckets, but each log object will report access log records for a specific source bucket.
Source and target buckets should be in the same region.
Source and target buckets should be different to avoid an infinite loop of logs issue.
Target bucket can be encrypted using SSS-S3 default encryption. However, Default encryption with AWS KMS keys (SSE-KMS) is not supported.
S3 uses a special log delivery account to write server access logs.
AWS recommends updating the bucket policy on the target bucket to grant access to the logging service principal (logging.s3.amazonaws.com) for access log delivery.
Access for access log delivery can also be granted to the S3 log delivery group through the bucket ACL. Granting access to the S3 log delivery group using your bucket ACL is not recommended.
Access log records are delivered on a best-effort basis. The completeness and timeliness of server logging is not guaranteed i.e. log record for a particular request might be delivered long after the request was actually processed, or it might not be delivered at all.
S3 Access Logs can be analyzed using data analysis tools or Athena.
Tagging
S3 provides the tagging subresource to store and manage tags on a bucket
Cost allocation tags can be added to the bucket to categorize and track AWS costs.
AWS can generate a cost allocation report with usage and costs aggregated by the tags applied to the buckets.
Location
AWS region needs to be specified during bucket creation and it cannot be changed.
S3 stores this information in the location subresource and provides an API for retrieving this information
Event Notifications
S3 notification feature enables notifications to be triggered when certain events happen in the bucket.
Notifications are enabled at the Bucket level
Notifications can be configured to be filtered by the prefix and suffix of the key name of objects. However, filtering rules cannot be defined with overlapping prefixes, overlapping suffixes, or prefix and suffix overlapping
S3 can publish the following events
New Object created events
Can be enabled for PUT, POST, or COPY operations
You will not receive event notifications from failed operations
Object Removal events
Can public delete events for object deletion, version object deletion or insertion of delete marker
You will not receive event notifications from automatic deletes from lifecycle policies or from failed operations.
Restore object events
restoration of objects archived to the S3 Glacier storage classes
Reduced Redundancy Storage (RRS) object lost events
Can be used to reproduce/recreate the Object
Replication events
for replication configurations that have S3 replication metrics or S3 Replication Time Control (S3 RTC) enabled
S3 can publish events to the following destination
For S3 to be able to publish events to the destination, the S3 principal should be granted the necessary permissions
S3 event notifications are designed to be delivered at least once. Typically, event notifications are delivered in seconds but can sometimes take a minute or longer.
S3 Replication enables automatic, asynchronous copying of objects across S3 buckets in the same or different AWS regions.
S3 Cross-Region Replication – CRR is used to copy objects across S3 buckets in different AWS Regions.
S3 Same-Region Replication – SRR is used to copy objects across S3 buckets in the same AWS Regions.
S3 Replication helps to
Replicate objects while retaining metadata
Replicate objects into different storage classes
Maintain object copies under different ownership
Keep objects stored over multiple AWS Regions
Replicate objects within 15 minutes
S3 can replicate all or a subset of objects with specific key name prefixes
S3 encrypts all data in transit across AWS regions using SSL
Object replicas in the destination bucket are exact replicas of the objects in the source bucket with the same key names and the same metadata.
Objects may be replicated to a single destination bucket or multiple destination buckets.
Cross-Region Replication can be useful for the following scenarios:-
Compliance requirement to have data backed up across regions
Minimize latency to allow users across geography to access objects
Operational reasons compute clusters in two different regions that analyze the same set of objects
Same-Region Replication can be useful for the following scenarios:-
Aggregate logs into a single bucket
Configure live replication between production and test accounts
Abide by data sovereignty laws to store multiple copies
Replication Requirements
source and destination buckets must be versioning-enabled
for CRR, the source and destination buckets must be in different AWS regions.
S3 must have permission to replicate objects from that source bucket to the destination bucket on your behalf.
If the source bucket owner also owns the object, the bucket owner has full permission to replicate the object. If not, the source bucket owner must have permission for the S3 actions s3:GetObjectVersionand s3:GetObjectVersionACLto read the object and object ACL
Setting up cross-region replication in a cross-account scenario (where the source and destination buckets are owned by different AWS accounts), the source bucket owner must have permission to replicate objects in the destination bucket.
if the source bucket has S3 Object Lock enabled, the destination buckets must also have S3 Object Lock enabled.
destination buckets cannot be configured as Requester Pays buckets
Replicated & Not Replicated
Only new objects created after you add a replication configuration are replicated. S3 does NOT retroactively replicate objects that existed before you added replication configuration.
Objects encrypted using customer provided keys (SSE-C), objects encrypted at rest under an S3 managed key (SSE-S3) or a KMS key stored in AWS Key Management Service (SSE-KMS).
S3 replicates only objects in the source bucket for which the bucket owner has permission to read objects and read ACLs
Any object ACL updates are replicated, although there can be some delay before S3 can bring the two in sync. This applies only to objects created after you add a replication configuration to the bucket.
S3 does NOT replicate objects in the source bucket for which the bucket owner does not have permission.
Updates to bucket-level S3 subresources are NOT replicated, allowing different bucket configurations on the source and destination buckets
Only customer actions are replicated & actions performed by lifecycle configuration are NOT replicated
Replication chaining is NOT allowed, Objects in the source bucket that are replicas, created by another replication, are NOT replicated.
S3 does NOT replicate the delete marker by default. However, you can add delete marker replication to non-tag-based rules to override it.
S3 does NOT replicate deletion by object version ID. This protects data from malicious deletions.
S3 Inventory
S3 Inventory helps manage the storage and can be used to audit and report on the replication and encryption status of the objects for business, compliance, and regulatory needs.
S3 inventory provides a scheduled alternative to the S3 synchronous List API operation.
S3 inventory provides CSV, ORC, or Apache Parquet output files that list the objects and their corresponding metadata on a daily or weekly basis for an S3 bucket or a shared prefix.
Requester Pays
By default, buckets are owned by the AWS account that created it (the bucket owner) and the AWS account pays for storage costs, downloads, and data transfer charges associated with the bucket.
Using Requester Pays subresource:-
Bucket owner specifies that the requester requesting the download will be charged for the download
However, the bucket owner still pays the storage costs
Enabling Requester Pays on a bucket
disables anonymous access to that bucket
does not support BitTorrent
does not support SOAP requests
cannot be enabled for end-user logging bucket
Torrent
Default distribution mechanism for S3 data is via client/server download
Bucket owner bears the cost of Storage as well as the request and transfer charges which can increase linearly for a popular object
S3 also supports the BitTorrent protocol
BitTorrent is an open-source Internet distribution protocol
BitTorrent addresses this problem by recruiting the very clients that are downloading the object as distributors themselves
S3 bandwidth rates are inexpensive, but BitTorrent allows developers to further save on bandwidth costs for a popular piece of data by letting users download from Amazon and other users simultaneously
Benefit for a publisher is that for large, popular files the amount of data actually supplied by S3 can be substantially lower than what it would have been serving the same clients via client/server download
Any object in S3 that is publicly available and can be read anonymously can be downloaded via BitTorrent
Torrent file can be retrieved for any publicly available object by simply adding a “?torrent” query string parameter at the end of the REST GET request for the object
Generating the .torrent for an object takes time proportional to the size of that object, so its recommended to make a first torrent request yourself to generate the file so that subsequent requests are faster
Torrent is enabled only for objects that are less than 5 GB in size.
Torrent subresource can only be retrieved, and cannot be created, updated, or deleted
Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
Open to further feedback, discussion and correction.
An organization’s security policy requires multiple copies of all critical data to be replicated across at least a primary and backup data center. The organization has decided to store some critical data on Amazon S3. Which option should you implement to ensure this requirement is met?
Use the S3 copy API to replicate data between two S3 buckets in different regions
You do not need to implement anything since S3 data is automatically replicated between regions
Use the S3 copy API to replicate data between two S3 buckets in different facilities within an AWS Region
You do not need to implement anything since S3 data is automatically replicated between multiple facilities within an AWS Region
A customer wants to track access to their Amazon Simple Storage Service (S3) buckets and also use this information for their internal security and access audits. Which of the following will meet the Customer requirement?
Enable AWS CloudTrail to audit all Amazon S3 bucket access.
Enable server access logging for all required Amazon S3 buckets
Enable the Requester Pays option to track access via AWS Billing
Enable Amazon S3 event notifications for Put and Post.
A user is enabling a static website hosting on an S3 bucket. Which of the below mentioned parameters cannot be configured by the user?
Error document
Conditional error on object name
Index document
Conditional redirection on object name
Company ABCD is running their corporate website on Amazon S3 accessed from http//www.companyabcd.com. Their marketing team has published new web fonts to a separate S3 bucket accessed by the S3 endpoint: https://s3-us-west1.amazonaws.com/abcdfonts. While testing the new web fonts, Company ABCD recognized the web fonts are being blocked by the browser. What should Company ABCD do to prevent the web fonts from being blocked by the browser?
Enable versioning on the abcdfonts bucket for each web font
Create a policy on the abcdfonts bucket to enable access to everyone
Add the Content-MD5 header to the request for webfonts in the abcdfonts bucket from the website
Configure the abcdfonts bucket to allow cross-origin requests by creating a CORS configuration
Company ABCD is currently hosting their corporate site in an Amazon S3 bucket with Static Website Hosting enabled. Currently, when visitors go to http://www.companyabcd.com the index.html page is returned. Company C now would like a new page welcome.html to be returned when a visitor enters http://www.companyabcd.com in the browser. Which of the following steps will allow Company ABCD to meet this requirement? Choose 2 answers.
Upload an html page named welcome.html to their S3 bucket
S3 scales to support very high request rates. If the request rate grows steadily, S3 automatically partitions the buckets as needed to support higher request rates.
S3 can achieve at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per prefix in a bucket.
If the typical workload involves only occasional bursts of 100 requests per second and less than 800 requests per second, AWS scales and handle it.
If the typical workload involves a request rate for a bucket to more than 300 PUT/LIST/DELETE requests per second or more than 800 GET requests per second, it’s recommended to open a support case to prepare for the workload and avoid any temporary limits on your request rate.
S3 best practice guidelines can be applied only if you are routinely processing 100 or more requests per second
Workloads that include a mix of request types
If the request workload is typically a mix of GET, PUT, DELETE, or GET Bucket (list objects), choosing appropriate key names for the objects ensures better performance by providing low-latency access to the S3 index
This behavior is driven by how S3 stores key names.
S3 maintains an index of object key names in each AWS region.
Object keys are stored lexicographically (UTF-8 binary ordering) across multiple partitions in the index i.e. S3 stores key names in alphabetical order.
Object keys are stored in across multiple partitions in the index and the key name dictates which partition the key is stored in
Using a sequential prefix, such as timestamp or an alphabetical sequence, increases the likelihood that S3 will target a specific partition for a large number of keys, overwhelming the I/O capacity of the partition.
Introduce some randomness in the key name prefixes, the key names, and the I/O load, will be distributed across multiple index partitions.
It also ensures scalability regardless of the number of requests sent per second.
Transfer Acceleration
S3 Transfer Acceleration enables fast, easy, and secure transfers of files over long distances between the client and an S3 bucket.
Transfer Acceleration takes advantage of CloudFront’s globally distributed edge locations. As the data arrives at an edge location, data is routed to S3 over an optimized network path.
GET-intensive Workloads
CloudFront can be used for performance optimization and can help by
distributing content with low latency and high data transfer rate.
caching the content and thereby reducing the number of direct requests to S3
providing multiple endpoints (Edge locations) for data availability
available in two flavors as Web distribution or RTMP distribution
To fast data transport over long distances between a client and an S3 bucket, use S3 Transfer Acceleration. Transfer Acceleration uses the globally distributed edge locations in CloudFront to accelerate data transport over geographical distances
PUTs/GETs for Large Objects
AWS allows Parallelizing the PUTs/GETs request to improve the upload and download performance as well as the ability to recover in case it fails
performing multiple uploads at the same time and maximizing network bandwidth utilization
quick recovery from failures, as only the part that failed to upload, needs to be re-uploaded
ability to pause and resume uploads
begin an upload before the Object size is known
For GETs, the range HTTP header can help to improve the downloads by
allowing the object to be retrieved in parts instead of the whole object
quick recovery from failures, as only the part that failed to download needs to be retried.
List Operations
Object key names are stored lexicographically in S3 indexes, making it hard to sort and manipulate the contents of LIST
S3 maintains a single lexicographically sorted list of indexes
Build and maintain Secondary Index outside of S3 for e.g. DynamoDB or RDS to store, index and query objects metadata rather than performing operations on S3
Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
Open to further feedback, discussion and correction.
A media company produces new video files on-premises every day with a total size of around 100GB after compression. All files have a size of 1-2 GB and need to be uploaded to Amazon S3 every night in a fixed time window between 3am and 5am. Current upload takes almost 3 hours, although less than half of the available bandwidth is used. What step(s) would ensure that the file uploads are able to complete in the allotted time window?
Increase your network bandwidth to provide faster throughput to S3
Upload the files in parallel to S3 using multipart upload
Pack all files into a single archive, upload it to S3, then extract the files in AWS
Use AWS Import/Export to transfer the video files
You are designing a web application that stores static assets in an Amazon Simple Storage Service (S3) bucket. You expect this bucket to immediately receive over 150 PUT requests per second. What should you do to ensure optimal performance?
Use multi-part upload.
Add a random prefix to the key names.
Amazon S3 will automatically manage performance at this scale.
Use a predictable naming scheme, such as sequential numbers or date time sequences, in the key names
You have an application running on an Amazon Elastic Compute Cloud instance, that uploads 5 GB video objects to Amazon Simple Storage Service (S3). Video uploads are taking longer than expected, resulting in poor application performance. Which method will help improve performance of your application?
Enable enhanced networking
Use Amazon S3 multipart upload
Leveraging Amazon CloudFront, use the HTTP POST method to reduce latency.
Use Amazon Elastic Block Store Provisioned IOPs and use an Amazon EBS-optimized instance
Which of the following methods gives you protection against accidental loss of data stored in Amazon S3? (Choose 2)
Set bucket policies to restrict deletes, and also enable versioning
By default, versioning is enabled on a new bucket so you don’t have to worry about it (Not enabled by default)
Build a secondary index of your keys to protect the data (improves performance only)
Back up your bucket to a bucket owned by another AWS account for redundancy
A startup company hired you to help them build a mobile application that will ultimately store billions of image and videos in Amazon S3. The company is lean on funding, and wants to minimize operational costs, however, they have an aggressive marketing plan, and expect to double their current installation base every six months. Due to the nature of their business, they are expecting sudden and large increases to traffic to and from S3, and need to ensure that it can handle the performance needs of their application. What other information must you gather from this customer in order to determine whether S3 is the right option?
You must know how many customers that company has today, because this is critical in understanding what their customer base will be in two years. (No. of customers do not matter)
You must find out total number of requests per second at peak usage.
You must know the size of the individual objects being written to S3 in order to properly design the key namespace. (Size does not relate to the key namespace design but the count does)
In order to build the key namespace correctly, you must understand the total amount of storage needs for each S3 bucket. (S3 provided unlimited storage the key namespace design would depend on the number)
A document storage company is deploying their application to AWS and changing their business model to support both free tier and premium tier users. The premium tier users will be allowed to store up to 200GB of data and free tier customers will be allowed to store only 5GB. The customer expects that billions of files will be stored. All users need to be alerted when approaching 75 percent quota utilization and again at 90 percent quota use. To support the free tier and premium tier users, how should they architect their application?
The company should utilize an amazon simple workflow service activity worker that updates the users data counter in amazon dynamo DB. The activity worker will use simple email service to send an email if the counter increases above the appropriate thresholds.
The company should deploy an amazon relational data base service relational database with a store objects table that has a row for each stored object along with size of each object. The upload server will query the aggregate consumption of the user in questions (by first determining the files store by the user, and then querying the stored objects table for respective file sizes) and send an email via Amazon Simple Email Service if the thresholds are breached. (Good Approach to use RDS but with so many objects might not be a good option)
The company should write both the content length and the username of the files owner as S3 metadata for the object. They should then create a file watcher to iterate over each object and aggregate the size for each user and send a notification via Amazon Simple Queue Service to an emailing service if the storage threshold is exceeded. (List operations on S3 not feasible)
The company should create two separated amazon simple storage service buckets one for data storage for free tier users and another for data storage for premium tier users. An amazon simple workflow service activity worker will query all objects for a given user based on the bucket the data is stored in and aggregate storage. The activity worker will notify the user via Amazon Simple Notification Service when necessary (List operations on S3 not feasible as well as SNS does not address email requirement)
Your company host a social media website for storing and sharing documents. the web application allow users to upload large files while resuming and pausing the upload as needed. Currently, files are uploaded to your php front end backed by Elastic Load Balancing and an autoscaling fleet of amazon elastic compute cloud (EC2) instances that scale upon average of bytes received (NetworkIn) After a file has been uploaded. it is copied to amazon simple storage service(S3). Amazon Ec2 instances use an AWS Identity and Access Management (AMI) role that allows Amazon s3 uploads. Over the last six months, your user base and scale have increased significantly, forcing you to increase the auto scaling groups Max parameter a few times. Your CFO is concerned about the rising costs and has asked you to adjust the architecture where needed to better optimize costs. Which architecture change could you introduce to reduce cost and still keep your web application secure and scalable?
Replace the Autoscaling launch Configuration to include c3.8xlarge instances; those instances can potentially yield a network throughput of 10gbps. (no info of current size and might increase cost)
Re-architect your ingest pattern, have the app authenticate against your identity provider as a broker fetching temporary AWS credentials from AWS Secure token service (GetFederation Token). Securely pass the credentials and s3 endpoint/prefix to your app. Implement client-side logic to directly upload the file to amazon s3 using the given credentials and S3 Prefix. (will not provide the ability to handle pause and restarts)
Re-architect your ingest pattern, and move your web application instances into a VPC public subnet. Attach a public IP address for each EC2 instance (using the auto scaling launch configuration settings). Use Amazon Route 53 round robin records set and http health check to DNS load balance the app request this approach will significantly reduce the cost by bypassing elastic load balancing. (ELB is not the bottleneck)
Re-architect your ingest pattern, have the app authenticate against your identity provider as a broker fetching temporary AWS credentials from AWS Secure token service (GetFederation Token). Securely pass the credentials and s3 endpoint/prefix to your app. Implement client-side logic that used the S3 multipart upload API to directly upload the file to Amazon s3 using the given credentials and s3 Prefix. (multipart allows one to start uploading directly to S3 before the actual size is known or complete data is downloaded)
If an application is storing hourly log files from thousands of instances from a high traffic web site, which naming scheme would give optimal performance on S3?
Sequential
instanceID_log-HH-DD-MM-YYYY
instanceID_log-YYYY-MM-DD-HH
HH-DD-MM-YYYY-log_instanceID (HH will give some randomness to start with instead of instaneId where the first characters would be i-)
