AWS S3 Glacier

May 15, 2023 ~ Last updated on : June 14, 2023 ~ jayendrapatil ~ 4 Comments

AWS S3 Glacier

S3 Glacier is a storage service optimized for archival, infrequently used data, or “cold data.”
S3 Glacier is an extremely secure, durable, and low-cost storage service for data archiving and long-term backup.
provides average annual durability of 99.999999999% (11 9’s) for an archive.
redundantly stores data in multiple facilities and on multiple devices within each facility.
synchronously stores the data across multiple facilities before returning SUCCESS on uploading archives, to enhance durability.
performs regular, systematic data integrity checks and is built to be automatically self-healing.
enables customers to offload the administrative burdens of operating and scaling storage to AWS, without having to worry about capacity planning, hardware provisioning, data replication, hardware failure detection, recovery, or time-consuming hardware migrations.
offers a range of storage classes and patterns
- S3 Glacier Instant Retrieval
  - Use for archiving data that is rarely accessed and requires milliseconds retrieval.
- S3 Glacier Flexible Retrieval (formerly the S3 Glacier storage class)
  - Use for archives where portions of the data might need to be retrieved in minutes.
  - offers a range of data retrievals options where the retrieval time varies from minutes to hours.
    - Expedited retrieval: 1-5 mins
    - Standard retrieval: 3-5 hours
    - Bulk retrieval: 5-12 hours
- S3 Glacier Deep Archive
  - Use for archiving data that rarely need to be accessed.
  - Data stored has a default retrieval time of 12 hours.
- S3 Glacier Flexible Retrieval and S3 Glacier Deep Archive objects are not available for real-time access.
is a great storage choice when low storage cost is paramount, with data rarely retrieved, and retrieval latency is acceptable. S3 should be used if applications require fast, frequent real-time access to the data.
can store virtually any kind of data in any format.
allows interaction through AWS Management Console, Command Line Interface CLI, and SDKs or REST-based APIs.
- AWS Management console can only be used to create and delete vaults.
- Rest of the operations to upload, download data, and create jobs for retrieval need CLI, SDK, or REST-based APIs.
Use cases include
- Digital media archives
- Data that must be retained for regulatory compliance
- Financial and healthcare records
- Raw genomic sequence data
- Long-term database backups

S3 Glacier Storage Classes

S3 Glacier Instant Retrieval

Use for archiving data that is rarely accessed and requires milliseconds retrieval.

S3 Glacier Flexible Retrieval (S3 Glacier Storage Class)

Use for archives where portions of the data might need to be retrieved in minutes.
Data has a minimum storage duration period of 90 days and can be accessed in as little as 1-5 minutes by using an expedited retrieval
You can also request free Bulk retrievals in up to 5-12 hours.
S3 supports restore requests at a rate of up to 1,000 transactions per second, per AWS account.

S3 Glacier Deep Archive

Use for archiving data that rarely needs to be accessed.
S3 Glacier Deep Archive is the lowest cost storage option in AWS.
Retrieval costs can be reduced further using bulk retrieval, which returns data within 48 hours.
Data stored has a minimum storage duration period of 180 days
Data stored has a default retrieval time of 12 hours.
S3 supports restore requests at a rate of up to 1,000 transactions per second, per AWS account.

S3 Glacier Flexible Data Retrievals Options

Glacier provides three options for retrieving data with varying access times and costs: Expedited, Standard, and Bulk retrievals.

Expedited Retrievals

Expedited retrievals allow quick access to the data when occasional urgent requests for a subset of archives are required.
Data has a minimum storage duration period of 90 days
Data accessed are typically made available within 1-5 minutes.
There are two types of Expedited retrievals: On-Demand and Provisioned.
- On-Demand requests are like EC2 On-Demand instances and are available the vast majority of the time.
- Provisioned requests are guaranteed to be available when needed.

Standard Retrievals

Standard retrievals allow access to any of the archives within several hours.
Standard retrievals typically complete within 3-5 hours.

Bulk Retrievals

Bulk retrievals are Glacier’s lowest-cost retrieval option, enabling retrieval of large amounts, even petabytes, of data inexpensively in a day.
Bulk retrievals typically complete within 5-12 hours.

S3 Glacier Data Model

Glacier data model core concepts include vaults and archives and also include job and notification configuration resources

Vault

A vault is a container for storing archives.
Each vault resource has a unique address, which comprises the region the vault was created and the unique vault name within the region and account for e.g. https://glacier.us-west-2.amazonaws.com/111122223333/vaults/examplevault
Vault allows the storage of an unlimited number of archives.
Glacier supports various vault operations which are region-specific.
An AWS account can create up to 1,000 vaults per region.

Jobs

A Job is required to retrieve an Archive and vault inventory list
Data retrieval requests are asynchronous operations, are queued and some jobs can take about four hours to complete.
A job is first initiated and then the output of the job is downloaded after the job is completed.
Vault inventory jobs need the vault name.
Data retrieval jobs need both the vault name and the archive id, with an optional description
A vault can have multiple jobs in progress at any point in time and can be identified by Job ID, assigned when is it created for tracking
Glacier maintains job information such as job type, description, creation date, completion date, and job status and can be queried
After the job completes, the job output can be downloaded in full or partially by specifying a byte range.

Notification Configuration

As the jobs are asynchronous, Glacier supports a notification mechanism to an SNS topic when the job completes
SNS topic for notification can either be specified with each individual job request or with the vault
Glacier stores the notification configuration as a JSON document

Glacier Supported Operations

Vault Operations

Glacier provides operations to create and delete vaults.
A vault can be deleted only if there are no archives in the vault as of the last computed inventory and there have been no writes to the vault since the last inventory (as the inventory is prepared periodically)
Vault Inventory
- Vault inventory helps retrieve a list of archives in a vault with information such as archive ID, creation date, and size for each archive
- Inventory for each vault is prepared periodically, every 24 hours
- Vault inventory is updated approximately once a day, starting on the day the first archive is uploaded to the vault.
- When a vault inventory job is, Glacier returns the last inventory it generated, which is a point-in-time snapshot and not real-time data.
Vault Metadata or Description can also be obtained for a specific vault or for all vaults in a region, which provides information such as
- creation date,
- number of archives in the vault,
- total size in bytes used by all the archives in the vault,
- and the date the vault inventory was generated
S3 Glacier also provides operations to set, retrieve, and delete a notification configuration on the vault. Notifications can be used to identify vault events.

Archive Operations

S3 Glacier provides operations to upload, download and delete archives.
All archive operations must either be done using AWS CLI or SDK. It cannot be done using AWS Management Console.
An existing archive cannot be updated, it has to be deleted and uploaded.

Archive Upload

An archive can be uploaded in a single operation (1 byte to up to 4 GB in size) or in parts referred to as Multipart upload (40 TB)
Multipart Upload helps to
- improve the upload experience for larger archives.
- upload archives in parts, independently, parallelly and in any order
- faster recovery by needing to upload only the part that failed upload and not the entire archive.
- upload archives without even knowing the size
- upload archives from 1 byte to about 40,000 GB (10,000 parts * 4 GB) in size
To upload existing data to Glacier, consider using the AWS ~~Import/Export~~ Snowball service, which accelerates moving large amounts of data into and out of AWS using portable storage devices for transport. AWS transfers the data directly onto and off of storage devices using Amazon’s high-speed internal network, bypassing the Internet.
Glacier returns a response that includes an archive ID that is unique in the region in which the archive is stored.
Glacier does not support any additional metadata information apart from an optional description. Any additional metadata information required should be maintained on the client side.

Archive Download

Downloading an archive is an asynchronous operation and is the 2 step process
- Initiate an archive retrieval job
  - When a Job is initiated, a job ID is returned as a part of the response.
  - Job is executed asynchronously and the output can be downloaded after the job completes.
  - A job can be initiated to download the entire archive or a portion of the archive.
- After the job completes, download the bytes
  - An archive can be downloaded as all the bytes or a specific byte range to download only a portion of the output
  - Downloading the archive in chunks helps in the event of a download failure, as only that part needs to be downloaded
  - Job completion status can be checked by
    - Check status explicitly (Not Recommended)
      - periodically poll the describe job operation request to obtain job information
    - Completion notification
      - An SNS topic can be specified, when the job is initiated or with the vault, to be used to notify job completion

About Range Retrievals

S3 Glacier allows retrieving an archive either in whole (default) or a range, or a portion.
Range retrievals need a range to be provided that is megabyte aligned.
Glacier returns a checksum in the response which can be used to verify if any errors in the download by comparing it with the checksum computed on the client side.
Specifying a range of bytes can be helpful when:
- Control bandwidth costs
  - Glacier allows retrieval of up to 5 percent of the average monthly storage (pro-rated daily) for free each month
  - Scheduling range retrievals can help in two ways.
    - meet the monthly free allowance of 5 percent by spreading out the data requested
    - if the amount of data retrieved doesn’t meet the free allowance percentage, scheduling range retrievals enable a reduction of the peak retrieval rate, which determines the retrieval fees.
- Manage your data downloads
  - Glacier allows retrieved data to be downloaded for 24 hours after the retrieval request completes
  - Only portions of the archive can be retrieved so that the schedule of downloads can be managed within the given download window.
- Retrieve a targeted part of a large archive
  - Retrieving an archive in a range can be useful if an archive is uploaded as an aggregate of multiple individual files, and only a few files need to be retrieved

Archive Deletion

An archive can be deleted from the vault only one at a time
This operation is idempotent. Deleting an already-deleted archive does not result in an error
AWS applies a pro-rated charge for items that are deleted prior to 90 days, as it is meant for long-term storage

Archive Update

An existing archive cannot be updated and must be deleted and re-uploaded, which would be assigned a new archive id

S3 Glacier Vault Lock

S3 Glacier Vault Lock helps deploy and enforce compliance controls for individual S3 Glacier vaults with a vault lock policy.
Specify controls such as “write once read many” (WORM) can be enforced using a vault lock policy and the policy can be locked for future edits.
Once locked, the policy can no longer be changed.

S3 Glacier Security

S3 Glacier supports data in transit encryption using Secure Sockets Layer (SSL) or client-side encryption.
All data is encrypted on the server side with Glacier handling key management and key protection. It uses AES-256, one of the strongest block ciphers available
Security and compliance of S3 Glacier are assessed by third-party auditors as part of multiple AWS compliance programs including SOC, HIPAA, PCI DSS, FedRAMP, etc.

AWS Certification Exam Practice Questions

Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).

AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.

AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated

Open to further feedback, discussion and correction.

What is Amazon Glacier?
1. You mean Amazon “Iceberg”: it’s a low-cost storage service.
2. A security tool that allows to “freeze” an EBS volume and perform computer forensics on it.
3. A low-cost storage service that provides secure and durable storage for data archiving and backup
4. It’s a security tool that allows to “freeze” an EC2 instance and perform computer forensics on it.
Amazon Glacier is designed for: (Choose 2 answers)
1. Active database storage
2. Infrequently accessed data
3. Data archives
4. Frequently accessed data
5. Cached session data
An organization is generating digital policy files which are required by the admins for verification. Once the files are verified they may not be required in the future unless there is some compliance issue. If the organization wants to save them in a cost effective way, which is the best possible solution?
1. AWS RRS
2. AWS S3
3. AWS RDS
4. AWS Glacier
A user has moved an object to Glacier using the life cycle rules. The user requests to restore the archive after 6 months. When the restore request is completed the user accesses that archive. Which of the below mentioned statements is not true in this condition?
1. The archive will be available as an object for the duration specified by the user during the restoration request
2. The restored object’s storage class will be RRS (After the object is restored the storage class still remains GLACIER. Read more)
3. The user can modify the restoration period only by issuing a new restore request with the updated period
4. The user needs to pay storage for both RRS (restored) and Glacier (Archive) Rates
To meet regulatory requirements, a pharmaceuticals company needs to archive data after a drug trial test is concluded. Each drug trial test may generate up to several thousands of files, with compressed file sizes ranging from 1 byte to 100MB. Once archived, data rarely needs to be restored, and on the rare occasion when restoration is needed, the company has 24 hours to restore specific files that match certain metadata. Searches must be possible by numeric file ID, drug name, participant names, date ranges, and other metadata. Which is the most cost-effective architectural approach that can meet the requirements?
1. Store individual files in Amazon Glacier, using the file ID as the archive name. When restoring data, query the Amazon Glacier vault for files matching the search criteria. (Individual files are expensive and does not allow searching by participant names etc)
2. Store individual files in Amazon S3, and store search metadata in an Amazon Relational Database Service (RDS) multi-AZ database. Create a lifecycle rule to move the data to Amazon Glacier after a certain number of days. When restoring data, query the Amazon RDS database for files matching the search criteria, and move the files matching the search criteria back to S3 Standard class. (As the data is not needed can be stored to Glacier directly and the data need not be moved back to S3 standard)
3. Store individual files in Amazon Glacier, and store the search metadata in an Amazon RDS multi-AZ database. When restoring data, query the Amazon RDS database for files matching the search criteria, and retrieve the archive name that matches the file ID returned from the database query. (Individual files and Multi-AZ is expensive)
4. First, compress and then concatenate all files for a completed drug trial test into a single Amazon Glacier archive. Store the associated byte ranges for the compressed files along with other search metadata in an Amazon RDS database with regular snapshotting. When restoring data, query the database for files that match the search criteria, and create restored files from the retrieved byte ranges.
5. Store individual compressed files and search metadata in Amazon Simple Storage Service (S3). Create a lifecycle rule to move the data to Amazon Glacier, after a certain number of days. When restoring data, query the Amazon S3 bucket for files matching the search criteria, and retrieve the file to S3 reduced redundancy in order to move it back to S3 Standard class. (Once the data is moved from S3 to Glacier the metadata is lost, as Glacier does not have metadata and must be maintained externally)
A user is uploading archives to Glacier. The user is trying to understand key Glacier resources. Which of the below mentioned options is not a Glacier resource?
1. Notification configuration
2. Archive ID
3. Job
4. Archive

References

Glacier Developer Guide

AWS Storage Services Cheat Sheet

February 9, 2017 ~ Last updated on : February 6, 2023 ~ jayendrapatil ~ 17 Comments

AWS Storage Services Cheat Sheet

AWS Storage Services

Simple Storage Service – S3

provides key-value based object storage with unlimited storage, unlimited objects up to 5 TB for the internet
offers an extremely durable, highly available, and infinitely scalable data storage infrastructure at very low costs.
is Object-level storage (not a Block level storage) and cannot be used to host OS or dynamic websites (but can work with Javascript SDK)
provides durability by redundantly storing objects on multiple facilities within a region
regularly verifies the integrity of data using checksums and provides the auto-healing capability
S3 resources consist of globally unique buckets with objects and related metadata. The data model is a flat structure with no hierarchies or folders.
S3 Replication enables automatic, asynchronous copying of objects across S3 buckets in the same or different AWS regions using SRR or CRR. Replication needs versioning enabled on either side.
S3 Transfer Acceleration helps speed data transport over long distances between a client and an S3 bucket using CloudFront edge locations.
S3 supports cost-effective Static Website hosting with Client-side scripts.
S3 CORS – Cross-Origin Resource Sharing allows cross-origin access to S3 resources.
S3 Access Logs enables tracking access requests to an S3 bucket.
S3 notification feature enables notifications to be triggered when certain events happen in the bucket.
S3 Inventory helps manage the storage and can be used to audit and report on the replication and encryption status of the objects for business, compliance, and regulatory needs.
Requestor Pays help bucket owner to specify that the requester requesting the download will be charged for the download.
S3 Batch Operations help perform large-scale batch operations on S3 objects and can perform a single operation on lists of specified S3 objects.
Pre-Signed URLs can be used shared for uploading/downloading objects for a limited time without requiring AWS security credentials.
Multipart Uploads allows
- parallel uploads with improved throughput and bandwidth utilization
- fault tolerance and quick recovery from network issues
- ability to pause and resume uploads
- begin an upload before the final object size is known
Versioning
- helps preserve, retrieve, and restore every version of every object
- protect from unintended overwrites and accidental deletions
- protects individual files but does NOT protect from Bucket deletion
MFA (Multi-Factor Authentication) can be enabled for additional security for the deletion of objects.
Integrates with CloudTrail, CloudWatch, and SNS for event notifications
S3 Storage Classes
- S3 Standard
  - default storage class, ideal for frequently accessed data
  - 99.999999999% durability & 99.99% availability
  - Low latency and high throughput performance
  - designed to sustain the loss of data in a two facilities
- S3 Standard-Infrequent Access (S3 Standard-IA)
  - optimized for long-lived and less frequently accessed data
  - designed to sustain the loss of data in a two facilities
  - 99.999999999% durability & 99.9% availability
  - suitable for objects greater than 128 KB kept for at least 30 days
- S3 One Zone-Infrequent Access (S3 One Zone-IA)
  - optimized for rapid access, less frequently access data
  - ideal for secondary backups and reproducible data
  - stores data in a single AZ, data stored in this storage class will be lost in the event of AZ destruction.
  - 99.999999999% durability & 99.5% availability
- ~~S3 Reduced Redundancy Storage (Not Recommended)~~
  - ~~designed for noncritical, reproducible data stored at lower levels of redundancy than the STANDARD storage class~~
  - ~~reduces storage costs~~
  - ~~99.99% durability & 99.99% availability~~
  - ~~designed to sustain the loss of data in a single facility~~
- S3 Glacier
  - suitable for low cost data archiving, where data access is infrequent
  - provides retrieval time of minutes to several hours
    - Expedited – 1 to 5 minutes
    - Standard – 3 to 5 hours
    - Bulk – 5 to 12 hours
  - 99.999999999% durability & 99.9% availability
  - Minimum storage duration of 90 days
- S3 Glacier Deep Archive (S3 Glacier Deep Archive)
  - provides lowest cost data archiving, where data access is infrequent
  - 99.999999999% durability & 99.9% availability
  - provides retrieval time of several (12-48) hours
    - Standard – 12 hours
    - Bulk – 48 hours
  - Minimum storage duration of 180 days
  - supports long-term retention and digital preservation for data that may be accessed once or twice a year
Lifecycle Management policies
- transition to move objects to different storage classes and Glacier
- expiration to remove objects and object versions
- can be applied to both current and non-current objects, in case, versioning is enabled.
Data Consistency Model
- provides strong read-after-write consistency for PUT and DELETE requests of objects in the S3 bucket in all AWS Regions
- updates to a single key are atomic
- does not currently support object locking for concurrent writes
S3 Security
- IAM policies – grant users within your own AWS account permission to access S3 resources
- Bucket and Object ACL – grant other AWS accounts (not specific users) access to S3 resources
- Bucket policies – allows to add or deny permissions across some or all of the objects within a single bucket
- S3 Access Points simplify data access for any AWS service or customer application that stores data in S3.
- S3 Glacier Vault Lock helps deploy and enforce compliance controls for individual S3 Glacier vaults with a vault lock policy.
- S3 VPC Gateway Endpoint enables private connections between a VPC and S3, without requiring that you use an internet gateway, NAT device, VPN connection, or AWS Direct Connect connection.
- Support SSL encryption of data in transit and data encryption at rest
S3 Data Encryption
- supports data at rest and data in transit encryption
- Server-Side Encryption
  - SSE-S3 – encrypts S3 objects using keys handled & managed by AWS
  - SSE-KMS – leverage AWS Key Management Service to manage encryption keys. KMS provides control and audit trail over the keys.
  - SSE-C – when you want to manage your own encryption keys. AWS does not store the encryption key. Requires HTTPS.
- Client-Side Encryption
  - Client library such as the S3 Encryption Client
  - Clients must encrypt data themselves before sending it to S3
  - Clients must decrypt data themselves when retrieving from S3
  - Customer fully manages the keys and encryption cycle
S3 Best Practices
- use random hash prefix for keys and ensure a random access pattern, as S3 stores object lexicographically randomness helps distribute the contents across multiple partitions for better performance
- use parallel threads and Multipart upload for faster writes
- use parallel threads and Range Header GET for faster reads
- for list operations with a large number of objects, it’s better to build a secondary index in DynamoDB
- use Versioning to protect from unintended overwrites and deletions, but this does not protect against bucket deletion
- use VPC S3 Endpoints with VPC to transfer data using Amazon internal network

Instance Store

provides temporary or ephemeral block-level storage for an EC2 instance
is physically attached to the Instance
deliver very high random I/O performance, which is a good option when storage with very low latency is needed
cannot be dynamically resized
data persists when an instance is rebooted
data does not persists if the
- underlying disk drive fails
- instance stops i.e. if the EBS backed instance with instance store volumes attached is stopped
- instance terminates
can be attached to an EC2 instance only when the instance is launched
is ideal for the temporary storage of information that changes frequently, such as buffers, caches, scratch data, and other temporary content, or for data that is replicated across a fleet of instances, such as a load-balanced pool of web servers.

Elastic Block Store – EBS

is virtual network-attached block storage
provides highly available, reliable, durable, block-level storage volumes that can be attached to a running instance
provides high durability and are redundant in an AZ, as the data is automatically replicated within that AZ to prevent data loss due to any single hardware component failure
persists and is independent of EC2 lifecycle
multiple volumes can be attached to a single EC2 instance
can be detached & attached to another EC2 instance in that same AZ only
volumes are Zonal i.e. created in a specific AZ and CAN’T span across AZs
snapshots
for making volume available to different AZ, create a snapshot of the volume and restore it to a new volume in any AZ within the region
for making the volume available to different Region, the snapshot of the volume can be copied to a different region and restored as a volume
PIOPS is designed to run transactions applications that require high and consistent IO for e.g. Relation database, NoSQL, etc
~~volumes CANNOT be shared with multiple EC2 instances, use EFS instead~~
Multi-Attach enables attaching a single Provisioned IOPS SSD (io1 or io2) volume to multiple instances that are in the same AZ.

EBS Encryption

allow encryption using the EBS encryption feature.
All data stored at rest, disk I/O, and snapshots created from the volume are encrypted.
uses 256-bit AES algorithms (AES-256) and an Amazon-managed KMS
Snapshots of encrypted EBS volumes are automatically encrypted.

EBS Snapshots

helps create backups of EBS volumes
are incremental
occur asynchronously, consume the instance IOPS
are regional and CANNOT span across regions
can be copied across regions to make it easier to leverage multiple regions for geographical expansion, data center migration, and disaster recovery
can be shared by making them public or with specific AWS accounts by modifying the access permissions of the snapshots
support EBS encryption
- Snapshots of encrypted volumes are automatically encrypted
- Volumes created from encrypted snapshots are automatically encrypted
- All data in flight between the instance and the volume is encrypted
- Volumes created from an unencrypted snapshot owned or have access to can be encrypted on the fly.
- Encrypted snapshot owned or having access to, can be encrypted with a different key during the copy process.
can be automated using AWS Data Lifecycle Manager

EBS vs Instance Store

Refer blog post @ EBS vs Instance Store

Glacier

suitable for archiving data, where data access is infrequent and a retrieval time of several hours (3 to 5 hours) is acceptable (Not true anymore with enhancements from AWS)
provides a high durability by storing archive in multiple facilities and multiple devices at a very low cost storage
performs regular, systematic data integrity checks and is built to be automatically self healing
aggregate files into bigger files before sending them to Glacier and use range retrievals to retrieve partial file and reduce costs
improve speed and reliability with multipart upload
automatically encrypts the data using AES-256
upload or download data to Glacier via SSL encrypted endpoints

EFS

fully-managed, easy to set up, scale, and cost-optimize file storage
can automatically scale from gigabytes to petabytes of data without needing to provision storage
provides managed NFS (network file system) that can be mounted on and accessed by multiple EC2 in multiple AZs simultaneously
highly durable, highly scalable and highly available.
- stores data redundantly across multiple Availability Zones
- grows and shrinks automatically as files are added and removed, so you there is no need to manage storage procurement or provisioning.
expensive (3x gp2), but you pay per use
uses the Network File System version 4 (NFS v4) protocol
is compatible with all Linux-based AMIs for EC2, POSIX file system (~Linux) that has a standard file API
does not support Windows AMI
offers the ability to encrypt data at rest using KMS and in transit.
can be accessed from on-premises using an AWS Direct Connect or AWS VPN connection between the on-premises datacenter and VPC.
can be accessed concurrently from servers in the on-premises datacenter as well as EC2 instances in the Amazon VPC
Performance mode
- General purpose (default)
  - latency-sensitive use cases (web server, CMS, etc…)
- Max I/O
  - higher latency, throughput, highly parallel (big data, media processing)
Storage Tiers
- Standard
  - for frequently accessed files
  - ideal for active file system workloads and you pay only for the file system storage you use per month
- Infrequent access (EFS-IA)
  - a lower cost storage class that’s cost-optimized for files infrequently accessed i.e. not accessed every day
  - cost to retrieve files, lower price to store
- EFS Lifecycle Management with choosing an age-off policy allows moving files to EFS IA
- Lifecycle Management automatically moves the data to the EFS IA storage class according to the lifecycle policy. for e.g., you can move files automatically into EFS IA fourteen days of not being accessed.
- EFS is a shared POSIX system for Linux systems and does not work for Windows

Amazon FSx for Windows

is a fully managed, highly reliable, and scalable Windows file system share drive
supports SMB protocol & Windows NTFS
supports Microsoft Active Directory integration, ACLs, user quotas
built on SSD, scale up to 10s of GB/s, millions of IOPS, 100s PB of data
is accessible from Windows, Linux, and MacOS compute instances
can be accessed from the on-premise infrastructure
can be configured to be Multi-AZ (high availability)
supports encryption of data at rest and in transit
provides data deduplication, which enables further cost optimization by removing redundant data.
data is backed-up daily to S3

Amazon FSx for Lustre

provides easy and cost effective way to launch and run the world’s most popular high-performance file system.
is a type of parallel distributed file system, for large-scale computing
Lustre is derived from “Linux” and “cluster”
Machine Learning, High Performance Computing (HPC) esp. Video Processing, Financial Modeling, Electronic Design Automation
scales up to 100s GB/s, millions of IOPS, sub-ms latencies
seamless integration with S3, it transparently presents S3 objects as files and allows you to write changed data back to S3.
can “read S3” as a file system (through FSx)
can write the output of the computations back to S3 (through FSx)
supports encryption of data at rest and in transit
can be used from on-premise servers

CloudFront

provides low latency and high data transfer speeds for distribution of static, dynamic web or streaming content to web users
delivers the content through a worldwide network of data centers called Edge Locations
keeps persistent connections with the origin servers so that the files can be fetched from the origin servers as quickly as possible.
dramatically reduces the number of network hops that users’ requests must pass through
supports multiple origin server options, like AWS hosted service for e.g. S3, EC2, ELB or an on premise server, which stores the original, definitive version of the objects
single distribution can have multiple origins and Path pattern in a cache behavior determines which requests are routed to the origin
supports Web Download distribution and RTMP Streaming distribution
- Web distribution supports static, dynamic web content, on demand using progressive download & HLS and live streaming video content
- RTMP supports streaming of media files using Adobe Media Server and the Adobe Real-Time Messaging Protocol (RTMP) ONLY
supports HTTPS using either
- dedicated IP address, which is expensive as dedicated IP address is assigned to each CloudFront edge location
- Server Name Indication (SNI), which is free but supported by modern browsers only with the domain name available in the request header
For E2E HTTPS connection,
- Viewers -> CloudFront needs either self signed certificate, or certificate issued by CA or ACM
- CloudFront -> Origin needs certificate issued by ACM for ELB and by CA for other origins
Security
- Origin Access Identity (OAI) can be used to restrict the content from S3 origin to be accessible from CloudFront only
- supports Geo restriction (Geo-Blocking) to whitelist or blacklist countries that can access the content
- Signed URLs
  - for RTMP distribution as signed cookies aren’t supported
  - to restrict access to individual files, for e.g., an installation download for your application.
  - users using a client, for e.g. a custom HTTP client, that doesn’t support cookies
- Signed Cookies
  - provide access to multiple restricted files, for e.g., video part files in HLS format or all of the files in the subscribers’ area of a website.
  - don’t want to change the current URLs
- integrates with AWS WAF, a web application firewall that helps protect web applications from attacks by allowing rules configured based on IP addresses, HTTP headers, and custom URI strings
supports GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE to get object & object headers, add, update, and delete objects
- only caches responses to GET and HEAD requests and, optionally, OPTIONS requests
- does not cache responses to PUT, POST, PATCH, DELETE request methods and these requests are proxied back to the origin
object removal from cache
- would be removed upon expiry (TTL) from the cache, by default 24 hrs
- can be invalidated explicitly, but has a cost associated, however might continue to see the old version until it expires from those caches
- objects can be invalidated only for Web distribution
- change object name, versioning, to serve different version
supports adding or modifying custom headers before the request is sent to origin which can be used to
- validate if user is accessing the content from CDN
- identifying CDN from which the request was forwarded from, in case of multiple CloudFront distribution
- for viewers not supporting CORS to return the Access-Control-Allow-Origin header for every request
supports Partial GET requests using range header to download object in smaller units improving the efficiency of partial downloads and recovery from partially failed transfers
supports compression to compress and serve compressed files when viewer requests include Accept-Encoding: gzip in the request header
supports different price class to include all regions, to include only least expensive regions and other regions to exclude most expensive regions
supports access logs which contain detailed information about every user request for both web and RTMP distribution

AWS Import/Export

accelerates moving large amounts of data into and out of AWS using portable storage devices for transport and transfers data directly using Amazon’s high speed internal network, bypassing the internet.
suitable for use cases with
- large datasets
- low bandwidth connections
- first time migration of data
Importing data to several types of AWS storage, including EBS snapshots, S3 buckets, and Glacier vaults.
Exporting data out from S3 only, with versioning enabled only the latest version is exported
Import data can be encrypted (optional but recommended) while export is always encrypted using Truecrypt
Amazon will wipe the device if specified, however it will not destroy the device

AWS Storage Options – S3 & Glacier

May 23, 2016 ~ Last updated on : June 8, 2018 ~ jayendrapatil ~ 30 Comments

Amazon S3

highly-scalable, reliable, and low-latency data storage infrastructure at very low costs.
provides a simple web services interface that can be used to store and retrieve any amount of data, at any time, from within Amazon EC2 or from anywhere on the web.
allows you to write, read, and delete objects containing from 1 byte to 5 terabytes of data each.
number of objects you can store in an Amazon S3 bucket is virtually unlimited.
highly secure, supporting encryption at rest, and providing multiple mechanisms to provide fine-grained control of access to Amazon S3 resources.
highly scalable, allowing concurrent read or write access to Amazon S3 data by many separate clients or application threads.
provides data lifecycle management capabilities, allowing users to define rules to automatically archive Amazon S3 data to Amazon Glacier, or to delete data at end of life.

Ideal Use Cases

Storage & Distribution of static web content and media
- frequently used to host static websites and provides a highly-available and highly-scalable solution for websites with only static content, including HTML files, images, videos, and client-side scripts such as JavaScript
- works well for fast growing websites hosting data intensive, user-generated content, such as video and photo sharing sites as no storage provisioning is required
- content can either be directly served from Amazon S3 since each object in Amazon S3 has a unique HTTP URL address
- can also act as an Origin store for the Content Delivery Network (CDN) such as Amazon CloudFront
- it works particularly well for hosting web content with extremely spiky bandwidth demands because of S3’s elasticity
Data Store for Large Objects
- can be paired with RDS or NoSQL database and used to store large objects for e.g. file or objects, while the associated metadata for e.g. name, tags, comments etc. can be stored in RDS or NoSQL database where it can be indexed and queried providing faster access to relevant data
Data store for computation and large-scale analytics
- commonly used as a data store for computation and large-scale analytics, such as analyzing financial transactions, clickstream analytics, and media transcoding.
- data can be accessed from multiple computing nodes concurrently without being constrained by a single connection because of its horizontal scalability
Backup and Archival of critical data
- used as a highly durable, scalable, and secure solution for backup and archival of critical data, and to provide disaster recovery solutions for business continuity.
- stores objects redundantly on multiple devices across multiple facilities, it provides the highly-durable storage infrastructure needed for these scenarios.
- it’s versioning capability is available to protect critical data from inadvertent deletion

Anti-Patterns

Amazon S3 has following Anti-Patterns where it is not an optimal solution

Dynamic website hosting
- While Amazon S3 is ideal for hosting static websites, dynamic websites requiring server side interaction, scripting or database interaction cannot be hosted and should rather be hosted on Amazon EC2
Backup and archival storage
- Data requiring long term archival storage with infrequent read access can be stored more cost effectively in Amazon Glacier
Structured Data Query
- Amazon S3 doesn’t offer query capabilities, so to read an object the object name and key must be known. Instead pair up S3 with RDS or Dynamo DB to store, index and query metadata about Amazon S3 objects
- NOTE – S3 now provides query capabilities and also Athena can be used
Rapidly Changing Data
- Data that needs to updated frequently might be better served by a storage solution with lower read/write latencies, such as Amazon EBS volumes, RDS or Dynamo DB.
File System
- Amazon S3 uses a flat namespace and isn’t meant to serve as a standalone, POSIX-compliant file system. However, by using delimiters (commonly either the ‘/’ or ‘’ character) you are able construct your keys to emulate the hierarchical folder structure of file system within a given bucket.

Performance

Access to Amazon S3 from within Amazon EC2 in the same region is fast.
Amazon S3 is designed so that server-side latencies are insignificant relative to Internet latencies.
Amazon S3 is also built to scale storage, requests, and users to support a virtually unlimited number of web-scale applications.
If Amazon S3 is accessed using multiple threads, multiple applications, or multiple clients concurrently, total Amazon S3 aggregate throughput will typically scale to rates that far exceed what any single server can generate or consume.

Durability & Availability

Amazon S3 storage provides provides the highest level of data durability and availability, by automatically and synchronously storing your data across both multiple devices and multiple facilities within the selected geographical region
Error correction is built-in, and there are no single points of failure. Amazon S3 is designed to sustain the concurrent loss of data in two facilities, making it very well-suited to serve as the primary data storage for mission-critical data.
Amazon S3 is designed for 99.999999999% (11 nines) durability per object and 99.99% availability over a one-year period.
Amazon S3 data can be protected from unintended deletions or overwrites using Versioning.
Versioning can be enabled with MFA (Multi Factor Authentication) Delete on the bucket, which would require two forms of authentication to delete an object
For Non Critical and Reproducible data for e.g. thumbnails, transcoded media etc., S3 Reduced Redundancy Storage (RRS) can be used, which provides a lower level of durability at a lower storage cost
RRS is designed to provide 99.99% durability per object over a given year. While RRS is less durable than standard Amazon S3, it is still designed to provide 400 times more durability than a typical disk drive

Cost Model

With Amazon S3, you pay only for what you use and there is no minimum fee.
Amazon S3 has three pricing components: storage (per GB per month), data transfer in or out (per GB per month), and requests (per n thousand requests per month).

Scalability & Elasticity

Amazon S3 has been designed to offer a very high level of scalability and elasticity automatically
Amazon S3 supports a virtually unlimited number of files in any bucket
Amazon S3 bucket can store a virtually unlimited number of bytes
Amazon S3 allows you to store any number of objects (files) in a single bucket, and Amazon S3 will automatically manage scaling and distributing redundant copies of your information to other servers in other locations in the same region, all using Amazon’s high-performance infrastructure.

Interfaces

Amazon S3 provides standards-based REST and ~~SOAP web services~~ APIs for both management and data operations.
NOTE – SOAP support over HTTP is deprecated, but it is still available over HTTPS. New Amazon S3 features will not be supported for SOAP. We recommend that you use either the REST API or the AWS SDKs.
Amazon S3 provides easier to use higher level toolkit or SDK in different languages (Java, .NET, PHP, and Ruby) that wraps the underlying APIs
Amazon S3 Command Line Interface (CLI) provides a set of high-level, Linux-like Amazon S3 file commands for common operations, such as ls, cp, mv, sync, etc. They also provide the ability to perform recursive uploads and downloads using a single folder-level Amazon S3 command, and supports parallel transfers.
AWS Management Console provides the ability to easily create and manage Amazon S3 buckets, upload and download objects, and browse the contents of your Amazon S3 buckets using a simple web-based user interface
All interfaces provide the ability to store Amazon S3 objects (files) in uniquely-named buckets (top-level folders), with each object identified by an unique Object key within that bucket.