AWS FSx for Lustre
- Amazon FSx for Lustre, is a fully managed service, that makes it easy and cost effective to launch and run the world’s most popular high-performance (HPC) Lustre file system.
- Lustre is an open source file system designed for applications that require fast storage – where you want your storage to keep up with your compute
- FSx handles the traditional complexity of setting up and managing high-performance Lustre file systems
- FSx for Lustre is ideal for use cases where speed matters, such as machine learning, high performance computing (HPC), video processing, financial modeling, genome sequencing, and electronic design automation (EDA)
- Amazon FSx provides multiple deployment options to optimize cost
- Scratch file systems
- designed for temporary storage and short-term processing of data.
- data is not replicated and does not persist if a file server fails.
- Persistent file systems
- designed for long-term storage and workloads.
- is highly available, and data is automatically replicated within the AZ that is associated with the file system.
- data volumes attached to the file servers are replicated independently from the file servers to which they are attached.
- Scratch file systems
- FSx for Lustre is compatible with the most popular Linux-based AMIs, including Amazon Linux, Amazon Linux 2, Red Hat Enterprise Linux (RHEL), CentOS, SUSE Linux and Ubuntu.
- FSx for Lustre can be accessed from a Linux instance, by installing the open-source Lustre client and mounting the file system using standard Linux commands.
FSx for Lustre with S3
- Amazon FSx also integrates seamlessly with S3, making it easy to process cloud data sets with the Lustre high-performance file system.
- Amazon FSx for Lustre file system transparently presents S3 objects as files and allows writing changed data back to S3.
- Amazon FSx for Lustre file system can be linked with a specified S3 bucket, making the data in the S3 accessible to the file system.
- S3 objects’ names and prefixes will be visible as files and directories
- Amazon S3 objects are lazy loaded by default.
- Objects are only loaded into the file system only when first accessed by the applications.
- Amazon FSx for Lustre automatically loads the corresponding objects from S3 when accessed
- Subsequent reads of these files are served directly out of the file system with low, consistent latencies.
- Amazon FSx for Lustre file system can optionally batch hydrated
- Amazon FSx for Lustre uses parallel data transfer techniques to transfer data from S3 at up to hundreds of GBs/s.
- Files from the file system can be exported back to the S3 bucket
FSx for Lustre Security
- FSx for Lustre provides encryption at rest for the file system and the backups, by default, using KMS
- FSx encrypts data-in-transit when accessed from supported EC2 instances only
FSx for Lustre Scalability
- Amazon FSx for Lustre file systems scale to hundreds of GB/s of throughput and millions of IOPS.
- FSx for Lustre also supports concurrent access to the same file or directory from thousands of compute instances.
- FSx for Lustre provides consistent, sub-millisecond latencies for file operations.
FSx for Lustre Availability and Durability
- On a scratch file system, file servers are not replaced if they fail and data is not replicated.
- On a persistent file system, if a file server becomes unavailable it is replaced automatically and within minutes.
- Amazon FSx for Lustre provides a parallel file system, where data is stored across multiple network file servers to maximize performance and reduce bottlenecks, and each server has multiple disks.
- Amazon FSx takes daily automatic incremental backups of the file systems, and allows manual backups at any point.
- Backups are highly durable and file-system-consistent
AWS Certification Exam Practice Questions
- Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
- AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
- AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
- Open to further feedback, discussion and correction.
- A solutions architect is designing storage for a high performance computing (HPC) environment based on Amazon Linux. The workload stores and processes a large amount of engineering drawings that require shared storage and heavy computing. Which storage option would be the optimal solution?
- Amazon Elastic File System (Amazon EFS)
- Amazon FSx for Lustre
- Amazon EC2 instance store
- Amazon EBS Provisioned IOPS SSD (io1)