AWS Storage Options – EBS & Instance Store

Udemy June Discount Braincert-AWS-Certified-SA-Professional-Practice-Exam

Amazon Elastic Block Store (EBS) volumes

  • EBS provides a durable block-level storage for use with Amazon EC2 instances (virtual machines)
  • EBS volumes are off-instance, network-attached storage (NAS) that persists independently from the running life of a single Amazon EC2 instance
  • Amazon EBS volume is attached to an instance, can be used like a physical hard drive, typically by formatting it with the file system of your choice and using the file I/O interface provided by the instance operating system.
  • Amazon EBS volume can be used to boot an Amazon EC2 instance (Amazon EBS-root AMIs only), and multiple Amazon EBS volumes can be attached to a single Amazon EC2 instance.
  • Amazon EBS volume can be attached to a Single EC2 instance only at any point of time
  • Amazon EBS provides ability to take point-in-time snapshots, which are persisted in S3. These snapshots can be used to instantiate new EC2 volumes and to protect data for long-term durability
  • EBS snapshots can be copied across AWS region as well, making it easier to leverage multiple AWS regions for geographical expansion, data center migration and disaster recovery

Ideal Usage Patterns

  • Amazon EBS is meant for data that changes relatively frequently and requires long-term persistence.
  • Amazon EBS volume provide access to raw block-level storage and is particularly well-suited for use as the primary storage for a database or file system
  • Amazon EBS Provisioned IOPS volumes are particularly well-suited for use with databases applications that require a high and consistent rate of random disk reads and writes

Anti-Patterns

  • Temporary Storage
    • Amazon EBS volume persists independent of the attached EC2 life cycle.
    • For temporary storage such as caches, buffers, queues etc it is better to use local instance store volumes, SQS or Elastic Cache
  • Highly-durable storage
    • Amazon EBS volumes with less than 20 GB of modified data since the last snapshot are designed for between 99.5% and 99.9% annual durability; volumes with more modified data can be expected to have proportionally lower durability
    • For highly-durable storage, use Amazon S3 or Amazon Glacier which provides 99.999999999% annual durability per object
  • Static data or web content
    • For static web content, where data infrequently changes, EBS with EC2 would require a web server to serve the pages.
    • Amazon S3 may represent a more cost-effective and scalable solution for storing this fixed information and is served directly out of S3.

Performance

  • Amazon EBS provides two volume types: standard volumes and Provisioned IOPS volumes which differ in performance characteristics and pricing model, allowing you to tailor your storage performance and cost to the needs of your applications.
  • EBS Volumes can be attached and striped across multiple similarly-provisioned Amazon EBS volumes using RAID 0 or logical volume manager software, thus aggregating available IOPs, total volume throughput, and total volume size.
  • Standard volumes offer cost effective storage for applications with moderate or bursty I/O requirements. Standard volumes are also well suited for use as boot volumes, where the burst capability provides fast instance start-up times.
  • Provisioned IOPS volumes are designed to deliver predictable, high performance for I/O intensive workloads such as databases. With Provisioned IOPS, you specify an IOPS rate when creating a volume, and then Amazon EBS provisions that rate for the lifetime of the volume.
  • As Amazon EBS volumes are network-attached devices, other network I/O performed by the instance, as well as the total load on the shared network, can affect individual Amazon EBS volume performance. EBS optimized instances can be launched which deliver dedicated throughput between Amazon EC2 and Amazon EBS and enables instances to fully utilize the Provisioned IOPS on an Amazon EBS volume,
  • Each separate Amazon EBS volume can be configured as Amazon EBS standard or Amazon EBS Provisioned IOPS as needed. Alternatively, you could stripe your data

Durability & Availability

  • Amazon EBS volumes are designed to be highly available and reliable
  • EBS volume data is replicated across multiple servers in a single Availability Zone to prevent the loss of data from the failure of any single component
  • Durability of your Amazon EBS volume depends on both the size of your volume and the amount of data that has changed since your last snapshot
  • Amazon EBS snapshots are incremental, point-in-time backups, containing only the data blocks changed since the last snapshot. Amazon EBS volumes that operate with 20 GB or less of modified data since their most recent snapshot can expect an annual failure rate (AFR) between 0.1% and 0.5%. Amazon EBS volumes with more than 20 GB of modified data since the last snapshot should expect higher failure rates that are roughly proportional to the increase in modified data.
  • It is therefore recommended to create frequent snapshots to maximize both durability and availability of their Amazon EBS data
  • Amazon EBS snapshots provides an easy-to-use disk clone or disk image mechanism for backup, sharing, and disaster-recovery.

Cost Model

  • Amazon EBS pricing has 3 components: provisioned storage, I/O requests and snapshot storage
  • Standard volumes are charged per GB-month of provisioned storage and per million I/O requests
  • EBS Provisioned IOPS volumes are charged per GB-month of provisioned storage and per Provisioned IOPS-month
  • For both volumes, Amazon EBS snapshots are charged per GB-month of data stored. Amazon EBS snapshot copy is charged for the data transferred between regions, and for the standard Amazon EBS snapshot charges in the destination region.
  • For an EBS volume all storage is allocated at the time of volume creation, and that you are charged for this allocated storage even if you don’t write data to it.
  • For Amazon EBS snapshots, you are charged only for storage actually used (consumed). Note that Amazon EBS snapshots are incremental and compressed, so the storage used in any snapshot is generally much less than the storage consumed on an Amazon EBS volume

Scalability and Elasticity

  • Amazon EBS volumes can easily and rapidly be provisioned and released to scale in and out with the changing total storage demands
  • Amazon EBS volumes cannot be resized, and if additional storage is needed either
    • An additional volume can be attached
    • Create a snapshot and create a new volume from the snapshot with a higher volume size
  • NOTE : With latest AWS enhancements EBS volumes can be resize and enlarged.

Interfaces

  • Amazon offers management APIs for Amazon EBS in both SOAP and REST formats which can be used to create, delete, describe, attach, and detach Amazon EBS volumes for your Amazon EC2 instances as well as to create, delete, and describe snapshots from Amazon EBS to Amazon S3; and to copy snapshots from one region to another
  • Amazon also offers the same capabilities through AWS Management Console

Instance Store Volumes

  • Instance Store volumes, also referred to as Ephemeral Storage, provide temporary block-level storage and consists of preconfigured and pre-attached block of disk storage on the same physical server as the EC2 instance
  • Instance storage’s amount of disk storage depends on the Instance type and larger instances provide both more and larger instance store volumes. Smaller instance types such as micro instances can only be launched with EBS volumes.
  • Storage optimized instances provides special purpose instance storage targeted to specific uses case for e.g. HI1 provides very fast solid-state drive (SSD) backed instance storage are capable of supporting over 120,000 random read IOPS, and are optimized for very high random I/O performance and low cost per IOPS. While, HS1 instances are optimized for very high storage density, low storage cost, and high sequential I/O performance.
  • Instance store volumes, unlike EBS volumes, cannot be detached or attached to another instance

Ideal Usage Patterns

  • Amazon EC2 local instance store volumes are fast, free (that is, included in the price of the Amazon EC2 instance) “scratch volumes” best suited for storing temporary data that is continually changing, such as buffers, caches, scratch data or can easily be regenerated, or data that is replicated for durability
  • High I/O instances provide instance store volumes backed by SSD, and are ideally suited for many high performance database workloads. for e.g. applications include NoSQL databases like Cassandra and MongoDB.
  • High storage instances support much higher storage density per Amazon EC2 instance, and are ideally suited for applications that benefit from high sequential I/O performance across very large datasets. for e.g. applications include data warehouses, Hadoop storage nodes, seismic analysis, cluster file systems, etc.

Anti-Patterns

  • Persistent storage
    • For persistent virtual disk storage similar to a physical disk drive for files or other data that must persist longer than the lifetime of a single Amazon EC2 instance, Amazon EBS volumes or Amazon S3 are more appropriate.
  • Relational database storage
    • In most cases, relational databases require storage that persists beyond the lifetime of a single Amazon EC2 instance, making Amazon EBS volumes the natural choice.
  • Shared storage
    • Instance store volumes are dedicated to a single Amazon EC2 instance, and cannot be shared with other systems or users. If you need storage that can be detached from one instance and attached to a different instance, or if you need the ability to share data easily, Amazon S3 or Amazon EBS volumes are the better choice.
  • Snapshots
    • If you need the convenience, long-term durability, availability, and shareability of point-in-time disk snapshots, Amazon EBS volumes are a better choice.

Performance

  • Non-SSD-based instance store volumes in most Amazon EC2 instance families have performance characteristics similar to standard Amazon EBS volumes.
  • Amazon EC2 instance virtual machine and the local instance store volumes are located in the same physical server, interaction with the storage is very fast, particularly for sequential access.
  • To further increase aggregate IOPS, or to improve sequential disk throughput, multiple instance store volumes can be grouped together using RAID 0 (disk striping) software.
  • Because the bandwidth to the disks is not limited by the network, aggregate sequential throughput for multiple instance volumes can be higher than for the same number of Amazon EBS volumes.
  • SSD instance store volumes in the Amazon EC2 high I/O instances provide from tens of thousands to hundreds of thousands of low-latency, random 4 KB random IOPS.
    Because of the I/O characteristics of SSD devices, write performance can be variable.
  • Instance store volumes on Amazon EC2 high storage instances provide very high storage density and high sequential read and write performance. High storage instances are capable of delivering 2.6 GB/sec of sequential read and write performance when using a block size of 2 MB.

Durability and Availability

  • Amazon EC2 local instance store volumes are not intended to be used as durable disk storage and they persists only during the life of the associate Amazon EC2 instance

Cost Model

  • Cost of the Amazon EC2 instance includes any local instance store volumes, if the instance type provides them.
  • While there is no additional charge for data storage on local instance store volumes, note that data transferred to and from Amazon EC2 instance store volumes from other Availability Zones or outside of an Amazon EC2 region may incur data transfer charges, and additional charges will apply for use of any persistent storage, such as Amazon S3, Amazon Glacier, Amazon EBS volumes, and Amazon EBS snapshots

Scalability and Elasticity

  • Local instance store volumes are tied to a particular Amazon EC2 instance, and are fixed in number and size for a given Amazon EC2 instance type, so the scalability and elasticity of this storage is tied to the number of Amazon EC2 instances.

Interfaces

  • Instance store volumes are specified using the block device mapping feature of the Amazon EC2 API and the AWS Management Console
  • To the Amazon EC2 instance, an instance store volume appears just like a local disk drive. To write to and read data from instance store volumes, use the native file system I/O interfaces of the chosen operating system.

AWS Certification Exam Practice Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  1. Which of the following provides the fastest storage medium?
    1. Amazon S3
    2. Amazon EBS using Provisioned IOPS (PIOPS)
    3. SSD Instance (ephemeral) store (SSD Instance Storage provides 100,000 IOPS on some instance types, much faster than any network-attached storage)
    4. AWS Storage Gateway

References

5 thoughts on “AWS Storage Options – EBS & Instance Store

  1. Hi Jayendra,

    >>High I/O instances provide instance store volumes backed by SSD, and are ideally suited for many high performance database workloads. for e.g. applications include NoSQL databases like Cassandra and MongoDB.

    Since SSD storage is ephemeral. Why would they be good candidates for database workloads? If the instance is restarted, the data will be lost so I think these are not good candidates for db workload. Amazon documentation (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/storage_instances.html) also says it is good — I think I’m missing a point. Could you please clarify?

    Cheers,
    Satish

    1. The reason for using SSD Instance store for NoSQL and other use cases is the performance. However, the data needs to be backed up or replicated to prevent any data loss if the instance fails.
      For e.g. AWS DyanamoDB the data items are stored on solid-state drives (SSDs) and are automatically replicated across three Availability Zones in an AWS region to provide built-in high availability and data durability.

    2. You can create an HA and durable architecture using Distributed Replicated Block Device (DRBD). Create RAID 0 by striping multiples instance store together. Replicate the same model in other AZ and user DRDB to replicate data in both AZ instance store. Add one more cluster of RAID 0 EBS volume and replicate data there using DRDB. Take the snapshot of EBS volume and store in S3.

  2. From the post,
    “Amazon EBS volumes cannot be resized, and if additional storage is needed either”

    Tested and found that EBS volume can be enlarged, but cannot be shrinked.

Leave a Reply

Your email address will not be published. Required fields are marked *