AWS EC2 EBS Monitoring

Overview

Amazon Web Services (AWS) automatically provides data, such as Amazon CloudWatch metrics and volume status checks to help monitor EBS volumes

EBS Monitoring

CloudWatch Monitoring

  • CloudWatch metrics are statistical data that you can use to view, analyze, and set alarms on the operational behavior of the EBS volumes
  • CloudWatch provides the below by default
    • Basic – Data, in 5-minute periods at no charge, which includes data from the root devices volumes for EBS backed instances
    • Detailed – Provisioned IOPS (SSD) volumes send one-minute metrics
  • EBS Metrics
    • VolumeReadBytes & VolumeWriteBytes
      • Provides information on the I/O operations in a specified period of time, in bytes
    • VolumeReadOps & VolumeWriteOps
      • Total number (count) of I/O operations in a specified period of time
    • VolumeTotalReadTime & VolumeTotalWriteTime
      • Total number of seconds spent by all operations that completed in a specified period of time
    • VolumeIdleTime
      • Total number of seconds, in a specific period, when the volume was idle (no read and write operations)
    • VolumeQueueLength
      • Number of read and write operations, in a specific period, waiting to be completed
    • VolumeThroughputPercentage (Provisioned IOPS (SSD) volumes only)
      • Percentage of I/O operations per second (IOPS) delivered of the total IOPS provisioned
    • VolumeConsumedReadWriteOps (Provisioned IOPS (SSD) volumes only)
      • Total amount of read and write operations (normalized to 256K capacity units) consumed in a specified period of time

Volume Status Checks Monitoring

EC2 EBS Volume Status Check Monitoring

  • Volume status checks are automated tests that run every 5 minutes and return a pass or fail status.
  • Volume check status
    • Ok – all the status checks passed
    • Imparied – if the status checks failed
    • Insufficient-Data – checks are still in progress
    • Warning – the I/O performance of the volume is below expectations
  • When Amazon EBS determines the volume’s data is potentially inconsistent, it disables the I/O to the EBS volume from the attached EC2 instance to prevent any data corruption. This leads to the status check to fail and the volume status to be impaired. Amazon waits for the I/O to be enabled, giving you an opportunity to perform consistency checks
  • If the auto disabling of I/O is not needed, it can be overridden by enabling the Auto-Enabled IO flag, which would make the EBS volume auto available immediately after impaired status
  • Events would be fire for notification whenever the I/O for an EBS volume is disabled
  • I/O performance status checks, applicable only for Provisioned IOPS (SSD) volumes, compares actual volume performance with the expected volume performance and alerts if performing below expectations. This status check is performed every 1 minutes, however is collected by CloudWatch every 5 mins.
  • While initializing Provisioned IOPS (SSD) volumes that were restored from snapshots, the performance of the volume may drop below 50 percent of its expected level, which causes the volume to display a warning state in the I/O Performance status check. This is expected and can be ignored.

EC2 EBS Volume Status

Volume Events Monitoring

  • Amazon EBS generates events for volume status checks
  • Each event includes a start time that indicates the time at which the event occurred, and a duration that indicates how long I/O for the volume was disabled
  • Events description can be Awaiting Action (to enabled I/O), IO enabled, IO Auto-Enabled, or whether the status check resulted in Normal, Degraded, Severely Degraded or stalled status

Sample Exam Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  1. A user has configured CloudWatch monitoring on an EBS backed EC2 instance. If the user has not attached any additional device, which of the below mentioned metrics will always show a 0 value?
    1. DiskReadBytes
    2. NetworkIn
    3. NetworkOut
    4. CPUUtilization

References

AWS EC2 Instance Types – Certification

EC2 Instance Types Overview

  • EC2 Instance types determines the hardware of the host computer used for the instance.
  • Each instance type offers different compute, memory, and storage capabilities and are grouped in instance families based on these capabilities
  • EC2 provides each instance with a consistent and predictable amount of CPU capacity, regardless of its underlying hardware.
  • EC2 dedicates some resources of the host computer, such as CPU, memory, and instance storage, to a particular instance.
  • EC2 shares other resources of the host computer, such as the network and the disk subsystem, among instances. If each instance on a host computer tries to use as much of one of these shared resources as possible, each receives an equal share of that resource. However, when a resource is under-utilized, an instance can consume a higher share of that resource while it’s available

Current Generation Instance Types

EC2 Instance Types

EC2 Instance Type selection criteria

  • Some Instance types support only HVM virtualization type while others support both the PV and HVM virtualization types. AWS, however, recommends using HVM for taking advantage of the underlying hardware
  • All the instances are available in a VPC, however, few instance types are not available in an EC2-classic. AWS recommends VPC to take advantage of enhanced networking, multiple IP addresses, finer security control etc.
  • Some Instances types support only EBS volumes, while others support both EBS and Instance store volumes. Some instances that support instance store volumes use solid state drives (SSD) to deliver very high random I/O performance
  • Some instances can be launched as EBS optimized instances with a dedicated capacity for Amazon EBS I/O
  • Some instances can be launched in placement group for to optimize instances for High Performance Computing
  • Some instance support Enhanced Networking,  to get significantly higher packet per second (PPS) performance, lower network jitter, and lower latencies
  • Some Instances allow EBS volumes to be encrypted

EBS-Optimized

  • EBS-optimized instance uses an optimized configuration stack and provides additional, dedicated capacity for EBS I/O
  • EBS-optimized instances enable you to get consistently high performance for the EBS volumes by eliminating contention between EBS I/O and other network traffic from the instance
  • EBS-optimized instances deliver dedicated throughput to EBS, with options between 500 Mbps and 4,000 Mbps, depending on the instance type you use.
  • When attached to an EBS–optimized instance, General Purpose (SSD) volumes are designed to deliver within 10 percent of their baseline and burst performance 99.9 percent of the time in a given year, and Provisioned IOPS (SSD) volumes are designed to deliver within 10 percent of their provisioned performance 99.9 percent of the time in a given year.
  • EBS optimization can be enabled for an instance that is not EBS–optimized, by default

Placement Groups

Refer to My Blog Post @ EC2 Placement Groups

Instance Types

Screen Shot 2016-04-15 at 7.06.50 AM.png

T2 Instances (General Purpose)

  • T2 instances are designed to provide moderate baseline performance and the capability to burst to significantly higher performance as required by your workload.
  • Mainly intended for workloads that don’t use the full CPU often or consistently, but occasionally need to burst.
  • T2 instances are well suited for
    • general purpose workloads, such as web servers, developer environments, remote desktops and small databases
  • Requirements
    • can be launched only with HVM AMI
    • can be launched into a  VPC only, and not supported on the EC2-Classic platform
    • are available as EBS-backed instances only
    • are available as On-Demand or Reserved instances, but do not allow spot instances
    • By default, 20 (soft limit) T2 instances can run simultaneously
    • cannot be launched as a Dedicated instance

CPU Credits

  • CPU Credits (Similar to I/O Credits in case of the EBS general purpose storage) provides the performance of a full CPU core for one minute
  • T2 instances provide a baseline level of CPU performance, while CPU governs the ability to burst above the baseline level
  • One CPU credit is equal to one vCPU running at 100% utilization for one minute. for e.g. you can have One vCPU running at 100% for One minute OR One vCPU running @ 50% for 2 minutes OR Two vCPU running @ 25% for 2 minutes
  • Each T2 instance receives a healthy initial credit balance for startup performance
  • Initial CPU credits do not expire, but they are used first when an instance uses CPU credits.
  • Each T2 instance then continuously (at a millisecond-level resolution) receives a set rate of CPU credits per hour, depending on instance size for e.g. t2.nano earns 3/hour while a t2.large earns 36/hour
  • Each T2 instance accumulates the CPU credit when it uses fewer CPU resources than its allowed baseline performance levels
  • Maximum earned credit balance for an instance is equal to the number of CPU credits received per hour times 24 hours for e.g. t2.nano can earn max 72 (24 * 3) credits
  • CPU credit balance is available for a period of 24 hours and it expires 24 hours after they were earned. Expired credits are removed from the balance before new ones are added
  • CPU credit cease to persist between an instance stop – start. However, after the start, the instance receives the initial CPU credits again
  • When credit balance is completely exhausted, the instance will perform at its baseline performance

C4 Instances (Compute Intensive)

  • C4 instances are ideal for compute-bound applications that benefit from high performance processors
  • Well suited for
    • Batch processing workloads,
    • Media transcoding,
    • High-traffic web servers, massively multiplayer online (MMO) gaming servers, and ad serving engines,
    • High-traffic web servers, massively multiplayer online (MMO) gaming servers, and ad serving engines
  • Features
    • are EBS-optimized, by default
    • can be enabled for Enhanced Networking capabilities
    • can be clustered in a placement group
  • requirements
    • requires 64-bit HVM AMI
    • can be launched into a  VPC only, and not supported on the EC2-Classic platform

G2 Instances (Graphic Intensive)

  • GPU instances provide  high parallel processing capability
  • Well suited for
    • to accelerate many scientific, engineering, and rendering applications by leveraging the Compute Unified Device Architecture (CUDA) or OpenCL parallel computing frameworks
    • graphics applications, including game streaming, 3-D application streaming, and other graphics workloads
  • Requirements
    • requires HVM AMI
    • can’t access GPU unless NVIDIA drivers installed
  • Features
    • can be clustered in a placement group

I2 Instances (I/O Intensive)

  • I2 instances are optimized to deliver tens of thousands of low-latency, random I/O operations per second (IOPS) to applications.
  • Well suited for applications
    • NoSQL databases (for example, Cassandra and MongoDB)
    • Clustered databases
    • Online transaction processing (OLTP) systems
  • Features
    • Primary data storage is SSD-based instance storage.
    • can be enabled for Enhanced Networking capabilities
    • can be clustered in a placement group
    • can enable EBS–optimization to obtain additional, dedicated capacity for Amazon EBS I/O
  • Requirements
    • requires HVM AMI
  • HI1 is the equivalent previous generation instance
    • supports both PV and HVM AMIs

D2 Instances (Density Intensive)

  • D2 instances are designed for workloads with very high storage density and that require high sequential read and write access to very large data sets on local storage.
  • Well suited for applications
    • Massive parallel processing (MPP) data warehouse
    • MapReduce and Hadoop distributed computing
    • Log or data processing applications
  • Features
    • Primary data storage for D2 instances is HDD-based instance storage
    • are EBS-optimized, by default
    • can be enabled for Enhanced Networking capabilities
    • can be clustered in a placement group
  • requirements
    • requires 64-bit HVM AMI
  • HS1 is the equivalent previous generation instance
    • supports both EBS and Instance store backed AMIs
    • supports both PV and HVM AMIs

AWS Certification Exam Practice Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  1. Which of the following instance types are available as Amazon EBS-backed only? Choose 2 answers
    1. General purpose T2
    2. General purpose M3
    3. Compute-optimized C4
    4. Compute-optimized C3
    5. Storage-optimized 12
  2. A t2.medium EC2 instance type must be launched with what type of Amazon Machine Image (AMI)?
    1. An Instance store Hardware Virtual Machine AMI
    2. An Instance store Paravirtual AMI
    3. An Amazon EBS-backed Hardware Virtual Machine AMI
    4. An Amazon EBS-backed Paravirtual AMI
  3. You have identified network throughput as a bottleneck on your m1.small EC2 instance when uploading data Into Amazon S3 In the same region. How do you remedy this situation? Add an additional ENI
    1. Change to a larger Instance
    2. Use DirectConnect between EC2 and S3
    3. Use EBS PIOPS on the local volume
  4. You are using an m1.small EC2 Instance with one 300 GB EBS volume to host a relational database. You determined that write throughput to the database needs to be increased. Which of the following approaches can help achieve this? Choose 2 answers
    1. Use an array of EBS volumes (Striping to increase throughput)
    2. Enable Multi-AZ mode.
    3. Place the instance in an Auto Scaling Groups
    4. Add an EBS volume and place into RAID 5 (RAID 5 is not recommended as it provides parity and EBS volumes are already replicated across multiple servers in an Availability Zone for availability and durability, so AWS recommends striping for performance rather than durability)
    5. Increase the size of the EC2 Instance.
    6. Put the database behind an Elastic Load Balancer.
  5. You are tasked with setting up a cluster of EC2 Instances for a NoSQL database. The database requires random read IO disk performance up to a 100,000 IOPS at 4KB block side per node. Which of the following EC2 instances will perform the best for this workload?
    1. A High-Memory Quadruple Extra Large (m2.4xlarge) with EBS-Optimized set to true and a PIOPs EBS volume
    2. A Cluster Compute Eight Extra Large (cc2.8xlarge) using instance storage
    3. High I/O Quadruple Extra Large (hi1.4xlarge) using instance storage
    4. A Cluster GPU Quadruple Extra Large (cg1.4xlarge) using four separate 4000 PIOPS EBS volumes in a RAID 0 configuration
  6. You are implementing a URL whitelisting system for a company that wants to restrict outbound HTTP’S connections to specific domains from their EC2-hosted applications you deploy a single EC2 instance running proxy software and configure It to accept traffic from all subnets and EC2 instances in the VPC. You configure the proxy to only pass through traffic to domains that you define in its whitelist configuration You have a nightly maintenance window or 10 minutes where ail instances fetch new software updates. Each update Is about 200MB In size and there are 500 instances In the VPC that routinely fetch updates After a few days you notice that some machines are failing to successfully download some, but not all of their updates within the maintenance window The download URLs used for these updates are correctly listed in the proxy’s whitelist configuration and you are able to access them manually using a web browser on the instances What might be happening? (Choose 2 answers)
  7. You are running the proxy on an undersized EC2 instance type so network throughput is not sufficient for all instances to download their updates in time.
    1. You have not allocated enough storage to the EC2 instance running me proxy so the network buffer is filling up causing some requests to fall
    2. You are running the proxy in a public subnet but have not allocated enough EIPs to support the needed network throughput through the Internet Gateway (IGW)
    3. You are running the proxy on a affluently-sized EC2 instance in a private subnet and its network throughput is being throttled by a NAT running on an undersized EC2 instance
    4. The route table for the subnets containing the affected EC2 instances is not configured to direct network traffic for the software update locations to the proxy.

References

AWS EC2 AMI (Amazon Machine Image)

Overview

  • An Amazon Machine Image (AMI) provides the information required to launch an instance, which is a virtual server in the cloud.
  • An AMI, selected when you launch an instance, is basically an template and can be used to launch as many instances as needed
  • Within an VPC, you can also launch instances from as many different AMIs as you need
  • An AMI includes the following:
    • A template for the root volume for the instance for e.g. an operating system, an application server, and applications
    • Launch permissions that control which AWS accounts can use the AMI to launch instances for e.g. AWS account ids with whom the AMI is shared
    • A block device mapping that specifies the volumes to attach to the instance when it’s launched
  • AMIs can be either
    • AWS managed, provided and published AMIs
    • Third party or Community provided public custom AMIs
    • Private AMIs created by other AWS accounts and shared with you
    • Private and Custom AMIs created by you
  • AMI lifecycle
    • After you create and register an AMI
    • you can use it to launch new instances. (You can also launch instances from an AMI if the AMI owner grants you launch permissions.)
    • You can copy an AMI to the same region or to different regions.
    • When you are finished launching instance from an AMI, you can deregister the AMI

AMI Types

  • Region & Availability Zone
    • AMIs are specific to a region and if needed in other region must be copied over
  • Operating system
    • AMIS are available in a variety of OS flavors for e.g. linux, windows, ubuntu etc
  • Architecture (32-bit or 64-bit)
  • Launch Permissions
    • Launch permissions define who has access to the AMI
      • Public – Accessible to all AWS accounts
      • Explicit – Shared with specific AWS accounts
      • Private – Owned and available for AMI creator AWS account only
  • Root device storage
    • AMIs can have EBS or Instance store as the root device storage
    • EBS backed
      • EBS volume are independent of the EC2 instance lifecycle and can persist independently
      • EBS backed instances can be stopped without losing the volumes
      • EBS instance can also be persist without losing the volumes on instance termination, if the Delete On Termination flag is disabled
      • EBS backed instances boot up uch faster than the Instance store backed instances as only the parts required to boot the instance needs to be retrieved from the snapshot before the instance is made available
      • AMI creation is much easier for AMIs backed by Amazon EBS. The CreateImage API action creates your Amazon EBS-backed AMI and registers it
    • Instance Store backed
      • Instance store is an ephermal storage and is dependant on the lifecycle of the Instance
      • Instance store is deleted if the instance is terminated or if the EBS backed instance, with attached instance store volumes, is stopped
      • Instance store volumes cannot be stopped
      • Instance store volumes have their AMI in S3 and have higher boot times compared to EBS backed instances, as all the parts have to be retrieved from Amazon S3 before the instance is made available
      • To create Linux AMIs backed by instance store, you must create an AMI from your instance on the instance itself using the Amazon EC2 AMI tools.
    • More detailed @ EBS vs Instance Store

Amazon Linux AMI

Amazon Linux AMI is a supported and maintained Linux image provided by AWS with the following features

  • A stable, secure, and high-performance execution environment for applications running on Amazon EC2.
    • Amazon Linux does not allow remote root SSH by default.
    • Password authentication is disabled to prevent brute-force password attacks.
    • Instances launched using Amazon Linux AMI must be provided with a key pair at launch to enable SSH logins
    • Inbound security group must allow SSH access
    • By default, the only account that can log in remotely using SSH is ec2-user; this account also has sudo privileges.
    • Amazon Linux AMIs are configured to download and install security updates at launch time.
  • Provided at no additional charge to Amazon EC2 users.
  • Repository access to multiple versions of MySQL, PostgreSQL, Python, Ruby, Tomcat, and many more common packages.
  • Updated on a regular basis to include the latest components, and these updates are also made available in the yum repositories for installation on running instances.
  • Includes pre-installed packages to enable easy integration with AWS services, such as the AWS CLI, Amazon EC2 API and AMI tools, the Boto library for Python, and the Elastic Load Balancing tools.

Linux Virtualization Types

  • Linux Amazon Machine Images use one of two types of virtualization: paravirtual (PV) or hardware virtual machine (HVM).
  • Main difference between PV and HVM AMIs is the way in which they boot and whether they can take advantage of special hardware extensions (CPU, network, and storage) for better performance.
  • For the best performance, AWS recommends use of current generation instance types and HVM AMIs when you launch your instances.
  • HVM AMIs
    • HVM AMIs are presented with a fully virtualized set of hardware and boot by executing the master boot record of the root block device of your image.
    • HVM virtualization type provides the ability to run an operating system directly on top of a virtual machine without any modification, as if it were run on the bare-metal hardware.
    • Amazon EC2 host system emulates some or all of the underlying hardware that is presented to the guest.
    • HVM guests, unlike PV guest, can take advantage of hardware extensions that provide fast access to the underlying hardware on the host system.
    • HVM AMIs are required to take advantage of enhanced networking and GPU processing. In order to pass through instructions to specialized network and GPU devices, the OS needs to be able to have access to the native hardware platform; HVM virtualization provides this access.
    • All current generation instance types support HVM AMIs.The CC2, CR1, HI1, and HS1 previous generation instance types support HVM AMIs.
  • PV AMIs
    • PV AMIs boot with a special boot loader called PV-GRUB, which starts the boot cycle and then chain loads the kernel specified in the menu.lst file on your image.
    • Paravirtual guests can run on host hardware that does not have explicit support for virtualization, but they cannot take advantage of special hardware extensions such as enhanced networking or GPU processing.
    • C3 and M3 current generation instance types support PV AMIs. The C1, HI1, HS1, M1, M2, and T1 previous generation instance types support PV AMIs.

Shared AMIs

  • Shared AMI is an AMI that can be created and shared with others for use
  • To get started with EC2, a Shared AMI with all the components needed can be used to start with and then add custom components as and when needed
  • Shared AMI can be risky as Amazon does not perform an detailed checks and vouch for the integrity and security of these AMIs
  • Before using a Shared AMI – Check for any pre-installed credentials that would allow unwanted access to your instance by a third party and no pre-configured remote logging that could transmit sensitive data to a third party.
  • Amazon allows you to share an AMI, by defining launch permissions, to all (making it public) or only to a specific AWS accounts
  • Launch permissions work at the AWS account level only; and can’t be used to restrict specific users within an AWS account.
  • Guidelines for Shared Linux AMIs
    • Update the AMI Tools at Boot Time
      • Update the AMI tools or any software during startup.
      • Take into account the software updates not not break any software and consider the WAN traffic as the downloads will be charged to the AMI user
    • Disable Password-Based Remote Logins for Root
      • Fixed root passwords can be a security risk and needs to be disabled
    • Disable Local Root Access
      • Working with shared AMIs, a best practice is to disable direct root logins
    • Remove SSH Host Key Pairs
      • Remove the existing SSH host key pairs located in /etc/ssh, which forces SSH to generate new unique SSH key pairs when someone launches an instance using your AMI, improving security and reducing the likelihood of “man-in-the-middle” attacks
    • Install Public Key Credentials
      • Amazon EC2 allows users to specify a public-private key pair name when launching an instance.
      • A valid key pair name needs to be provided when launching an instance, the public key, the portion of the key pair that Amazon Ec2 maintains on the server, is made available to the instancethrough an HTTP query against the instance metadata and appended to the autohrized keys
      • Users can launch instances of your AMI with a key pair and log in without requiring a root password
    • Disabling sshd DNS Checks (Optional)
      • Disabling sshd DNS checks slightly weakens your sshd security. However, if DNS resolution fails, SSH logins still work. If you do not disable sshd checks, DNS resolution failures prevent all logins.
    • Identify Yourself
      • AMI is only represented by an account ID without any further information, so it is better to provide more information to help describe the AMI
    • Protect Yourself
      • Don’t store any senstive data or software on the AMI
      • Exclude & Skip any directories which hold the sensitive data or secret information and delete the shell history before creating an AMI

AMI Creation

EBS-Backed Linux AMI

Screen Shot 2016-04-12 at 7.39.18 AM.png

  • Amazon EBS-Backed Linux AMI can be created from the instance directly or from te EBS snapshot
  • Amazon EBS-backed linux AMI creation process :-
    1. Select an AMI #1 similar to what you want to have your new AMI #2
    2. Launch an Instance from AMI #1 and configure the instance accordingly
    3. Stop the instance to ensure data integrity and then create AMI #2 OR
    4. Create an EBS snapshot and then create an AMI #2 from the snapshot
    5. Amazon automatically register the EBS-backed AMI for you
    6. AMI #2 an be used to now launch new instances
  • By default, Amazon EC2 shuts down the instance, takes snapshots of any attached volumes, creates and registers the AMI, and then reboots the instance.
  • Shut down & Reboot of the Instance can be prevented using the No reboot option, however the file system integrity of the created image can’t be guaranteed
  • Amazon EC2 creates snapshots of the instance’s root volume and any other EBS volumes attached to your instance. If any volumes attached to the instance are encrypted, the new AMI only launches successfully on instances that support Amazon EBS encryption
  • For any additional instance-store volumes or EBS volumes, the block device mapping for the new AMI contains information for these volumes, and the block device mappings for instances that you launch from the new AMI automatically contain information for these volumes.
  • While data on EBS volumes persists, the Instance-store volumes specified in the block device mapping for the new instance are new and don’t contain any data from the instance store volumes of the instance you used to create the AMI.
  • Its more efficient to create an EBS-backed AMI with EBS snapshots already taken as the snapshot created during AMI creation is just an incremental one
  • You are charged for the storage of both the AMI and the snapshots

Instance Store-Backed Linux AMI

Screen Shot 2016-04-12 at 7.39.28 AM.png

  • Instance Store-Backed Linux AMI creation process
    1. Select an AMI #1 similar to what you want to have your new AMI #2
    2. Launch an Instance from AMI #1 and configure the instance accordingly
    3. Bundle the Instance. It takes several minutes for the bundling process to complete.
    4. After the process completes, you have a bundle, which consists of an image manifest (image.manifest.xml) and files (image.part.xx) that contain a template for the root volume.
    5. Upload the bundle to your Amazon S3 bucket
    6. Register the Instance Store-backed AMI.
    7. Launching an instance using the new AMI #2, the root volume for the instance is created using the bundle that you uploaded to Amazon S3.
  • Charges are incurred for rhe storage space used by the bundle in Amazon S3 until deleted
  • For additional instance store not root volumes, the block device mapping for the new AMI contains information for these volumes, and the block device mappings for instances that you launch from the new AMI automatically contain information for these volumes.

Deregistering AMI

  • Charges are incurred on the AMI created and they can be deregistered, if not needed.
  • Deregistering an AMI does not delete the EBS snapshots or the bundles in the S3 buckets and have to be removed seperately
  • Once deregistered, new instances cannot be launched from the AMI. However, it does not impact already created instances from the AMI
  • Clean up EBS-Backed AMI
    • Screen Shot 2016-04-12 at 8.11.28 AM.png
    • Deregister the EBS-Backed AMI
    • Delete the EBS Snapshot, as deregistering the AMI doesn’t impact the snapshot
  • Clean up Instance Store-backed AMI
    • Screen Shot 2016-04-12 at 8.11.21 AM.png
    • Deregister the EBS-Backed AMI
    • Delete the bundle from the S3 bucket, as deregistering the AMI doesn’t effect the bundles stored in the S3 bucket

AMIs with Encrypted Snapshots

  • AMIs, with EBS backed snapshots, can be attached with both an encrypted root and data volume
  • AMIs copy image can be used to create AMIs with encrypted snapshots from AMIs with unencrypted snapshots. By default, copy image preserves the encryption status of the snapshots
  • Snapshots can be encrypted with either your default AWS Key Management Service customer master key (CMK), or with a custom key that you specify

AMI Copying

  • Amazon EBS-backed AMIs and instance store-backed AMIs can be copied.
  • Copying an AMI
    • An identical target AMI is created, but with its own unqiue identifier
    • For EBS Backed AMI, an identical but distinct root and data snapshots are created
    • Encryption status of the snapshots are preserved
    • However, Launch permissions, user-defined tags, or Amazon S3 bucket permissions are not copied from the source AMI to the new AMI. After the copy operation is complete, different launch permissions, user-defined tags, and Amazon S3 bucket permissions to the new AMI
  • Source AMI can be deregistered without any impact to the Target AMI
  • You can copy your owned AMIs, AMIs shared with you with proper permissions
  • AMIs are created specific to a region and can be copied within or across regions which can help to aid in consistent global deployment and build highly scalable and available applications
  • AMI copy image can be used to encrypt an AMI from an unencrypted AMI
  • AMI copy image can’t be used to create an unencrypted an AMI from an encrypted AMI

Sample Exam Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  1. A user has launched an EC2 instance from an instance store backed AMI. The infrastructure team wants to create an AMI from the running instance. Which of the below mentioned credentials is not required while creating the AMI?
    1. AWS account ID
    2. 509 certificate and private key
    3. AWS login ID to login to the console
    4. Access key and secret access key
  1. A user has launched an EC2 Windows instance from an instance store backed AMI. The user wants to convert the AMI to an EBS backed AMI. How can the user convert it?
    1. Attach an EBS volume to the instance and unbundle all the AMI bundled data inside the EBS
    2. A Windows based instance store backed AMI cannot be converted to an EBS backed AMI
    3. It is not possible to convert an instance store backed AMI to an EBS backed AMI
    4. Attach an EBS volume and use the copy command to copy all the ephemeral content to the EBS Volume
  1. A user has launched two EBS backed EC2 instances in the US-East-1a region. The user wants to change the zone of one of the instances. How can the user change it?
    1. Stop one of the instances and change the availability zone
    2. The zone can only be modified using the AWS CLI
    3. From the AWS EC2 console, select the Actions – > Change zones and specify new zone
    4. Create an AMI of the running instance and launch the instance in a separate AZ
  1. A user has launched a large EBS backed EC2 instance in the US-East-1a region. The user wants to achieve Disaster Recovery (DR) for that instance by creating another small instance in Europe. How can the user achieve DR?
    1. Copy the running instance using the “Instance Copy” command to the EU region
    2. Create an AMI of the instance and copy the AMI to the EU region. Then launch the instance from the EU AMI
    3. Copy the instance from the US East region to the EU region
    4. Use the “Launch more like this” option to copy the instance from one region to another
  1. A user has launched an EC2 instance store backed instance in the US-East-1a zone. The user created AMI #1 and copied it to the Europe region. After that, the user made a few updates to the application running in the US-East-1a zone. The user makes an AMI#2 after the changes. If the user launches a new instance in Europe from the AMI #1 copy, which of the below mentioned statements is true?
    1. The new instance will have the changes made after the AMI copy as AWS just copies the reference of the original AMI during the copying. Thus, the copied AMI will have all the updated data
    2. The new instance will have the changes made after the AMI copy since AWS keeps updating the AMI
    3. It is not possible to copy the instance store backed AMI from one region to another
    4. The new instance in the EU region will not have the changes made after the AMI copy
  1. George has shared an EC2 AMI created in the US East region from his AWS account with Stefano. George copies the same AMI to the US West region. Can Stefano access the copied AMI of George’s account from the US West region?
    1. No, copy AMI does not copy the permission
    2. It is not possible to share the AMI with a specific account
    3. Yes, since copy AMI copies all private account sharing permissions
    4. Yes, since copy AMI copies all the permissions attached with the AMI

References

AWS EC2 Best Practices

AWS recommends the following to get maximum benefit and satisfaction from EC2

Security & Network

  • Implement the least permissive rules for your security group.
  • Regularly patch, update, and secure the operating system and applications on your instance
  • Launch your instances into a VPC instead of EC2-Classic (If aws account is newly created VPC is used by default)
  • Manage access to AWS resources and APIs using identity federation, IAM users, and IAM roles
  • Establish credential management policies and procedures for creating, distributing, rotating, and revoking AWS access credentials

Storage

  • EC2 supports Instance store and EBS volumes, so its best to understand the implications of the root device type for data persistence, backup, and recovery
  • Use separate Amazon EBS volumes for the operating system (root device) versus your data.
  • Ensure that the data volume (with your data) persists after instance termination
  • Use the instance store available for your instance to only store temporary data. (Remember that the data stored in instance store is deleted when you stop or terminate your instance)
  • If you use instance store for database storage, ensure that you have a cluster with a replication factor that ensures fault tolerance.

Resource Management

  • Use instance metadata and custom resource tags to track and identify your AWS resources
  • View your current limits for Amazon EC2. Plan to request any limit increases in advance of the time that you’ll need them.

Backup & Recovery

  • Regularly back up your instance using Amazon EBS snapshots (not done automatically) or a backup tool.
  • Implement High Availability by deploying critical components of the application across multiple Availability Zones, and replicate the data appropriately
  • Monitor and respond to events.
  • Design your applications to handle dynamic IP addressing when your instance restarts.
  • Implement failover. For a basic solution, you can manually attach a network interface or Elastic IP address to a replacement instance
  • Regularly test the process of recovering your instances and Amazon EBS volumes if they fail.

References

AWS EC2 – Placement Group – Certification

Placement Group Overview

  • A Placement Group is a logical grouping of instances within a single Availability Zone and are recommended for applications that benefits from low network latency, high network throughput, or both.
  • Placement group don’t span Availability Zones
  • Placement group is only available within a single Availability Zone either in the same VPC or peered VPCs
  • Placement group is more of an hint to AWS that the instances need to be launched close together
  • Using placement groups enables applications to participate in a low-latency, 10 Gbps network.

AWS EC2 Placement Group

Placement Group Key Points

  • Provides Low network latency & high network throughput and is recommened for HPC (High Performance Computing)
  • Should be in the same Availability Zone and can’t span multiple AZs
  • Should have unique name within AWS account
  • To provide the lowest latency, and the highest packet-per-second network performance for the placement group, choose an instance type that supports enhanced networking
  • Only Specific Instance types (General Purpose, GPU, Compute, Memory, Storage Optimized – m4.10xlarge, c4.8xlarge, c3.8xlarge, g2.8xlarge, r3.8xlarge, i2.8xlarge, d2.8xlarge)  which support 10 Gigabyte network performance can take full advantage of Placement Group
  • Placement Groups cannot be merged. Instances in one placement group should be terminated, and then relaunched into an other placement group
  • Existing EC2 instance can’t be moved into a Placement Group. For moving an instance,
    • create an AMI from the existing instance,
    • and then launch a new instance from the AMI into a placement group.
  • AWS recommends using the same instance type for all instances in a placement group
  • Placement group can span peered VPCs

Placement Groups Best Practices

  • Use homogenous instance types
  • Launch all the placement group instances at the same time
  • Not a best fit for horizontally scalable web services
  • Ensure there is enough capacity

AWS Certification Exam Practice Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  1. What is a placement group?
    • A collection of Auto Scaling groups in the same Region
    • Feature that enables EC2 instances to interact with each other via high bandwidth, low latency connections
    • A collection of Elastic Load Balancers in the same Region or Availability Zone
    • A collection of authorized Cloud Front edge locations for a distribution
  2. In order to optimize performance for a compute cluster that requires low inter-node latency, which feature in the following list should you use?
    • AWS Direct Connect
    • Placement Groups
    • VPC private subnets
    • EC2 Dedicated Instances
    • Multiple Availability Zones

References

EC2 User Guide – Placement Groups

AWS Bastion Host

Bastion Host Overview

  • Bastion means a structure for Fortification to protect things behind it
  • In AWS, a Bastion host (also referred to as a Jump server) can be used to securely access instances in the private subnets.
  • Bastion host launched in the Public subnets would act as a primary access point from the Internet and acts as a proxy to other instances.

Screen Shot 2016-03-14 at 7.27.49 AM

Key points

  • Bastion host is deployed in the Public subnet and acts as a proxy or a gateway between you and your instances
  • Bastion host is a security measure that helps to reduce attack on your infrastructure and you have to concentrate to hardening a single layer
  • Bastion host allows you to login to instances in the Private subnet securely without having to store the private keys on the Bastion host (using ssh-agent forwarding or RDP gateways)
  • Bastion host security can be further tightened to allow SSH/RDP access from specific trusted IPs or corporate IP ranges
  • Bastion host for your AWS infrastructure shouldn’t be used for any other purpose, as that could open unnecessary security holes
  • Security for all the Instances in the private subnet should be hardened to accept SSH/RDP connections only from the Bastion host
  • Deploy a Bastion host within each Availability Zone for HA, cause if the Bastion instance or the AZ hosting the Bastion server goes down the ability to connect to your private instances is lost completely

Sample Exam Questions

  1. A customer is running a multi-tier web application farm in a virtual private cloud (VPC) that is not connected to their corporate network. They are connecting to the VPC over the Internet to manage all of their Amazon EC2 instances running in both the public and private subnets. They have only authorized the bastion-security-group with Microsoft Remote Desktop Protocol (RDP) access to the application instance security groups, but the company wants to further limit administrative access to all of the instances in the VPC. Which of the following Bastion deployment scenarios will meet this requirement?
    1. Deploy a Windows Bastion host on the corporate network that has RDP access to all instances in the VPC.
    2. Deploy a Windows Bastion host with an Elastic IP address in the public subnet and allow SSH access to the bastion from anywhere.
    3. Deploy a Windows Bastion host with an Elastic IP address in the private subnet, and restrict RDP access to the bastion from only the corporate public IP addresses.
    4. Deploy a Windows Bastion host with an auto-assigned Public IP address in the public subnet, and allow RDP access to the bastion from only the corporate public IP addresses.
  2. You are designing a system that has a Bastion host. This component needs to be highly available without human intervention. Which of the following approaches would you select?
    1. Run the bastion on two instances one in each AZ
    2. Run the bastion on an active Instance in one AZ and have an AMI ready to boot up in the event of failure
    3. Configure the bastion instance in an Auto Scaling group Specify the Auto Scaling group to include multiple AZs but have a min-size of 1 and max-size of 1
    4. Configure an ELB in front of the bastion instance
  3. You’ve been brought in as solutions architect to assist an enterprise customer with their migration of an ecommerce platform to Amazon Virtual Private Cloud (VPC) The previous architect has already deployed a 3- tier VPC. The configuration is as follows: VPC vpc-2f8t>C447
    IGW ig-2d8bc445
    NACL acl-2080c448
    Subnets and Route Tables:
    Web server’s subnet-258Dc44d
    Application server’s subnet-248bc44c
    Database server’s subnet-9189c6f9
    Route Tables:
    rtb-218DC449
    rtb-238bc44b
    Associations:
    Subnet-258bc44d: rtb-2i8bc449
    Subnet-248DC44C rtb-238tX44b
    Subnet-9189c6f9 rtb-238Dc 44b
    You are now ready to begin deploying EC2 instances into the VPC. Web servers must have direct access to the internet Application and database servers cannot have direct access to the internet. Which configuration below will allow you the ability to remotely administer your application and database servers, as well as allow these servers to retrieve updates from the Internet?

    1. Create a bastion and NAT Instance in subnet-248bc44c and add a route from rtb-238bc44b to subnet- 258bc44d.
    2. Add a route from rtD-238bc44D to igw-2d8bc445 and add a bastion and NAT instance within suonet- 248bc44c.
    3. Create a Bastion and NAT Instance in subnet-258bc44d. Add a route from rtb-238bc44b to igw-2d8bc445. And a new NACL that allows access between subnet-258bc44d and subnet-248bc44c.
    4. Create a Bastion and NAT instance in subnet-258Dc44d and add a route from rtb-238Dc44D to the NAT instance.
  4. You are tasked with setting up a Linux bastion host for access to Amazon EC2 instances running in your VPC. Only clients connecting from the corporate external public IP address 72.34.51.100 should have SSH access to the host. Which option will meet the customer requirement?
    1. Security Group Inbound Rule: Protocol – TCP. Port Range – 22, Source 72.34.51.100/32
    2. Security Group Inbound Rule: Protocol – UDP, Port Range – 22, Source 72.34.51.100/32
    3. Network ACL Inbound Rule: Protocol – UDP, Port Range – 22, Source 72.34.51.100/32
    4. Network ACL Inbound Rule: Protocol – TCP, Port Range-22, Source 72.34.51.100/0

AWS EBS Volume Types – Certification

AWS EBS Volume Types

  • AWS provides the following EBS volume types, which differ in performance characteristics and price which can be tailored for storage performance and cost to the needs of the applications:
    • General Purpose SSD Volumes (gp2)
    • Provisioned IOPS SSD Volumes (io1)
    • Magnetic Volumes (standard)

EBS Volume Types Comparision

General Purpose SSD Volumes (gp2)

  • General Purpose SSD volumes offer cost-effective storage that is ideal for a broad range of workloads.
  • General Purpose SSD volumes deliver single-digit millisecond latencies
  • General Purpose SSD volumes can range in size from 1 GiB to 16 TiB.
  • General Purpose SSD volumes has a maximum throughput of 160 MiB/s (at 214 GiB and larger).
  • GP2 provides a baseline performance of 3 IOPS/GiB
  • GP2 provides the ability to burst to 3,000 IOPS for extended periods of time for volume size less then 1 TiB and up to a maximum of 10,000 IOPS (at 3,334 GiB).
  • If the volume performance is frequently limited to the baseline level (due to an empty I/O credit balance),
    • consider using a larger General Purpose SSD volume (with a higher baseline performance level) or
    • switching to a Provisioned IOPS SSD volume for workloads that require sustained IOPS performance greater than 10,000 IOPS.

I/O Credits and Burst Performance

  • I/O credits represent the available bandwidth that your General Purpose SSD volume can use to burst large amounts of I/O when more than the baseline performance is needed.
  • General Purpose SSD volume performance is governed by volume size, which dictates the baseline performance level of the volume for e.g. 100 GiB volume has a 300 IOPS @ 3 IOPS/GiB
  • General Purpose SSD volume size also determines how quickly it accumulates I/O credits for e.g. 100 GiB with a performance of 300 IOPS can accumulate 180K IOPS/10 mins (300 * 60 * 10).
  • Larger volumes have higher baseline performance levels and accumulate I/O credits faster for e.g. 1 TiB has a baseline performance of 3000 IOPS
  • More credits the volume has for I/O, the more time it can burst beyond its baseline performance level and the better it performs when more performance is needed for e.g. 300 GiB volume with 180K I/O credit can burst @ 3000 IOPS for 1 minute (180K/3000)
  • Each volume receives an initial I/O credit balance of 5,400,000 I/O credits, which is enough to sustain the maximum burst performance of 3,000 IOPS for 30 minutes.
  • Initial credit balance is designed to provide a fast initial boot cycle for boot volumes and a good bootstrapping experience for other applications.
  • Each volume can accumulate I/O credits over a period of time which can be to burst to the required performance level, up to a max of 3,000 IOPS
  • Unused I/O credit cannot go beyond 54,00,000 I/O credits.

IOPS vs Volume size

  • Volumes till 1 TiB can burst up to 3000 IOPS over an above its baseline performance
  • Volumes larger than 1 TiB have a baseline performance that is already equal or greater than the maximum burst performance, and their I/O credit balance never depletes.
  • Baseline performance cannot be beyond 10000 IOPS for General Purpose SSD volumes and this limit is reached @ 3333 GiB

IOPS vs Volume Size

Baseline Performance

  • Formula – 3 IOPS i.e. GiB * 3
  • Calculation example
    • 1 GiB volume size =  3 IOPS (1 * 3 IOPS)
    • 250 GiB volume size = 750 IOPS (250* 3 IOPS)

Maximum burst duration @ 3000 IOPS

  • How much time can 5400000 IO credit be sustained @ the burst performance of 3000 IOPS. Subtract the baseline performance from 3000 IOPS which would be contributed by the volume size
  • Formula – 5400000/(3000 – Baseline performance)
  • Calculation example
    • 1 GiB volume size @ 3000 IOPS with 5400000 the burst performance can be maintained for 5400000/(3000-3) = 1802 secs
    • 250 GiB volume size @ 3000 IOPS with 5400000 the burst performance can be maintained for 5400000/(3000-250) = 2400 secs

Time to fill the 5400000 I/O credit balance

  • Formula – 5400000/Baseline performance
  • Calculation
    • 1 GiB volume size @ 3 IOPS would require 5400000/3 = 1800000 secs
    • 250 GiB volume size @ 750 IOPS would require 5400000/750 = 7200 secs

Provisioned IOPS SSD Volumes

  • Provisioned IOPS SSD volumes are designed to meet the needs of I/O intensive workloads, particularly database workloads, that are sensitive to storage performance and consistency in random access I/O throughput.
  • IOPS rate can be specified when the volume is created, and EBS delivers within 10 percent of the provisioned IOPS performance 99.9 percent of the time over a given year.
  • Provisioned IOPS SSD volume can range in size from 4 GiB to 16 TiB
  • Provisioned IOPS SSD volumes have a throughput limit range of 256 KiB for each IOPS provisioned, up to a maximum of 320 MiB/s (at 1,280 IOPS).
  • Provisioned IOPS SSD volume can be provision up to 20,000 IOPS per volume. The ratio of IOPS provisioned to the volume size requested can be a maximum of 30; for example, a volume with 3,000 IOPS must be at least 100 GiB.
  • Provisioned IOPS SSD volumes can be striped together in a RAID configuration for larger size and greater performance over 20000 IOPS

Magnetic Volumes (standard)

Magnetic volumes provide the lowest cost per gigabyte of all EBS volume types. Magnetic volumes are backed by magnetic drives and are ideal for workloads performing sequential reads, workloads where data is accessed infrequently, and scenarios where the lowest storage cost is important.

  • Magnetic volumes can range in size from1 GiB to 1 TiB
  • These volumes deliver approximately 100 IOPS on average, with burst capability of up to hundreds of IOPS
  • Magnetic volumes can be striped together in a RAID configuration for larger size and greater performance.

References

AWS – EC2 – Troubleshooting Connecting to an Instance

  1. Verify the Security groups are properly configured to allow ssh access from the ip to the EC2 instance. For Security groups, Inbound traffic from the public ip address should be enabled
  2. Verify the NACLs are properly configured to allow ssh access from the ip to the EC2 instance. For NACLs, Inbound traffic from the public ip address should be enabled as well as the Outbound traffic for the response should be enabled
  3. Verify you are using the private key file that corresponds to the key pair that you selected when you launched the instance
  4. Verify you are connecting with the appropriate user name for your AMI. Mind the user names used to connect to the EC2 instance are different depending upon the AMI (which also determines the OS for the Instance)
  5. Private User key file is not recognized by the Server

Exam Scenario Question

  1. You try to connect via SSH to a newly created Amazon EC2 instance and get one of the following error messages: “Network error: Connection timed out” or “Error connecting to instance], reason: -> Connection timed out: connect,” You have confirmed that the network and security group rules are configured correctly and the instance is passing status checks. What steps should you take to identify the source of the behavior? Choose 2 answers
    • Verify that the private key file corresponds to the Amazon EC2 key pair assigned at launch.
    • Verify that your IAM user policy has permission to launch Amazon EC2 instances.
    • Verify that you are connecting with the appropriate user name for your AMI.
    • Verify that the Amazon EC2 Instance was launched with the proper IAM role.
    • Verify that your federation trust to AWS has been established.

 

References

EC2 User Guide