Author: jayendrapatil

AWS Certified Solution Architect – Professional Exam Learning Path (Obsolete)

~ Last updated on : ~ jayendrapatil ~ 195 Comments

AWS Certified Solution Architect – Professional Exam Learning Path

NOTE : Refer to Solutions Architect – Professional SAP-C01 learning path for the latest exam.

I recently cleared the AWS Certified Solution Architect Professional Exam with 93% after almost 2 months of preparation

Topic Level Scoring:
1.0 High Availability and Business Continuity: 100%
2.0 Costing: 75%
3.0 Deployment Management: 100%
4.0 Network Design: 85%
5.0 Data Storage: 90%
6.0 Security: 92%
7.0 Scalability & Elasticity: 100%
8.0 Cloud Migration & Hybrid Architecture: 85%

AWS Solution Architect – Professional exam is quite an exhaustive exam with 77 questions in 180 minutes and covers a lot of AWS services and the combinations how they work and integrate together. However, the questions are bit old and has not kept pace with the fast changing AWS enhancements

If looking for Associate Preparation Guide, please refer

Refer to the AWS Solution Architect – Professional Exam Blue Print

AWS Solution Architect – Professional exam basically validates the following

  • Identify and gather requirements in order to define a solution to be built on AWS
  • Evolve systems by introducing new services and features
  • Assess the tradeoffs and implications of architectural decisions and choices for applications deployed in AWS
  • Design an optimal system by meeting project requirements while maximizing characteristics such as scalability, security, reliability, durability, and cost effectiveness
  • Evaluate project requirements and make recommendations for implementation, deployment, and provisioning applications on AWS
  • Provide best practice and architectural guidance over the lifecycle of a project
AWS Solutions Architect - Professional Exam Break Up

AWS Cloud Computing Whitepapers

AWS Certified Solution Architect Professional Exam Contents

Domain 1.0: High Availability and Business Continuity

  • 1.1 Demonstrate ability to architect the appropriate level of availability based on stakeholder requirements
  • 1.2 Demonstrate ability to implement DR for systems based on RPO and RTO
  • 1.3 Determine appropriate use of multi-Availability Zones vs. multi-Region architectures
  • 1.4 Demonstrate ability to implement self-healing capabilities
  • 1.5 High Availability vs. Fault Tolerance

Domain 2.0: Costing

  • 2.1 Demonstrate ability to make architectural decisions that minimize and optimize infrastructure cost
  • 2.2 Apply the appropriate AWS account and billing set-up options based on scenario
  • 2.3 Ability to compare and contrast the cost implications of different architectures

Domain 3.0: Deployment Management

  • 3.1 Ability to manage the lifecycle of an application on AWS
  • 3.2 Demonstrate ability to implement the right architecture for development, testing, and staging environments
  • 3.3 Position and select most appropriate AWS deployment mechanism based on scenario

Domain 4.0: Network Design for a complex large scale deployment

  • 4.1 Demonstrate ability to design and implement networking features of AWS
  • 4.2 Demonstrate ability to design and implement connectivity features of AWS

Domain 5.0: Data Storage for a complex large scale deployment

  • 5.1 Demonstrate ability to make architectural trade off decisions involving storage options
    • includes Storage Options patterns and anti patterns for S3, EBS,  Instance Store
  • 5.2 Demonstrate ability to make architectural trade off decisions involving database options
    • includes Storage Options patterns and anti patterns RDS, DynamoDB, Database on EC2
  • 5.3 Demonstrate ability to implement the most appropriate data storage architecture
  • 5.4 Determine use of synchronous versus asynchronous replication

Domain 6.0: Security

Domain 7.0: Scalability and Elasticity

  • 7.1 Demonstrate the ability to design a loosely coupled system
  • 7.2 Demonstrate ability to implement the most appropriate front-end scaling architecture
  • 7.3 Demonstrate ability to implement the most appropriate middle-tier scaling architecture
  • 7.4 Demonstrate ability to implement the most appropriate data storage scaling architecture
  • 7.5 Determine trade-offs between vertical and horizontal scaling
    • includes basic understanding of horizontal scaling is scale in/out and vertical scaling is scale up/down

Domain 8.0: Cloud Migration and Hybrid Architecture

Other services like SWF manual task and ability to retry, SNS Mobile Push, SES for durable email, Elastic Transcoder, CloudSearch for search, Data Pipeline for disaster recovery, CloudWatch provides durable storage for logs, EMR how to improve performance

AWS Solution Architect – Professional Exam Resources

AWS Certification Exam Cheat Sheet

~ Last updated on : ~ jayendrapatil ~ 57 Comments

AWS Certification Exam Cheat Sheet

AWS Certification Exams cover a lot of topics and a wide range of services with minute details for features, patterns, anti patterns and their integration with other services. This blog post is just to have a quick summary of all the services and key points for a quick glance before you appear for the exam

AWS Global Infrastructure

AWS Region, AZs, Edge locations

  • Each region is a separate geographic area, completely independent, isolated from the other regions & helps achieve the greatest possible fault tolerance and stability
  • Communication between regions is across the public Internet
  • Each region has multiple Availability Zones
  • Each AZ is physically isolated, geographically separated from each other and designed as an independent failure zone
  • AZs are connected with low-latency private links (not public internet)
  • Edge locations are locations maintained by AWS through a worldwide network of data centers for the distribution of content to reduce latency.

AWS Local Zones

  • AWS Local Zones place select AWS services closer to end-users, which allows running highly-demanding applications that require single-digit millisecond latencies to the end-users such as media & entertainment content creation, real-time gaming, machine learning etc.
  • AWS Local Zones provide a high-bandwidth, secure connection between local workloads and those running in the AWS Region, allowing you to seamlessly connect to the full range of in-region services through the same APIs and tool sets.

AWS Wavelength

  • AWS infrastructure deployments embed AWS compute and storage services within the telecommunications providers’ datacenters and help seamlessly access the breadth of AWS services in the region.
  • AWS Wavelength brings services to the edge of the 5G network, without leaving the mobile provider’s network reducing the extra network hops, minimizing the latency to connect to an application from a mobile device.

AWS Outposts

  • AWS Outposts bring native AWS services, infrastructure, and operating models to virtually any data center, co-location space, or on-premises facility.
  • AWS Outposts is designed for connected environments and can be used to support workloads that need to remain on-premises due to low latency, compliance or local data processing needs.

Refer details @ AWS Global Infrastructure

AWS Services

AWS Organizations

  • AWS Organizations offers policy-based management for multiple AWS accounts
  • Organizations allows creation of groups of accounts and then apply policies to those groups
  • Organizations enables you to centrally manage policies across multiple accounts, without requiring custom scripts and manual processes.
  • Organizations helps simplify the billing for multiple accounts by enabling the setup of a single payment method for all the accounts in the organization through consolidated billing

Consolidate Billing

  • Paying account with multiple linked accounts
  • Paying account is independent and should be only used for billing purpose
  • Paying account cannot access resources of other accounts unless given exclusively access through Cross Account roles
  • All linked accounts are independent and soft limit of 20
  • One bill per AWS account
  • provides Volume pricing discount for usage across the accounts
  • allows unused Reserved Instances to be applied across the group
  • Free tier is not applicable across the accounts

Tags & Resource Groups

  • are metadata, specified as key/value pairs with the AWS resources
  • are for labelling purposes and helps managing, organizing resources
  • can be inherited when created resources created from Auto Scaling, Cloud Formation, Elastic Beanstalk etc
  • can be used for
    • Cost allocation to categorize and track the AWS costs
    • Conditional Access Control policy to define permission to allow or deny access on resources based on tags
  • Resource Group is a collection of resources that share one or more tags

IDS/IPS

  • Promiscuous mode is not allowed, as AWS and Hypervisor will not deliver any traffic to instances this is not specifically addressed to the instance
  • IDS/IPS strategies
    • Host Based Firewall – Forward Deployed IDS where the IDS itself is installed on the instances
    • Host Based Firewall – Traffic Replication where IDS agents installed on instances which send/duplicate the data to a centralized IDS system
    • In-Line Firewall – Inbound IDS/IPS Tier (like a WAF configuration) which identifies and drops suspect packets

DDOS Mitigation

  • Minimize the Attack surface
    • use ELB/CloudFront/Route 53 to distribute load
    • maintain resources in private subnets and use Bastion servers
  • Scale to absorb the attack
    • scaling helps buy time to analyze and respond to an attack
    • auto scaling with ELB to handle increase in load to help absorb attacks
    • CloudFront, Route 53 inherently scales as per the demand
  • Safeguard exposed resources
    • user Route 53 for aliases to hide source IPs and Private DNS
    • use CloudFront geo restriction and Origin Access Identity
    • use WAF as part of the infrastructure
  • Learn normal behavior (IDS/WAF)
    • analyze and benchmark to define rules on normal behavior
    • use CloudWatch
  • Create a plan for attacks

AWS Services Region, AZ, Subnet VPC limitations

  • Services like IAM (user, role, group, SSL certificate), Route 53, STS are Global and available across regions
  • All other AWS services are limited to Region or within Region and do not exclusively copy data across regions unless configured
  • AMI are limited to region and need to be copied over to other region
  • EBS volumes are limited to the Availability Zone, and can be migrated by creating snapshots and copying them to another region
  • Reserved instances are limited to Availability Zone and (can be migrated to other Availability Zone now) cannot be migrated to another region
  • RDS instances are limited to the region and can be recreated in a different region by either using snapshots or promoting a Read Replica
  • Placement groups are limited to the Availability Zone
    • Cluster Placement groups are limited to single Availability Zones
    • Spread Placement groups can span across multiple Availability Zones
  • S3 data is replicated within the region and can be move to another region using cross region replication
  • DynamoDB maintains data within the region can be replicated to another region using DynamoDB cross region replication (using DynamoDB streams) or Data Pipeline using EMR (old method)
  • Redshift Cluster span within an Availability Zone only, and can be created in other AZ using snapshots

Disaster Recovery Whitepaper

  • RTO is the time it takes after a disruption to restore a business process to its service level and RPO acceptable amount of data loss measured in time before the disaster occurs
  • Techniques (RTO & RPO reduces and the Cost goes up as we go down)
    • Backup & Restore – Data is backed up and restored, within nothing running
    • Pilot light – Only minimal critical service like RDS is running and rest of the services can be recreated and scaled during recovery
    • Warm Standby – Fully functional site with minimal configuration is available and can be scaled during recovery
    • Multi-Site – Fully functional site with identical configuration is available and processes the load
  • Services
    • Region and AZ to launch services across multiple facilities
    • EC2 instances with the ability to scale and launch across AZs
    • EBS with Snapshot to recreate volumes in different AZ or region
    • AMI to quickly launch preconfigured EC2 instances
    • ELB and Auto Scaling to scale and launch instances across AZs
    • VPC to create private, isolated section
    • Elastic IP address as static IP address
    • ENI with pre allocated Mac Address
    • Route 53 is highly available and scalable DNS service to distribute traffic across EC2 instances and ELB in different AZs and regions
    • Direct Connect for speed data transfer (takes time to setup and expensive then VPN)
    • S3 and Glacier (with RTO of 3-5 hours) provides durable storage
    • RDS snapshots and Multi AZ support and Read Replicas across regions
    • DynamoDB with cross region replication
    • Redshift snapshots to recreate the cluster
    • Storage Gateway to backup the data in AWS
    • Import/Export to move large amount of data to AWS (if internet speed is the bottleneck)
    • CloudFormation, Elastic Beanstalk and Opsworks as orchestration tools for automation and recreate the infrastructure

 

AWS Certification – Application Services – Cheat Sheet

~ Last updated on : ~ jayendrapatil ~ 9 Comments

SQS

  • extremely scalable queue service and potentially handles millions of messages
  • helps build fault tolerant, distributed loosely coupled applications
  • stores copies of the messages on multiple servers for redundancy and high availability
  • guarantees At-Least-Once Delivery, but does not guarantee Exact One Time Delivery which might result in duplicate messages (Not true anymore with the introduction of FIFO queues)
  • does not maintain or guarantee message order, and if needed sequencing information needs to be added to the message itself (Not true anymore with the introduction of FIFO queues)
  • supports multiple readers and writers interacting with the same queue as the same time
  • holds message for 4 days, by default, and can be changed from 1 min – 14 days after which the message is deleted
  • message needs to be explicitly deleted by the consumer once processed
  • allows send, receive and delete batching which helps club up to 10 messages in a single batch while charging price for a single message
  • handles visibility of the message to multiple consumers using Visibility Timeout, where the message once read by a consumer is not visible to the other consumers till the timeout occurs
  • can handle load and performance requirements by scaling the worker instances as the demand changes (Job Observer pattern)
  • message sample allowing short and long polling
    • returns immediately vs waits for fixed time for e.g. 20 secs
    • might not return all messages as it samples a subset of servers vs returns all available messages
    • repetitive vs helps save cost with long connection
  • supports delay queues to make messages available after a certain delay, can you used to differentiate from priority queues
  • supports dead letter queues, to redirect messages which failed to process after certain attempts instead of being processed repeatedly
  • Design Patterns
    • Job Observer Pattern can help coordinate number of EC2 instances with number of job requests (Queue Size) automatically thus Improving cost effectiveness and performance
    • Priority Queue Pattern can be used to setup different queues with different handling either by delayed queues or low scaling capacity for handling messages in lower priority queues

SNS

  • delivery or sending of messages to subscribing endpoints or clients
  • publisher-subscriber model
  • Producers and Consumers communicate asynchronously with subscribers by producing and sending a message to a topic
  • supports Email (plain or JSON), HTTP/HTTPS, SMS, SQS
  • supports Mobile Push Notifications to push notifications directly to mobile devices with services like Amazon Device Messaging (ADM), Apple Push Notification Service (APNS), Google Cloud Messaging (GCM) etc. supported
  • order is not guaranteed and No recall available
  • integrated with Lambda to invoke functions on notifications
  • for Email notifications, use SNS or SES directly, SQS does not work

SWF

  • orchestration service to coordinate work across distributed components
  • helps define tasks, stores, assigns tasks to workers, define logic, tracks and monitors the task and maintains workflow state in a durable fashion
  • helps define tasks which can be executed on AWS cloud or on-premises
  • helps coordinating tasks across the application which involves managing intertask dependencies, scheduling, and concurrency in accordance with the logical flow of the application
  • supports built-in retries, timeouts and logging
  • supports manual tasks
  • Characteristics
    • deliver exactly once
    • uses long polling, which reduces number of polls without results
    • Visibility of task state via API
    • Timers, signals, markers, child workflows
    • supports versioning
    • keeps workflow history for a user-specified time
  • AWS SWF vs AWS SQS
    • task-oriented vs message-oriented
    • track of all tasks and events vs needs custom handling

SES

  • highly scalable and cost-effective email service
  • uses content filtering technologies to scan outgoing emails to check standards and email content for spam and malware
  • supports full fledged emails to be sent as compared to SNS where only the message is sent in Email
  • ideal for sending bulk emails at scale
  • guarantees first hop
  • eliminates the need to support custom software or applications to do heavy lifting of email transport

AWS Networking & Content Delivery Services Cheat Sheet

AWS Networking & Content Delivery Services
~ Last updated on : ~ jayendrapatil ~ 27 Comments

AWS Networking & Content Delivery Services Cheat Sheet

AWS Networking & Content Delivery Services

Virtual Private Cloud – VPC

  • helps define a logically isolated dedicated virtual network within the AWS
  • provides control of IP addressing using CIDR block from a minimum of /28 to a maximum of /16 block size
  • supports IPv4 and IPv6 addressing
  • cannot be extended once created
  • can be extended by associating secondary IPv4 CIDR blocks to VPC
  • Components
    • Internet gateway (IGW) provides access to the Internet
    • Virtual gateway (VGW) provides access to the on-premises data center through VPN and Direct Connect connections
    • VPC can have only one IGW and VGW
    • Route tables determine network traffic routing from the subnet
    • Ability to create a subnet with VPC CIDR block
    • A Network Address Translation (NAT) server provides outbound Internet access for EC2 instances in private subnets
    • Elastic IP addresses are static, persistent public IP addresses
    • Instances launched in the VPC will have a Private IP address and can have a Public or an Elastic IP address associated with it
    • Security Groups and NACLs help define security
    • Flow logs – Capture information about the IP traffic going to and from network interfaces in your VPC
  • Tenancy option for instances
    • shared, by default, allows instances to be launched on shared tenancy
    • dedicated allows instances to be launched on a dedicated hardware
  • Route Tables
    • defines rules, termed as routes, which determine where network traffic from the subnet would be routed
    • Each VPC has a Main Route table and can have multiple custom route tables created
    • Every route table contains a local route that enables communication within a VPC which cannot be modified or deleted
    • Route priority is decided by matching the most specific route in the route table that matches the traffic
  • Subnets
    • map to AZs and do not span across AZs
    • have a CIDR range that is a portion of the whole VPC.
    • CIDR ranges cannot overlap between subnets within the VPC.
    • AWS reserves 5 IP addresses in each subnet – first 4 and last one
    • Each subnet is associated with a route table which define its behavior
      • Public subnets – inbound/outbound Internet connectivity via IGW
      • Private subnets – outbound Internet connectivity via an NAT or VGW
      • Protected subnets – no outbound connectivity and used for regulated workloads
  • Elastic Network Interface (ENI)
    • a default ENI, eth0, is attached to an instance which cannot be detached with one or more secondary detachable ENIs (eth1-ethn)
    • has primary private, one or more secondary private, public, Elastic IP address, security groups, MAC address and source/destination check flag attributes associated
    • AN ENI in one subnet can be attached to an instance in the same or another subnet, in the same AZ and the same VPC
    • Security group membership of an ENI can be changed
    • with pre-allocated Mac Address can be used for applications with special licensing requirements
  • Security Groups vs NACLs – Network Access Control Lists
    • Stateful vs Stateless
    • At instance level vs At subnet level
    • Only allows Allow rule vs Allows both Allow and Deny rules
    • Evaluated as a Whole vs Evaluated in defined Order
  • Elastic IP
    • is a static IP address designed for dynamic cloud computing.
    • is associated with an AWS account, and not a particular instance
    • can be remapped from one instance to another instance
    • is charged for non-usage, if not linked for any instance or instance associated is in a stopped state
  • NAT
    • allows internet access to instances in the private subnets.
    • performs the function of both address translation and port address translation (PAT)
    • needs source/destination check flag to be disabled as it is not the actual destination of the traffic for NAT Instance.
    • NAT gateway is an AWS managed NAT service that provides better availability, higher bandwidth, and requires less administrative effort
    • are not supported for IPv6 traffic
    • NAT Gateway supports private NAT with fixed private IPs.
  • Egress-Only Internet Gateways
    • outbound communication over IPv6 from instances in the VPC to the Internet, and prevents the Internet from initiating an IPv6 connection with your instances
    • supports IPv6 traffic only
  • Shared VPCs
    • allows multiple AWS accounts to create their application resources, such as EC2 instances, RDS databases, Redshift clusters, and AWS Lambda functions, into shared, centrally-managed VPCs

VPC Peering

  • allows routing of traffic between the peer VPCs using private IP addresses with no IGW or VGW required.
  • No single point of failure and bandwidth bottlenecks
  • supports inter-region VPC peering
  • Limitations
    • IP space or CIDR blocks cannot overlap
    • cannot be transitive
    • supports a one-to-one relationship between two VPCs and has to be explicitly peered.
    • does not support edge-to-edge routing.
    • supports only one connection between any two VPCs
  • Private DNS values cannot be resolved
  • Security groups from peered VPC can now be referred to, however, the VPC should be in the same region.

VPC Endpoints

  • enables private connectivity from VPC to supported AWS services and VPC endpoint services powered by PrivateLink
  • does not require a public IP address, access over the Internet, NAT device, a VPN connection, or Direct Connect
  • traffic between VPC & AWS service does not leave the Amazon network
  • are virtual devices.
  • are horizontally scaled, redundant, and highly available VPC components that allow communication between instances in the VPC and services without imposing availability risks or bandwidth constraints on the network traffic.
  • Gateway Endpoints
    • is a gateway that is a target for a specified route in the route table, used for traffic destined to a supported AWS service.
    • only S3 and DynamoDB are currently supported
  • Interface Endpoints OR Private Links
    • is an elastic network interface with a private IP address that serves as an entry point for traffic destined to a supported service
    • supports services include AWS services, services hosted by other AWS customers and partners in their own VPCs (referred to as endpoint services), and supported AWS Marketplace partner services.
    • Private Links
      • provide fine-grained access control
      • provides a point-to-point integration.
      • supports overlapping CIDR blocks.
      • supports transitive routing

CloudFront

  • provides low latency and high data transfer speeds for the distribution of static, dynamic web, or streaming content to web users.
  • delivers the content through a worldwide network of data centers called Edge Locations or Point of Presence (PoPs)
  • keeps persistent connections with the origin servers so that the files can be fetched from the origin servers as quickly as possible.
  • dramatically reduces the number of network hops that users’ requests must pass through
  • supports multiple origin server options, like AWS hosted service for e.g. S3, EC2, ELB, or an on-premise server, which stores the original, definitive version of the objects
  • single distribution can have multiple origins and Path pattern in a cache behavior determines which requests are routed to the origin
  • Web distribution supports static, dynamic web content, on-demand using progressive download & HLS, and live streaming video content
  • supports HTTPS using either
    • dedicated IP address, which is expensive as a dedicated IP address is assigned to each CloudFront edge location
    • Server Name Indication (SNI), which is free but supported by modern browsers only with the domain name available in the request header
  • For E2E HTTPS connection,
    • Viewers -> CloudFront needs either a certificate issued by CA or ACM
    • CloudFront -> Origin needs a certificate issued by ACM for ELB and by CA for other origins
  •  Security
    • Origin Access Identity (OAI) can be used to restrict the content from S3 origin to be accessible from CloudFront only
    • supports Geo restriction (Geo-Blocking) to whitelist or blacklist countries that can access the content
    • Signed URLs 
      • to restrict access to individual files, for e.g., an installation download for your application.
      • users using a client, for e.g. a custom HTTP client, that doesn’t support cookies
    • Signed Cookies
      • provide access to multiple restricted files, for e.g., video part files in HLS format or all of the files in the subscribers’ area of a website.
      • don’t want to change the current URLs
    • integrates with AWS WAF, a web application firewall that helps protect web applications from attacks by allowing rules configured based on IP addresses, HTTP headers, and custom URI strings
  • supports GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE to get object & object headers, add, update, and delete objects
    • only caches responses to GET and HEAD requests and, optionally, OPTIONS requests
    • does not cache responses to PUT, POST, PATCH, DELETE request methods and these requests are proxied back to the origin
  • object removal from the cache
    • would be removed upon expiry (TTL) from the cache, by default 24 hrs
    • can be invalidated explicitly, but has a cost associated, however, might continue to see the old version until it expires from those caches
    • objects can be invalidated only for Web distribution
    • use versioning or change object name, to serve a different version
  • supports adding or modifying custom headers before the request is sent to origin which can be used to
    • validate if a user is accessing the content from CDN
    • identifying CDN from which the request was forwarded, in case of multiple CloudFront distributions
    • for viewers not supporting CORS to return the Access-Control-Allow-Origin header for every request
  • supports Partial GET requests using range header to download objects in smaller units improving the efficiency of partial downloads and recovery from partially failed transfers
  • supports compression to compress and serve compressed files when viewer requests include Accept-Encoding: gzip in the request header
  • supports different price classes to include all regions, or only the least expensive regions and other regions without the most expensive regions
  • supports access logs which contain detailed information about every user request for both web and RTMP distribution

AWS VPN

  • AWS Site-to-Site VPN provides secure IPSec connections from on-premise computers or services to AWS over the Internet
  • is cheap, and quick to set up however it depends on the Internet speed
  • delivers high availability by using two tunnels across multiple Availability Zones within the AWS global network
  • VPN requires a Virtual Gateway – VGW and Customer Gateway – CGW for communication
  • VPN connection is terminated on VGW on AWS
  • Only one VGW can be attached to a VPC at a time
  • VGW supports both static and dynamic routing using Border Gateway Protocol (BGP)
  • VGW supports AWS-256 and SHA-2 for data encryption and integrity
  • AWS Client VPN is a managed client-based VPN service that enables secure access to AWS resources and resources in the on-premises network.
  • AWS VPN does not allow accessing the Internet through IGW or NAT Gateway, peered VPC resources, or VPC Gateway Endpoints from on-premises.
  • AWS VPN allows access accessing the Internet through NAT Instance and VPC Interface Endpoints from on-premises.

Direct Connect

  • is a network service that uses a private dedicated network connection to connect to AWS services.
  • helps reduce costs (long term), increases bandwidth, and provides a more consistent network experience than internet-based connections. 
  • supports Dedicated and Hosted connections
    • Dedicated connection is made through a 1 Gbps, 10 Gbps, or 100 Gbps Ethernet port dedicated to a single customer.
    • Hosted connections are sourced from an AWS Direct Connect Partner that has a network link between themselves and AWS. 
  • provides Virtual Interfaces
    • Private VIF to access instances within a VPC via VGW
    • Public VIF to access non VPC services
  • requires time to setup probably months, and should not be considered as an option if the turnaround time is less
  • does not provide redundancy, use either second direct connection or IPSec VPN connection
  • Virtual Private Gateway is on the AWS side and Customer Gateway is on the Customer side
  • route propagation is enabled on VGW and not on CGW
  • A link aggregation group (LAG) is a logical interface that uses the link aggregation control protocol (LACP) to aggregate multiple dedicated connections at a single AWS Direct Connect endpoint and treat them as a single, managed connection
  • Direct Connect vs VPN IPSec
    • Expensive to Setup and Takes time vs Cheap & Immediate
    • Dedicated private connections vs Internet
    • Reduced data transfer rate vs Internet data transfer cost
    • Consistent performance vs Internet inherent variability
    • Do not provide Redundancy vs Provides Redundancy

Route 53

  • provides highly available and scalable DNS, Domain Registration Service, and health-checking web services
  • Reliable and cost-effective way to route end users to Internet applications
  • Supports multi-region and backup architectures for High availability. ELB is limited to region and does not support multi-region HA architecture.
  • supports private Intranet facing DNS service
  • internal resource record sets only work for requests originating from within the VPC and currently cannot extend to on-premise
  • Global propagation of any changes made to the DN records within ~ 1min
  • supports Alias resource record set is a Route 53 extension to DNS.
    • It’s similar to a CNAME resource record set, but supports both for root domain – zone apex e.g. example.com, and for subdomains for e.g. www.example.com.
    • supports ELB load balancers, CloudFront distributions, Elastic Beanstalk environments, API Gateways, VPC interface endpoints, and  S3 buckets that are configured as websites.
  • CNAME resource record sets can be created only for subdomains and cannot be mapped to the zone apex record
  • supports Private DNS to provide an authoritative DNS within the VPCs without exposing the DNS records (including the name of the resource and its IP address(es) to the Internet.
  • Split-view (Split-horizon) DNS enables mapping the same domain publicly and privately. Requests are routed as per the origin.
  • Routing policy
    • Simple routing – simple round-robin policy
    • Weighted routing – assign weights to resource records sets to specify the proportion for e.g. 80%:20%
    • Latency based routing – helps improve global applications as requests are sent to the server from the location with minimal latency, is based on the latency and cannot guarantee users from the same geography will be served from the same location for any compliance reasons
    • Geolocation routing – Specify geographic locations by continent, country, the state limited to the US, is based on IP accuracy
    • Geoproximity routing policy – Use to route traffic based on the location of the resources and, optionally, shift traffic from resources in one location to resources in another.
    • Multivalue answer routing policy – Use to respond to DNS queries with up to eight healthy records selected at random.
    • Failover routing – failover to a backup site if the primary site fails and becomes unreachable
  • Weighted, Latency and Geolocation can be used for Active-Active while Failover routing can be used for Active-Passive multi-region architecture
  • Traffic Flow is an easy-to-use and cost-effective global traffic management service. Traffic Flow supports versioning and helps create policies that route traffic based on the constraints they care most about, including latency, endpoint health, load, geoproximity, and geography.
  • Route 53 Resolver is a regional DNS service that helps with hybrid DNS
    • Inbound Endpoints are used to resolve DNS queries from an on-premises network to AWS
    • Outbound Endpoints are used to resolve DNS queries from AWS to an on-premises network

AWS Global Accelerator

  • is a networking service that helps you improve the availability and performance of the applications to global users.
  • utilizes the Amazon global backbone network, improving the performance of the applications by lowering first-byte latency, and jitter, and increasing throughput as compared to the public internet. 
  • provides two static IP addresses serviced by independent network zones that provide a fixed entry point to the applications and eliminate the complexity of managing specific IP addresses for different AWS Regions and AZs.
  • always routes user traffic to the optimal endpoint based on performance, reacting instantly to changes in application health, the user’s location, and configured policies
  • improves performance for a wide range of applications over TCP or UDP by proxying packets at the edge to applications running in one or more AWS Regions.
  • is a good fit for non-HTTP use cases, such as gaming (UDP), IoT (MQTT), or Voice over IP, as well as for HTTP use cases that specifically require static IP addresses or deterministic, fast regional failover. 
  • integrates with AWS Shield for DDoS protection

Transit Gateway – TGW

  • is a highly available and scalable service to consolidate the AWS VPC routing configuration for a region with a hub-and-spoke architecture.
  • acts as a Regional virtual router and is a network transit hub that can be used to interconnect VPCs and on-premises networks.
  • traffic always stays on the global AWS backbone, data is automatically encrypted, and never traverses the public internet, thereby reducing threat vectors, such as common exploits and DDoS attacks.
  • is a Regional resource and can connect VPCs within the same AWS Region.
  • TGWs across the same or different regions can peer with each other.
  • provides simpler VPC-to-VPC communication management over VPC Peering with a large number of VPCs.
  • scales elastically based on the volume of network traffic.

AWS Management Tools Cheat Sheet

~ Last updated on : ~ jayendrapatil ~ 7 Comments

AWS Organizations

  • AWS Organizations is an account management service that enables consolidating multiple AWS accounts into an organization that can be created and centrally managed.
  • AWS Organizations enables you to
    • Automate AWS account creation and management, and provision resources with AWS CloudFormation Stacksets
    • Maintain a secure environment with policies and management of AWS security services
    • Govern access to AWS services, resources, and regions
    • Centrally manage policies across multiple AWS accounts
    • Audit your environment for compliance
    • View and manage costs with consolidated billing
    • Configure AWS services across multiple accounts

CloudFormation

  • gives developers and systems administrators an easy way to create and manage a collection of related AWS resources
  • Resources can be updated, deleted, and modified in an orderly, controlled and predictable fashion, in effect applying version control to the AWS infrastructure as code done for software code
  • CloudFormation Template is an architectural diagram, in JSON format, and Stack is the end result of that diagram, which is actually provisioned
  • template can be used to set up the resources consistently and repeatedly over and over across multiple regions and consists of
    • List of AWS resources and their configuration values
    • An optional template file format version number
    • An optional list of template parameters (input values supplied at stack creation time)
    • An optional list of output values like public IP address using the Fn::GetAtt function
    • An optional list of data tables used to lookup static configuration values for e.g., AMI names per AZ
  • supports Chef & Puppet Integration to deploy and configure right down the application layer
  • supports Bootstrap scripts to install packages, files, and services on the EC2 instances by simply describing them in the CF template
  • automatic rollback on error feature is enabled, by default, which will cause all the AWS resources that CF created successfully for a stack up to the point where an error occurred to be deleted
  • provides a WaitCondition resource to block the creation of other resources until a completion signal is received from an external source
  • allows DeletionPolicy attribute to be defined for resources in the template
    • retain to preserve resources like S3 even after stack deletion
    • snapshot to backup resources like RDS after stack deletion
  • DependsOn attribute to specify that the creation of a specific resource follows another
  • Service role is an IAM role that allows AWS CloudFormation to make calls to resources in a stack on the user’s behalf
  • Nested stacks can separate out reusable, common components and create dedicated templates to mix and match different templates but use nested stacks to create a single, unified stack
  • Change Sets presents a summary or preview of the proposed changes that CloudFormation will make when a stack is updated
  • Drift detection enables you to detect whether a stack’s actual configuration differs, or has drifted, from its expected configuration.
  • Termination protection helps prevent a stack from being accidentally deleted.
  • Stack policy can prevent stack resources from being unintentionally updated or deleted during a stack update.
  • StackSets extends the functionality of stacks by enabling you to create, update, or delete stacks across multiple accounts and Regions with a single operation.

Elastic BeanStalk

  • makes it easier for developers to quickly deploy and manage applications in the AWS cloud.
  • automatically handles the deployment details of capacity provisioning, load balancing, auto-scaling and application health monitoring
  • CloudFormation supports ElasticBeanstalk
  • provisions resources to support
    • a web application that handles HTTP(S) requests or
    • a web application that handles background-processing (worker) tasks
  • supports Out Of the Box
    • Apache Tomcat for Java applications
    • Apache HTTP Server for PHP applications
    • Apache HTTP server for Python applications
    • Nginx or Apache HTTP Server for Node.js applications
    • Passenger for Ruby applications
    • MicroSoft IIS 7.5 for .Net applications
    • Single and Multi Container Docker
  • supports custom AMI to be used
  • is designed to support multiple running environments such as one for Dev, QA, Pre-Prod and Production.
  • supports versioning and stores and tracks application versions over time allowing easy rollback to prior version
  • can provision RDS DB instance and connectivity information is exposed to the application by environment variables, but is NOT recommended for production setup as the RDS is tied up with the Elastic Beanstalk lifecycle and if deleted, the RDS instance would be deleted as well

OpsWorks

  • is a configuration management service that helps to configure and operate applications in a cloud enterprise by using Chef
  • helps deploy and monitor applications in stacks with multiple layers
  • supports preconfigured layers for Applications, Databases, Load Balancers, Caching
  • OpsWorks Stacks features is a set of lifecycle events – Setup, Configure, Deploy, Undeploy, and Shutdown – which automatically runs specified set of recipes at the appropriate time on each instance
  • Layers depend on Chef recipes to handle tasks such as installing packages on instances, deploying apps, running scripts, and so on
  • OpsWorks Stacks runs the recipes for each layer, even if the instance belongs to multiple layers
  • supports Auto Healing and Auto Scaling to monitor instance health, and provision new instances

CloudWatch

  • allows monitoring of AWS resources and applications in real time, collect and track pre configured or custom metrics and configure alarms to send notification or make resource changes based on defined rules
  • does not aggregate data across regions
  • stores the log data indefinitely, and the retention can be changed for each log group at any time
  • alarm history is stored for only 14 days
  • can be used an alternative to S3 to store logs with the ability to configure Alarms and generate metrics, however logs cannot be made public
  • Alarms exist only in the created region and the Alarm actions must reside in the same region as well

CloudTrail

  • records access to API calls for the AWS account made from AWS management console, SDKs, CLI and higher level AWS service
  • support many AWS services and tracks who did, from where, what & when
  • can be enabled per-region basis, a region can include global services (like IAM, STS etc), is applicable to all the supported services within that region
  • log files from different regions can be sent to the same S3 bucket
  • can be integrated with SNS to notify logs availability, CloudWatch logs log group for notifications when specific API events occur
  • call history enables security analysis, resource change tracking, trouble shooting and compliance auditing

AWS Compute Services Cheat Sheet

~ Last updated on : ~ jayendrapatil ~ 22 Comments

AWS Compute Services Cheat Sheet

AWS Compute Services

Elastic Cloud Compute – EC2

  • provides scalable computing capacity
  • Features
    • Virtual computing environments, known as EC2 instances
    • Preconfigured templates for EC2 instances, known as Amazon Machine Images (AMIs), that package the bits needed for the server (including the operating system and additional software)
    • Various configurations of CPU, memory, storage, and networking capacity for your instances, known as Instance types
    • Secure login information for your instances using key pairs (public-private keys where private is kept by user)
    • Storage volumes for temporary data that’s deleted when you stop or terminate your instance, known as Instance store volumes
    • Persistent storage volumes for data using Elastic Block Store (EBS)
    • Multiple physical locations for your resources, such as instances and EBS volumes, known as Regions and Availability Zones
    • A firewall to specify the protocols, ports, and source IP ranges that can reach your instances using Security Groups
    • Static IP addresses, known as Elastic IP addresses
    • Metadata, known as tags, can be created and assigned to EC2 resources
    • Virtual networks that are logically isolated from the rest of the AWS cloud, and can optionally connect to on-premises network, known as Virtual private clouds (VPCs)

Amazon Machine Image – AMI

    • template from which EC2 instances can be launched quickly
    • does NOT span across regions, and needs to be copied
    • can be shared with other specific AWS accounts or made public

Instance Types

  • T for applications needing general usage
    • T2 instances are Burstable Performance Instances that provide a baseline level of CPU performance with the ability to burst above the baseline.
    • T2 instances accumulate CPU Credits when they are idle, and consume CPU Credits when they are active.
    • T2 Unlimited Instances can sustain high CPU performance for as long as a workload needs it at an additional cost.
  • R for applications needing more RAM or Memory
  • C for applications needing more Compute
  • M for applications needing more Medium or Moderate performance on both Memory and CPU
  • I for applications needing more IOPS
  • G for applications needing more GPU

Instance Purchasing Option

  • On-Demand Instances
    • pay for instances and compute capacity that you use by the hour
    • no long-term commitments or up-front payments
  • Reserved Instances
    • provides lower hourly running costs by providing a billing discount
    • capacity reservation is applied to instances
    • suited if consistent, heavy, predictable usage
    • provides benefits with Consolidate Billing
    • can be modified to switch Availability Zones or the instance size within the same instance type, given the instance size footprint (Normalization factor) remains the same
    • pay for the entire term regardless of the usage
    • is not a physical instance that is launched, but rather a billing discount applied to the use of On-Demand Instances
  • Scheduled Reserved Instances
    • enable capacity reservations purchase that recurs on a daily, weekly, or monthly basis, with a specified start time and duration, for a one-year term.
    • Charges are incurred for the time that the instances are scheduled, even if they are not used
    • good choice for workloads that do not run continuously, but do run on a regular schedule
  • Spot Instances
    • cost-effective choice but does NOT guarantee availability
    • applications flexible in the timing when they can run and also able to handle interruption by storing the state externally
    • provides a two-minute warning if the instance is to be terminated to save any unsaved work
    • Spot blocks can also be launched with a required duration, which are not interrupted due to changes in the Spot price
    • Spot Fleet is a collection, or fleet, of Spot Instances, and optionally On-Demand Instances, which attempts to launch the number of Spot and On-Demand Instances to meet the specified target capacity
  • Dedicated Instances
    • is a tenancy option that enables instances to run in VPC on hardware that’s isolated, dedicated to a single customer
  • Dedicated Host
    • is a physical server with EC2 instance capacity fully dedicated to your use
  • Light, Medium, and Heavy Utilization Reserved Instances are no longer available for purchase and were part of the Previous Generation AWS EC2 purchasing model

Enhanced Networking

  • results in higher bandwidth, higher packet per second (PPS) performance, lower latency, consistency, scalability, and lower jitter
  • supported using Single Root – I/O Virtualization (SR-IOV) only on supported instance types
  • is supported only with a VPC (not EC2 Classic), HVM virtualization type and available by default on Amazon AMI but can be installed on other AMIs as well

Placement Group

  • Cluster Placement Group
    • provide low latency, High-Performance Computing via 10Gbps network
    • is a logical grouping on instances within a Single AZ
    • don’t span availability zones, can span multiple subnets but subnets must be in the same AZ
    • can span across peered VPCs for the same Availability Zones
    • existing instances can’t be moved into an existing placement group
    • An existing instance can be moved to a placement group, or moved from one placement group to another, or removed from a placement group, given it is in the stopped state.
    • for capacity errors, stop and start the instances in the placement group
    • use homogenous instance types which support enhanced networking and launch all the instances at once
  • Spread Placement Groups
    • is a group of instances that are each placed on distinct underlying hardware i.e. each instance on a distinct rack across AZ
    • recommended for applications that have a small number of critical instances that should be kept separate from each other.
    • reduces the risk of simultaneous failures that might occur when instances share the same underlying hardware.
  • Partition Placement Groups
    • is a group of instances spread across partitions i.e. group of instances spread across racks across AZs
    • reduces the likelihood of correlated hardware failures for the application.
    • can be used to spread deployment of large distributed and replicated workloads, such as HDFS, HBase, and Cassandra, across distinct hardware

EC2 Monitoring

  • CloudWatch provides monitoring for EC2 instances
  • Status monitoring helps quickly determine whether EC2 has detected any problems that might prevent instances from running applications.
  • Status monitoring includes
    • System Status checks – indicate issues with the underlying hardware
    • Instance Status checks – indicate issues with the underlying instance.

Elastic Load Balancer

  • Managed load balancing service and scales automatically
  • distributes incoming application traffic across multiple EC2 instances
  • is distributed system that is fault tolerant and actively monitored by AWS scales it as per the demand
  • are engineered to not be a single point of failure
  • need to Pre-Warm ELB if the demand is expected to shoot especially during load testing. AWS documentation does not mention it now.
  • supports routing traffic to instances in multiple AZs in the same region
  • performs Health Checks to route traffic only to the healthy instances
  • support Listeners with HTTP, HTTPS, SSL, TCP protocols
  • has an associated IPv4 and dual stack DNS name
  • can offload the work of encryption and decryption (SSL termination) so that the EC2 instances can focus on their main work
  • supports Cross Zone load balancing to help route traffic evenly across all EC2 instances regardless of the AZs they reside in
  • to help identify the IP address of a client
    • supports Proxy Protocol header for TCP/SSL connections
    • supports X-Forward headers for HTTP/HTTPS connections
  • supports Stick Sessions (session affinity) to bind a user’s session to a specific application instance,
    • it is not fault tolerant, if an instance is lost the information is lost
    • requires HTTP/HTTPS listener and does not work with TCP
    • requires SSL termination on ELB as it users the headers
  • supports Connection draining to help complete the in-flight requests in case an instance is deregistered
  • For High Availability, it is recommended to attach one subnet per AZ for at least two AZs, even if the instances are in a single subnet.
  • supports Static/Elastic IP (NLB only)
  • IPv4 & IPv6 support however VPC does not support IPv6. VPC now supports IPV6.
  • HTTPS listener does not support Client Side Certificate
  • For SSL termination at backend instances or support for Client Side Certificate use TCP for connections from the client to the ELB, use the SSL protocol for connections from the ELB to the back-end application, and deploy certificates on the back-end instances handling requests
  • supports a single SSL certificate, so for multiple SSL certificate multiple ELBs need to be created
  • Uses Server Name Indication to supports multiple SSL certificates

Application Load Balancer

  • supports HTTP and HTTPS (Secure HTTP) protocols
  • supports HTTP/2, which is enabled natively. Clients that support HTTP/2 can connect over TLS
  • supports WebSockets and Secure WebSockets natively
  • supports Request tracing, by default.
    • request tracing can be used to track HTTP requests from clients to targets or other services.
    • Load balancer upon receiving a request from a client, adds or updates the X-Amzn-Trace-Id header before sending the request to the target
  • supports containerized applications. Using Dynamic port mapping, ECS can select an unused port when scheduling a task and register the task with a target group using this port.
  • supports Sticky Sessions (Session Affinity) using load balancer generated cookies, to route requests from the same client to the same target
  • supports SSL termination, to decrypt the request on ALB before sending it to the underlying targets.
  • supports layer 7 specific features like X-Forwarded-For headers to help determine the actual client IP, port and protocol
  • automatically scales its request handling capacity in response to incoming application traffic.
  • supports hybrid load balancing, to route traffic to instances in VPC and an on-premises location
  • provides High Availability, by allowing more than one AZ to be specified
  • integrates with ACM to provision and bind a SSL/TLS certificate to the load balancer thereby making the entire SSL offload process very easy
  • supports multiple certificates for the same domain to a secure listener
  • supports IPv6 addressing, for an Internet facing load balancer
  • supports Cross-zone load balancing, and cannot be disabled.
  • supports Security Groups to control the traffic allowed to and from the load balancer.
  • provides Access Logs, to record all requests sent the load balancer, and store the logs in S3 for later analysis in compressed format
  • provides Delete Protection, to prevent the ALB from accidental deletion
  • supports Connection Idle Timeout – ALB maintains two connections for each request one with the Client (front end) and one with the target instance (back end). If no data has been sent or received by the time that the idle timeout period elapses, ALB closes the front-end connection
  • integrates with CloudWatch to provide metrics such as request counts, error counts, error types, and request latency
  • integrates with AWS WAF, a web application firewall that helps protect web applications from attacks by allowing rules configuration based on IP addresses, HTTP headers, and custom URI strings
  • integrates with CloudTrail to receive a history of ALB API calls made on the AWS account
  • back-end server authentication is NOT supported
  • does not provide Static, Elastic IP addresses

Network Load Balancer

  • handles volatile workloads and scale to millions of requests per second, without the need of pre-warming
  • offers extremely low latencies for latency-sensitive applications.
  • provides static IP/Elastic IP addresses for the load balancer
  • allows registering targets by IP address, including targets outside the VPC (on-premises) for the load balancer.
  • supports containerized applications. Using Dynamic port mapping, ECS can select an unused port when scheduling a task and register the task with a target group using this port.
  • monitors the health of its registered targets and routes the traffic only to healthy targets
  • enable cross-zone loading balancing only after creating the NLB
  • preserves client side source IP allowing the back-end to see client IP address. Target groups can be created with target type as instance ID or IP address. If targets registered by instance ID, the source IP addresses of the clients are preserved and provided to the applications. If register targets registered by IP address, the source IP addresses are the private IP addresses of the load balancer nodes.
  • supports both network and application target health checks.
  • supports long-lived TCP connections ideal for WebSocket type of applications
  • supports Zonal Isolation, which is designed for application architectures in a single zone and can be enabled in a single AZ to support architectures that require zonal isolation
  • does not support stick sessions

Auto Scaling

  • ensures correct number of EC2 instances are always running to handle the load by scaling up or down automatically as demand changes
  • cannot span multiple regions.
  • attempts to distribute instances evenly between the AZs that are enabled for the Auto Scaling group
  • performs checks either using EC2 status checks or can use ELB health checks to determine the health of an instance and terminates the instance if unhealthy, to launch a new instance
  • can be scaled using manual scaling, scheduled scaling or demand based scaling
  • cooldown period helps ensure instances are not launched or terminated before the previous scaling activity takes effect to allow the newly launched instances to start handling traffic and reduce load

AWS Auto Scaling & ELB

  • Auto Scaling & ELB can be used for High Availability and Redundancy by spanning Auto Scaling groups across multiple AZs within a region and then setting up ELB to distribute incoming traffic across those AZs
  • With Auto Scaling, use ELB health check with the instances to ensure that traffic is routed only to the healthy instances

Lambda

  • offers Serverless computing that allows applications and services to be built and run without thinking about servers.
  • helps run code without provisioning or managing servers, where you pay only for the compute time when the code is running.
  • is priced on a pay-per-use basis and there are no charges when the code is not running.
  • performs all the operational and administrative activities on your behalf, including capacity provisioning, monitoring fleet health, applying security patches to the underlying compute resources, deploying code, running a web service front end, and monitoring and logging the code.
  • does not provide access to the underlying compute infrastructure.
  • handles scalability and availability as it
    • provides easy scaling and high availability to the code without additional effort on your part.
    • is designed to process events within milliseconds.
    • is designed to run many instances of the functions in parallel.
    • is designed to use replication and redundancy to provide high availability for both the service and the functions it operates.
    • has no maintenance windows or scheduled downtimes for either.
    • has a default safety throttle for the number of concurrent executions per account per region.
    • has a higher latency immediately after a function is created, or updated, or if it has not been used recently.
    • for any function updates, there is a brief window of time, less than a minute, when requests would be served by both versions
  • Security
    • stores code in S3 and encrypts it at rest and performs additional integrity checks while the code is in use.
    • each function runs in its own isolated environment, with its own resources and file system view
    • supports Code Signing using AWS Signer, which offers trust and integrity controls that enable you to verify that only unaltered code from approved developers is deployed in the functions.
  • Functions must complete execution within 900 seconds. The default timeout is 3 seconds. The timeout can be set the timeout to any value between 1 and 900 seconds.
  • AWS Step Functions can help coordinate a series of Lambda functions in a specific order. Multiple functions can be invoked sequentially, passing the output of one to the other, and/or in parallel, while the state is being maintained by Step Functions.
  • AWS X-Ray helps to trace functions, which provides insights such as service overhead, function init time, and function execution time.
  • Lambda Provisioned Concurrency provides greater control over the performance of serverless applications.
  • Lambda@Edge allows you to run code across AWS locations globally without provisioning or managing servers, responding to end-users at the lowest network latency.
  • Lambda Extensions allow integration of Lambda with other third-party tools for monitoring, observability, security, and governance.
  • Compute Savings Plan can help save money for Lambda executions.
  • CodePipeline and CodeDeploy can be used to automate the serverless application release process.
  • RDS Proxy provides a highly available database proxy that manages thousands of concurrent connections to relational databases.
  • Supports Elastic File Store, to provide a shared, external, persistent, scalable volume using a fully managed elastic NFS file system without the need for provisioning or capacity management.
  • Supports Function URLs, a built-in HTTPS endpoint that can be invoked using the browser, curl, and any HTTP client.

AWS Storage Services Cheat Sheet

~ Last updated on : ~ jayendrapatil ~ 17 Comments

AWS Storage Services Cheat Sheet

AWS Storage Services

Simple Storage Service – S3

  • provides key-value based object storage with unlimited storage, unlimited objects up to 5 TB for the internet
  • offers an extremely durable, highly available, and infinitely scalable data storage infrastructure at very low costs.
  • is Object-level storage (not a Block level storage) and cannot be used to host OS or dynamic websites (but can work with Javascript SDK)
  • provides durability by redundantly storing objects on multiple facilities within a region
  • regularly verifies the integrity of data using checksums and provides the auto-healing capability
  • S3 resources consist of globally unique buckets with objects and related metadata. The data model is a flat structure with no hierarchies or folders.
  • S3 Replication enables automatic, asynchronous copying of objects across S3 buckets in the same or different AWS regions using SRR or CRR. Replication needs versioning enabled on either side.
  • S3 Transfer Acceleration helps speed data transport over long distances between a client and an S3 bucket using CloudFront edge locations.
  • S3 supports cost-effective Static Website hosting with Client-side scripts.
  • S3 CORS – Cross-Origin Resource Sharing allows cross-origin access to S3 resources.
  • S3 Access Logs enables tracking access requests to an S3 bucket.
  • S3 notification feature enables notifications to be triggered when certain events happen in the bucket.
  • S3 Inventory helps manage the storage and can be used to audit and report on the replication and encryption status of the objects for business, compliance, and regulatory needs.
  • Requestor Pays help bucket owner to specify that the requester requesting the download will be charged for the download.
  • S3 Batch Operations help perform large-scale batch operations on S3 objects and can perform a single operation on lists of specified S3 objects.
  • Pre-Signed URLs can be used shared for uploading/downloading objects for a limited time without requiring AWS security credentials.
  • Multipart Uploads allows
    • parallel uploads with improved throughput and bandwidth utilization
    • fault tolerance and quick recovery from network issues
    • ability to pause and resume uploads
    • begin an upload before the final object size is known
  • Versioning
    • helps preserve, retrieve, and restore every version of every object
    • protect from unintended overwrites and accidental deletions
    • protects individual files but does NOT protect from Bucket deletion
  • MFA (Multi-Factor Authentication) can be enabled for additional security for the deletion of objects.
  • Integrates with CloudTrail, CloudWatch, and SNS for event notifications
  • S3 Storage Classes
    • S3 Standard
      • default storage class, ideal for frequently accessed data
      • 99.999999999% durability & 99.99% availability
      • Low latency and high throughput performance
      • designed to sustain the loss of data in a two facilities
    • S3 Standard-Infrequent Access (S3 Standard-IA)
      • optimized for long-lived and less frequently accessed data
      • designed to sustain the loss of data in a two facilities
      • 99.999999999% durability & 99.9% availability
      • suitable for objects greater than 128 KB kept for at least 30 days
    • S3 One Zone-Infrequent Access (S3 One Zone-IA)
      • optimized for rapid access, less frequently access data
      • ideal for secondary backups and reproducible data
      • stores data in a single AZ, data stored in this storage class will be lost in the event of AZ destruction.
      • 99.999999999% durability & 99.5% availability
    • S3 Reduced Redundancy Storage (Not Recommended)
      • designed for noncritical, reproducible data stored at lower levels of redundancy than the STANDARD storage class
      • reduces storage costs
      • 99.99% durability & 99.99% availability
      • designed to sustain the loss of data in a single facility
    • S3 Glacier
      • suitable for low cost data archiving, where data access is infrequent
      • provides retrieval time of minutes to several hours
        • Expedited – 1 to 5 minutes
        • Standard – 3 to 5 hours
        • Bulk – 5 to 12 hours
      • 99.999999999% durability & 99.9% availability
      • Minimum storage duration of 90 days
    • S3 Glacier Deep Archive (S3 Glacier Deep Archive)
      • provides lowest cost data archiving, where data access is infrequent
      • 99.999999999% durability & 99.9% availability
      • provides retrieval time of several (12-48) hours
        • Standard – 12 hours
        • Bulk – 48 hours
      • Minimum storage duration of 180 days
      • supports long-term retention and digital preservation for data that may be accessed once or twice a year
  • Lifecycle Management policies
    • transition to move objects to different storage classes and Glacier
    • expiration to remove objects and object versions
    • can be applied to both current and non-current objects, in case, versioning is enabled.
  • Data Consistency Model
    • provides strong read-after-write consistency for PUT and DELETE requests of objects in the S3 bucket in all AWS Regions
    • updates to a single key are atomic
    • does not currently support object locking for concurrent writes
  • S3 Security
    • IAM policies – grant users within your own AWS account permission to access S3 resources
    • Bucket and Object ACL – grant other AWS accounts (not specific users) access to  S3 resources
    • Bucket policies – allows to add or deny permissions across some or all of the objects within a single bucket
    • S3 Access Points simplify data access for any AWS service or customer application that stores data in S3.
    • S3 Glacier Vault Lock helps deploy and enforce compliance controls for individual S3 Glacier vaults with a vault lock policy.
    • S3 VPC Gateway Endpoint enables private connections between a VPC and S3, without requiring that you use an internet gateway, NAT device, VPN connection, or AWS Direct Connect connection.
    • Support SSL encryption of data in transit and data encryption at rest
  • S3 Data Encryption
    • supports data at rest and data in transit encryption
    • Server-Side Encryption
      • SSE-S3 – encrypts S3 objects using keys handled & managed by AWS
      • SSE-KMS – leverage AWS Key Management Service to manage encryption keys. KMS provides control and audit trail over the keys.
      • SSE-C – when you want to manage your own encryption keys. AWS does not store the encryption key. Requires HTTPS.
    • Client-Side Encryption
      • Client library such as the S3 Encryption Client
      • Clients must encrypt data themselves before sending it to S3
      • Clients must decrypt data themselves when retrieving from S3
      • Customer fully manages the keys and encryption cycle
  • S3 Best Practices
    • use random hash prefix for keys and ensure a random access pattern, as S3 stores object lexicographically randomness helps distribute the contents across multiple partitions for better performance
    • use parallel threads and Multipart upload for faster writes
    • use parallel threads and Range Header GET for faster reads
    • for list operations with a large number of objects, it’s better to build a secondary index in DynamoDB
    • use Versioning to protect from unintended overwrites and deletions, but this does not protect against bucket deletion
    • use VPC S3 Endpoints with VPC to transfer data using Amazon internal network

Instance Store

  • provides temporary or ephemeral block-level storage for an EC2 instance
  • is physically attached to the Instance
  • deliver very high random I/O performance, which is a good option when storage with very low latency is needed
  • cannot be dynamically resized
  • data persists when an instance is rebooted
  • data does not persists if the
    • underlying disk drive fails
    • instance stops i.e. if the EBS backed instance with instance store volumes attached is stopped
    • instance terminates
  • can be attached to an EC2 instance only when the instance is launched
  • is ideal for the temporary storage of information that changes frequently, such as buffers, caches, scratch data, and other temporary content, or for data that is replicated across a fleet of instances, such as a load-balanced pool of web servers.

Elastic Block Store – EBS

  • is virtual network-attached block storage
  • provides highly available, reliable, durable, block-level storage volumes that can be attached to a running instance
  • provides high durability and are redundant in an AZ, as the data is automatically replicated within that AZ to prevent data loss due to any single hardware component failure
  • persists and is independent of EC2 lifecycle
  • multiple volumes can be attached to a single EC2 instance
  • can be detached & attached to another EC2 instance in that same AZ only
  • volumes are Zonal i.e. created in a specific AZ and CAN’T span across AZs
  • snapshots
  • for making volume available to different AZ, create a snapshot of the volume and restore it to a new volume in any AZ within the region
  • for making the volume available to different Region, the snapshot of the volume can be copied to a different region and restored as a volume
  • PIOPS is designed to run transactions applications that require high and consistent IO for e.g. Relation database, NoSQL, etc
  • volumes CANNOT be shared with multiple EC2 instances, use EFS instead
  • Multi-Attach enables attaching a single Provisioned IOPS SSD (io1 or io2) volume to multiple instances that are in the same AZ.

EBS Encryption

  • allow encryption using the EBS encryption feature.
  • All data stored at rest, disk I/O, and snapshots created from the volume are encrypted.
  • uses 256-bit AES algorithms (AES-256) and an Amazon-managed KMS
  • Snapshots of encrypted EBS volumes are automatically encrypted.

EBS Snapshots

  • helps create backups of EBS volumes
  • are incremental
  • occur asynchronously, consume the instance IOPS
  • are regional and CANNOT span across regions
  • can be copied across regions to make it easier to leverage multiple regions for geographical expansion, data center migration, and disaster recovery
  • can be shared by making them public or with specific AWS accounts by modifying the access permissions of the snapshots
  • support EBS encryption
    • Snapshots of encrypted volumes are automatically encrypted
    • Volumes created from encrypted snapshots are automatically encrypted
    • All data in flight between the instance and the volume is encrypted
    • Volumes created from an unencrypted snapshot owned or have access to can be encrypted on the fly.
    • Encrypted snapshot owned or having access to, can be encrypted with a different key during the copy process.
  • can be automated using AWS Data Lifecycle Manager

EBS vs Instance Store

Refer blog post @ EBS vs Instance Store

Glacier

  • suitable for archiving data, where data access is infrequent and a retrieval time of several hours (3 to 5 hours) is acceptable (Not true anymore with enhancements from AWS)
  • provides a high durability by storing archive in multiple facilities and multiple devices at a very low cost storage
  • performs regular, systematic data integrity checks and is built to be automatically self healing
  • aggregate files into bigger files before sending them to Glacier and use range retrievals to retrieve partial file and reduce costs
  • improve speed and reliability with multipart upload
  • automatically encrypts the data using AES-256
  • upload or download data to Glacier via SSL encrypted endpoints

EFS

  • fully-managed, easy to set up, scale, and cost-optimize file storage
  • can automatically scale from gigabytes to petabytes of data without needing to provision storage
  • provides managed NFS (network file system) that can be mounted on and accessed by multiple EC2 in multiple AZs simultaneously
  • highly durable, highly scalable and highly available.
    • stores data redundantly across multiple Availability Zones
    • grows and shrinks automatically as files are added and removed, so you there is no need to manage storage procurement or provisioning.
  • expensive (3x gp2), but you pay per use
  • uses the Network File System version 4 (NFS v4) protocol
  • is compatible with all Linux-based AMIs for EC2,  POSIX file system (~Linux) that has a standard file API
  • does not support Windows AMI
  • offers the ability to encrypt data at rest using KMS and in transit.
  • can be accessed from on-premises using an AWS Direct Connect or AWS VPN connection between the on-premises datacenter and VPC.
  • can be accessed concurrently from servers in the on-premises datacenter as well as EC2 instances in the Amazon VPC
  • Performance mode
    • General purpose (default)
      • latency-sensitive use cases (web server, CMS, etc…)
    • Max I/O
      • higher latency, throughput, highly parallel (big data, media processing)
  • Storage Tiers
    • Standard
      • for frequently accessed files
      • ideal for active file system workloads and you pay only for the file system storage you use per month
    • Infrequent access (EFS-IA)
      • a lower cost storage class that’s cost-optimized for files infrequently accessed i.e. not accessed every day
      • cost to retrieve files, lower price to store
    • EFS Lifecycle Management with choosing an age-off policy allows moving files to EFS IA
    • Lifecycle Management automatically moves the data to the EFS IA storage class according to the lifecycle policy. for e.g., you can move files automatically into EFS IA fourteen days of not being accessed.
    • EFS is a shared POSIX system for Linux systems and does not work for Windows

Amazon FSx for Windows

  • is a fully managed,  highly reliable, and scalable Windows file system share drive
  • supports SMB protocol & Windows NTFS
  • supports Microsoft Active Directory integration, ACLs, user quotas
  • built on SSD, scale up to 10s of GB/s, millions of IOPS, 100s PB of data
  • is accessible from Windows, Linux, and MacOS compute instances
  • can be accessed from the on-premise infrastructure
  • can be configured to be Multi-AZ (high availability)
  • supports encryption of data at rest and in transit
  • provides data deduplication, which enables further cost optimization by removing redundant data.
  • data is backed-up daily to S3

Amazon FSx for Lustre

  • provides easy and cost effective way to launch and run the world’s most popular high-performance file system.
  • is a type of parallel distributed file system, for large-scale computing
  • Lustre is derived from “Linux” and “cluster”
  • Machine Learning, High Performance Computing (HPC) esp. Video Processing, Financial Modeling, Electronic Design Automation
  • scales up to 100s GB/s, millions of IOPS, sub-ms latencies
  • seamless integration with S3, it transparently presents S3 objects as files and allows you to write changed data back to S3.
  • can “read S3” as a file system (through FSx)
  • can write the output of the computations back to S3 (through FSx)
  • supports encryption of data at rest and in transit
  • can be used from on-premise servers

CloudFront

  • provides low latency and high data transfer speeds for distribution of static, dynamic web or streaming content to web users
  • delivers the content through a worldwide network of data centers called Edge Locations
  • keeps persistent connections with the origin servers so that the files can be fetched from the origin servers as quickly as possible.
  • dramatically reduces the number of network hops that users’ requests must pass through
  • supports multiple origin server options, like AWS hosted service for e.g. S3, EC2, ELB or an on premise server, which stores the original, definitive version of the objects
  • single distribution can have multiple origins and Path pattern in a cache behavior determines which requests are routed to the origin
  • supports Web Download distribution and RTMP Streaming distribution
    • Web distribution supports static, dynamic web content, on demand using progressive download & HLS and live streaming video content
    • RTMP supports streaming of media files using Adobe Media Server and the Adobe Real-Time Messaging Protocol (RTMP) ONLY
  • supports HTTPS using either
    • dedicated IP address, which is expensive as dedicated IP address is assigned to each CloudFront edge location
    • Server Name Indication (SNI), which is free but supported by modern browsers only with the domain name available in the request header
  • For E2E HTTPS connection,
    • Viewers -> CloudFront needs either self signed certificate, or certificate issued by CA or ACM
    • CloudFront -> Origin needs certificate issued by ACM for ELB and by CA for other origins
  •  Security
    • Origin Access Identity (OAI) can be used to restrict the content from S3 origin to be accessible from CloudFront only
    • supports Geo restriction (Geo-Blocking) to whitelist or blacklist countries that can access the content
    • Signed URLs 
      • for RTMP distribution as signed cookies aren’t supported
      • to restrict access to individual files, for e.g., an installation download for your application.
      • users using a client, for e.g. a custom HTTP client, that doesn’t support cookies
    • Signed Cookies
      • provide access to multiple restricted files, for e.g., video part files in HLS format or all of the files in the subscribers’ area of a website.
      • don’t want to change the current URLs
    • integrates with AWS WAF, a web application firewall that helps protect web applications from attacks by allowing rules configured based on IP addresses, HTTP headers, and custom URI strings
  • supports GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE to get object & object headers, add, update, and delete objects
    • only caches responses to GET and HEAD requests and, optionally, OPTIONS requests
    • does not cache responses to PUT, POST, PATCH, DELETE request methods and these requests are proxied back to the origin
  • object removal from cache
    • would be removed upon expiry (TTL) from the cache, by default 24 hrs
    • can be invalidated explicitly, but has a cost associated, however might continue to see the old version until it expires from those caches
    • objects can be invalidated only for Web distribution
    • change object name, versioning, to serve different version
  • supports adding or modifying custom headers before the request is sent to origin which can be used to
    • validate if user is accessing the content from CDN
    • identifying CDN from which the request was forwarded from, in case of multiple CloudFront distribution
    • for viewers not supporting CORS to return the Access-Control-Allow-Origin header for every request
  • supports Partial GET requests using range header to download object in smaller units improving the efficiency of partial downloads and recovery from partially failed transfers
  • supports compression to compress and serve compressed files when viewer requests include Accept-Encoding: gzip in the request header
  • supports different price class to include all regions, to include only least expensive regions and other regions to exclude most expensive regions
  • supports access logs which contain detailed information about every user request for both web and RTMP distribution

AWS Import/Export

  • accelerates moving large amounts of data into and out of AWS using portable storage devices for transport and transfers data directly using Amazon’s high speed internal network, bypassing the internet.
  • suitable for use cases with
    • large datasets
    • low bandwidth connections
    • first time migration of data
  • Importing data to several types of AWS storage, including EBS snapshots, S3 buckets, and Glacier vaults.
  • Exporting data out from S3 only, with versioning enabled only the latest version is exported
  • Import data can be encrypted (optional but recommended) while export is always encrypted using Truecrypt
  • Amazon will wipe the device if specified, however it will not destroy the device

AWS Elastic Transcoder – Certification

~ Last updated on : ~ jayendrapatil ~ 4 Comments

AWS Elastic Transcoder

  • Amazon Elastic Transcoder is a highly scalable, easy-to-use and cost-effective way for developers and businesses to convert (or “transcode”) video files from their source format into versions that will play back on multiple devices like smartphones, tablets and PCs.
  • Elastic Transcoder is for any customer with media assets stored in S3 for e.g. developers creating apps or websites that publish user-generated content, enterprises and educational establishments converting training and communication videos, and content owners and broadcasters needing to convert media assets into web-friendly formats.
  • Elastic Transcoder features
    • can be used to convert files from different media formats into H.264/AAC/MP4 files at different resolutions, bitrates, and frame rates, and set up transcoding pipelines to transcode files in parallel.
    • can be configured to overlay up to four graphics, known as watermarks, over a video during transcoding
    • can be configured to transcode captions, or subtitles, from one format to another and supports embedded and sidebar caption types
    • provides clip stitching ability to stitch together parts, or clips, from multiple input files to create a single output
    • can be configured to create Thumbnails
  • Elastic Transcoder is integrated with CloudTrail, an AWS service that captures information about every request that is sent to the Elastic Transcoder API by your AWS account, including your IAM users

Elastic Transcoder Components

  • Presets
    • are templates that contain most of the settings for transcoding media files from one format to another.
    • Elastic Transcoder includes some default presets for common formats and ability to create customized presets
  • Jobs
    • do the work of transcoding and converts a file into up to 30 formats.
    • takes the input file to be transcoded, names of the transcoded files and several other settings as input
    • For each transcoded format a preset needs to be specified
  • Pipelines
    • are queues that manage the transcoding jobs.
    • Elastic Transcoder starts processing the jobs and transcoding into format (for multiple formats) in the order they are added.
    • can be paused to temporarily stop processing jobs
  • Notifications
    • help keep you apprised of the status of a job, i.e. started, completed, encounters warning or error
    • eliminate the need for polling to determine when a job has finished and can be configured during pipeline creation

AWS Certification Exam Practice Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  1. Your website is serving on-demand training videos to your workforce. Videos are uploaded monthly in high resolution MP4 format. Your workforce is distributed globally often on the move and using company-provided tablets that require the HTTP Live Streaming (HLS) protocol to watch a video. Your company has no video transcoding expertise and it required you might need to pay for a consultant. How do you implement the most cost-efficient architecture without compromising high availability and quality of video delivery?
    1. Elastic Transcoder to transcode original high-resolution MP4 videos to HLS. S3 to host videos with lifecycle Management to archive original flies to Glacier after a few days. CloudFront to serve HLS transcoded videos from S3
    2. A video transcoding pipeline running on EC2 using SQS to distribute tasks and Auto Scaling to adjust the number or nodes depending on the length of the queue S3 to host videos with Lifecycle Management to archive all files to Glacier after a few days CloudFront to serve HLS transcoding videos from Glacier
    3. Elastic Transcoder to transcode original high-resolution MP4 videos to HLS EBS volumes to host videos and EBS snapshots to incrementally backup original rues after a few days. CloudFront to serve HLS transcoded videos from EC2.
    4. A video transcoding pipeline running on EC2 using SQS to distribute tasks and Auto Scaling to adjust the number of nodes depending on the length of the queue. EBS volumes to host videos and EBS snapshots to incrementally backup original files after a few days. CloudFront to serve HLS transcoded videos from EC2

References

AWS CloudSearch – Certification

~ Last updated on : ~ jayendrapatil

AWS CloudSearch

  • CloudSearch is a fully-managed, full-featured search service in the AWS Cloud that makes it easy to set up, manage, and scale a search solution
  • CloudSearch
    • automatically provisions the required resources
    • deploys a highly tuned search index
    • easy configuration and can be up & running in less than one hour
    • search and ability to upload searchable data
    • automatically scales for data and traffic
    • self-healing clusters, and
    • high availability with Multi-AZ
  • CloudSearch uses Apache Solr as the underlying text search engine and
    • can be used to index and search both structured and unstructured data.
    • content can come from multiple sources and can include database fields along with files in a variety of formats, web pages, and so on.
    • supports indexing features like algorithmic stemming, dictionary stemming, stopword dictionary
    • can support customizable result ranking i.e. relevancy
    • supports search features for text search, different query types (range, boolean etc), sorting, facets for filtering, grouping etc
    • supports enhanced features for auto suggestions, highlighting, spatial search, fuzzy search etc
  • CloudSearch supports Multi-AZ option and it deploys additional instances in a second AZ in the same region.
  • CloudSearch can offer significantly lower total cost of ownership compared to operating and managing your own search environment

CloudSearch Search Domains, Data & Indexing

CloudSearch Architecture

  • Search domain is a data container and a set of services that make the data searchable
    • Document service that allows data uploading to domain for indexing
    • Search service that enables search requests against the indexed data
    • Configuration service for controlling the domains behavior (include relevance ranking)
  • Search domain can’t be automatically migrated from one region to another. New domain in the target region needs to be created, configured and data uploaded, and then the original domain deleted
  • Indexed data to be made searchable
    • can be submitted through a REST based web service url
    • has to be in JSON or XML format
    • is represented as a document with a unique document ID and multiple fields either to be search on to needed to be just retrieved
  • CloudSearch generates a search index from the document data according to the index fields configured for the domain
  • Data updates can be submitted by to add, update and delete documents
  • Data can be uploaded using secure and encrypted SSL HTTPS connection

CloudSearch Auto Scaling

CloudSearch Scaling

  • Search domains scale in two dimensions: data and traffic
  • A search instance is a single search engine in the cloud that indexes documents and responds to search requests with a finite amount of RAM and CPU resources for indexing data and processing requests.
  • Search domain can have one or more search partitions, portion of the data which fits on a single search instance, and the number of search partitions can change as the documents are indexed
  • CloudSearch can determine the size and number of search instances required to deliver low latency, high throughput search performance
  • When a search domain is created , a single instance is deployed
  • CloudSearch automatically scales the domain by adding instances as the volume of data or traffic increases
  • Scaling for data
    • CloudSearch handles scaling for data by
      • Vertical scaling by increasing the size of the instance, when the amount of data exceeds a single search instance
      • Horizontal scaling using search partitions, when the amount of data exceeds the capacity of the largest search instance type
    • Number of search instances required to hold the index partitions is sometimes referred to as the domain’s width.
    • CloudSearch reduces the number of partitions and size of search instances if the amount of data reduces
  • Scaling for traffic
    • CloudSearch handles Scaling for traffic by
      • Vertical scaling by increasing the size of the instance, when the amount of traffic exceeds a single search instance
      • Horizontal scaling by deploying a duplicate search instance to provide additional processing power i.e. the complete number of partitions are duplicated
    • CloudSearch reduces the number of partitions and size of search instances if the traffic reduces
    • Number of duplicate search instances is sometimes referred to as the domain’s depth.

CloudSearch Search Features

  • CloudSearch provides features to index and search both structured data and plain text as well as unstructured data like pdf, word documents
  • CloudSearch provides near real-time indexing for document updates
  • Indexing features include
    • tokenization,
    • stopwords,
    • stemming and
    • synonyms
  • Search features include
    • faceted search, free text search, Boolean search expressions,
    • customizable relevance ranking, query time rank expressions,
    • grouping
    • field weighting, searching and sorting
    • Other features like
      • Autocomplete suggestions
      • Highlighting
      • Geospatial search
      • New data types: date, double, 64 bit signed int, LatLon
      • Dynamic fields
      • Index field statistics
      • Sloppy phrase search
      • Term boosting
      • Enhanced range searching for all field types
      • Search filters that don’t affect relevance
      • Support for multiple query parsers: simple, structured, lucene, dismax
      • Query parser configuration options

AWS Certification Exam Practice Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  1. A newspaper organization has an on-premises application which allows the public to search its back catalogue and retrieve individual newspaper pages via a website written in Java. They have scanned the old newspapers into JPEGs (approx. 17TB) and used Optical Character Recognition (OCR) to populate a commercial search product. The hosting platform and software is now end of life and the organization wants to migrate its archive to AWS and produce a cost efficient architecture and still be designed for availability and durability. Which is the most appropriate?
    1. Use S3 with reduced redundancy to store and serve the scanned files, install the commercial search application on EC2 Instances and configure with auto-scaling and an Elastic Load Balancer. (Reusing Commercial search application which is nearing end of life not a good option for cost)
    2. Model the environment using CloudFormation. Use an EC2 instance running Apache webserver and an open source search application, stripe multiple standard EBS volumes together to store the JPEGs and search index. (storing JPEGs on EBS volumes not cost effective also answer does not address Open source solution availability)
    3. Use S3 with standard redundancy to store and serve the scanned files, use CloudSearch for query processing, and use Elastic Beanstalk to host the website across multiple availability zones. (Cost effective S3 storage, CloudSearch for Search and Highly available and durable web application)
    4. Use a single-AZ RDS MySQL instance to store the search index and the JPEG images use an EC2 instance to serve the website and translate user queries into SQL. (MySQL not an ideal solution to sore index and JPEG images for cost and performance)
    5. Use a CloudFront download distribution to serve the JPEGs to the end users and Install the current commercial search product, along with a Java Container for the website on EC2 instances and use Route53 with DNS round-robin. (Web Application not scalable, whats the source for JPEGs files through CloudFront)

References