Skip to content
- provides Relational Database service
- supports MySQL, MariaDB, PostgreSQL, Oracle, Microsoft SQL Server, and the new, MySQL-compatible Amazon Aurora DB engine
- as it is a managed service, shell (root ssh) access is not provided
- manages backups, software patching, automatic failure detection, and recovery
- supports use initiated manual backups and snapshots
- daily automated backups with database transaction logs enables Point in Time recovery up to the last five minutes of database usage
- snapshots are user-initiated storage volume snapshot of DB instance, backing up the entire DB instance and not just individual databases that can be restored as a independent RDS instance
- support encryption at rest using KMS as well as encryption in transit using SSL endpoints
- for encrypted database
- logs, snapshots, backups, read replicas are all encrypted as well
- cross region replicas and snapshots does not work across region (Note – this is possible now with latest AWS enhancement)
- Multi-AZ deployment
- provides high availability and automatic failover support and is NOT a scaling solution
- maintains a synchronous standby replica in a different AZ
- transaction success is returned only if the commit is successful both on the primary and the standby DB
- Oracle, PostgreSQL, MySQL, and MariaDB DB instances use Amazon technology, while SQL Server DB instances use SQL Server Mirroring
- snapshots and backups are taken from standby & eliminate I/O freezes
- during automatic failover, its seamless and RDS switches to the standby instance and updates the DNS record to point to standby
- failover can be forced with the Reboot with failover option
- Read Replicas
- uses the PostgreSQL, MySQL, and MariaDB DB engines’ built-in replication functionality to create a separate Read Only instance
- updates are asynchronously copied to the Read Replica, and data might be stale
- can help scale applications and reduce read only load
- requires automatic backups enabled
- replicates all databases in the source DB instance
- for disaster recovery, can be promoted to a full fledged database
- can be created in a different region for MySQL, Postgres and MariaDB, for disaster recovery, migration and low latency across regions
- RDS does not support all the features of underlying databases, and if required the database instance can be launched on an EC2 instance
- RMAN (Recovery Manager) can be used for Oracles backup and recovery when running on an EC2 instance
- fully managed NoSQL database service
- synchronously replicates data across three facilities in an AWS Region, giving high availability and data durability
- runs exclusively on SSDs to provide high I/O performance
- provides provisioned table reads and writes
- automatically partitions, reallocates and re-partitions the data and provisions additional server capacity as data or throughput changes
- provides Eventually consistent (by default) or Strongly Consistent option to be specified during an read operation
- creates and maintains indexes for the primary key attributes for efficient access of data in the table
- supports secondary indexes
- allows querying attributes other then the primary key attributes without impacting performance.
- are automatically maintained as sparse objects
- Local vs Global secondary index
- shares partition key + different sort key vs different partition + sort key
- search limited to partition vs across all partition
- unique attributes vs non unique attributes
- linked to the base table vs independent separate index
- only created during the base table creation vs can be created later
- cannot be deleted after creation vs can be deleted
- consumes provisioned throughput capacity of the base table vs independent throughput
- returns all attributes for item vs only projected attributes
- Eventually or Strongly vs Only Eventually consistent reads
- size limited to 10Gb per partition vs unlimited
- supports cross region replication using DynamoDB streams which leverages Kinesis and provides time-ordered sequence of item-level changes and can help for lower RPO, lower RTO disaster recovery
- Data Pipeline jobs with EMR can be used for disaster recovery with higher RPO, lower RTO requirements
- supports triggers to allow execution of custom actions or notifications based on item-level updates
- managed web service that provides in-memory caching to deploy and run Memcached or Redis protocol-compliant cache clusters
- ElastiCache with Redis,
- like RDS, supports Multi-AZ, Read Replicas and Snapshots
- Read Replicas are created across AZ within same region using Redis’s asynchronous replication technology
- Multi-AZ differs from RDS as there is no standby, but if the primary goes down a Read Replica is promoted as primary
- Read Replicas cannot span across regions, as RDS supports
- cannot be scaled out and if scaled up cannot be scaled down
- allows snapshots for backup and restore
- AOF can be enabled for recovery scenarios, to recover the data in case the node fails or service crashes. But it does not help in case the underlying hardware fails
- Enabling Redis Multi-AZ as a Better Approach to Fault Tolerance
- ElastiCache with Memcached
- can be scaled up by increasing size and scaled out by adding nodes
- nodes can span across multiple AZs within the same region
- cached data is spread across the nodes, and a node failure will always result in some data loss from the cluster
- supports auto discovery
- every node should be homogenous and of same instance type
- ElastiCache Redis vs Memcached
- complex data objects vs simple key value storage
- persistent vs non persistent, pure caching
- automatic failover with Multi-AZ vs Multi-AZ not supported
- scaling using Read Replicas vs using multiple nodes
- backup & restore supported vs not supported
- can be used state management to keep the web application stateless
- fully managed, fast and powerful, petabyte scale data warehouse service
- uses replication and continuous backups to enhance availability and improve data durability and can automatically recover from node and component failures
- provides Massive Parallel Processing (MPP) by distributing & parallelizing queries across multiple physical resources
- columnar data storage improving query performance and allowing advance compression techniques
- only supports Single-AZ deployments and the nodes are available within the same AZ, if the AZ supports Redshift clusters
- spot instances are NOT an option
- Amazon Redshift is a fully managed, fast and powerful, petabyte scale data warehouse service
- Redshift automatically helps
- set up, operate, and scale a data warehouse, from provisioning the infrastructure capacity
- patches and backs up the data warehouse, storing the backups for a user-defined retention period
- monitors the nodes and drives to help recovery from failures
- significantly lowers the cost of a data warehouse, but also makes it easy to analyze large amounts of data very quickly
- provide fast querying capabilities over structured data using familiar SQL-based clients and business intelligence (BI) tools using standard ODBC and JDBC connections.
- uses replication and continuous backups to enhance availability and improve data durability and can automatically recover from node and component failures.
- scale up or down with a few clicks in the AWS Management Console or with a single API call
- distribute & parallelize queries across multiple physical resources
- supports VPC, SSL, AES-256 encryption and Hardware Security Modules (HSMs) to protect the data in transit and at rest.
- Redshift only supports Single-AZ deployments and the nodes are available within the same AZ, if the AZ supports Redshift clusters
- Redshift provides monitoring using CloudWatch and metrics for compute utilization, storage utilization, and read/write traffic to the cluster are available with the ability to add user-defined custom metrics
- Redshift provides Audit logging and AWS CloudTrail integration
- Redshift can be easily enabled to a second region for disaster recovery.
- Massively Parallel Processing (MPP)
- automatically distributes data and query load across all nodes.
- makes it easy to add nodes to the data warehouse and enables fast query performance as the data warehouse grows.
- Columnar Data Storage
- organizes the data by column, as column-based systems are ideal for data warehousing and analytics, where queries often involve aggregates performed over large data sets
- columnar data is stored sequentially on the storage media, and require far fewer I/Os, greatly improving query performance
- Advance Compression
- Columnar data stores can be compressed much more than row-based data stores because similar data is stored sequentially on disk.
- employs multiple compression techniques and can often achieve significant compression relative to traditional relational data stores.
- doesn’t require indexes or materialized views and so uses less space than traditional relational database systems.
- automatically samples the data and selects the most appropriate compression scheme, when the data is loaded into an empty table
Redshift Single vs Multi-Node Cluster
- Single Node
- single node configuration enables getting started quickly and cost-effectively & scale up to a multi-node configuration as the needs grow
- Multi-node configuration requires a leader node that manages client connections and receives queries, and two or more compute nodes that store data and perform queries and computations.
- Leader node
- provisioned automatically and not charged for
- receives queries from client applications, parses the queries and develops execution plans, which are an ordered set of steps to process these queries.
- coordinates the parallel execution of these plans with the compute nodes, aggregates the intermediate results from these nodes and finally returns the results back to the client applications.
- Compute node
- can contain from 1-128 compute nodes, depending on the node type
- executes the steps specified in the execution plans and transmit data among themselves to serve these queries.
- intermediate results are sent back to the leader node for aggregation before being sent back to the client applications.
- supports Dense Storage or Dense Compute nodes (DC) instance type
- Dense Storage (DS) allow creation of very large data warehouses using hard disk drives (HDDs) for a very low price point
- Dense Compute (DC) allow creation of very high performance data warehouses using fast CPUs, large amounts of RAM and solid-state disks (SSDs)
- direct access to compute nodes is not allowed
Redshift Availability & Durability
- Redshift replicates the data within the data warehouse cluster and continuously backs up the data to S3 (11 9’s durability)
- Redshift mirrors each drive’s data to other nodes within the cluster.
- Redshift will automatically detect and replace a failed drive or node
- If a drive fails, Redshift
- cluster will remain available in the event of a drive failure
- the queries will continue with a slight latency increase while Redshift rebuilds the drive from replica of the data on that drive which is stored on other drives within that node
- single node clusters do not support data replication and the cluster needs to be restored from snapshot on S3
- In case of node failure(s), Redshift
- automatically provisions new node(s) and begins restoring data from other drives within the cluster or from S3
- prioritizes restoring the most frequently queried data so the most frequently executed queries will become performant quickly
- cluster will be unavailable for queries and updates until a replacement node is provisioned and added to the cluster
- In case of Redshift cluster AZ goes down, Redshift
- cluster is unavailable until power and network access to the AZ are restored
- cluster’s data is preserved and can be used once AZ becomes available
- cluster can be restored from any existing snapshots to a new AZ within the same region
Redshift Backup & Restore
- Redshift replicates all the data within the data warehouse cluster when it is loaded and also continuously backs up the data to S3
- Redshift always attempts to maintain at least three copies of the data
- Redshift enables automated backups of the data warehouse cluster with a 1-day retention period, by default, which can be extended to max 35 days
- Automated backups can be turned off by setting the retention period as 0
- Redshift can also asynchronously replicate the snapshots to S3 in another region for disaster recovery
- Redshift allows scaling of the cluster either by
- increasing the node instance type (Vertical scaling)
- increasing the number of nodes (Horizontal scaling)
- Redshift scaling changes are usually applied during the maintenance window or can be applied immediately
- Redshift scaling process
- existing cluster remains available for read operations only while a new data warehouse cluster gets created during scaling operations
- data from the compute nodes in the existing data warehouse cluster is moved in parallel to the compute nodes in the new cluster
- when the new data warehouse cluster is ready, the existing cluster will be temporarily unavailable while the canonical name record of the existing cluster is flipped to point to the new data warehouse cluster
Redshift vs EMR vs RDS
- RDS is ideal for
- structured data and running traditional relational databases while offloading database administration
- for online-transaction processing (OLTP) and for reporting and analysis
- Redshift is ideal for
- large volumes of structured data that needs to be persisted and queried using standard SQL and existing BI tools
- analytic and reporting workloads against very large data sets by harnessing the scale and resources of multiple nodes and using a variety of optimizations to provide improvements over RDS
- preventing reporting and analytic processing from interfering with the performance of the OLTP workload
- EMR is ideal for
- processing and transforming unstructured or semi-structured data to bring in to Amazon Redshift and
- for data sets that are relatively transitory, not stored for long-term use.
AWS Certification Exam Practice Questions
- Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
- AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
- AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
- Open to further feedback, discussion and correction.
- With which AWS services CloudHSM can be used (select 2)
- Amazon Redshift
- You have recently joined a startup company building sensors to measure street noise and air quality in urban areas. The company has been running a pilot deployment of around 100 sensors for 3 months. Each sensor uploads 1KB of sensor data every minute to a backend hosted on AWS. During the pilot, you measured a peak of 10 IOPS on the database, and you stored an average of 3GB of sensor data per month in the database. The current deployment consists of a load-balanced auto scaled Ingestion layer using EC2 instances and a PostgreSQL RDS database with 500GB standard storage. The pilot is considered a success and your CEO has managed to get the attention or some potential investors. The business plan requires a deployment of at least 100K sensors, which needs to be supported by the backend. You also need to store sensor data for at least two years to be able to compare year over year Improvements. To secure funding, you have to make sure that the platform meets these requirements and leaves room for further scaling. Which setup will meet the requirements?
- Add an SQS queue to the ingestion layer to buffer writes to the RDS instance (RDS instance will not support data for 2 years)
- Ingest data into a DynamoDB table and move old data to a Redshift cluster (Handle 10K IOPS ingestion and store data into Redshift for analysis)
- Replace the RDS instance with a 6 node Redshift cluster with 96TB of storage (Does not handle the ingestion issue)
- Keep the current architecture but upgrade RDS storage to 3TB and 10K provisioned IOPS (RDS instance will not support data for 2 years)
- Which two AWS services provide out-of-the-box user configurable automatic backup-as-a-service and backup rotation options? Choose 2 answers
- Amazon S3
- Amazon RDS
- Amazon EBS
- Amazon Redshift
- Your department creates regular analytics reports from your company’s log files. All log data is collected in Amazon S3 and processed by daily Amazon Elastic Map Reduce (EMR) jobs that generate daily PDF reports and aggregated tables in CSV format for an Amazon Redshift data warehouse. Your CFO requests that you optimize the cost structure for this system. Which of the following alternatives will lower costs without compromising average performance of the system or data integrity for the raw data?
- Use reduced redundancy storage (RRS) for PDF and CSV data in Amazon S3. Add Spot instances to Amazon EMR jobs. Use Reserved Instances for Amazon Redshift. (Spot instances impacts performance)
- Use reduced redundancy storage (RRS) for all data in S3. Use a combination of Spot instances and Reserved Instances for Amazon EMR jobs. Use Reserved instances for Amazon Redshift (Combination of the Spot and reserved with guarantee performance and help reduce cost. Also, RRS would reduce cost and guarantee data integrity, which is different from data durability)
- Use reduced redundancy storage (RRS) for all data in Amazon S3. Add Spot Instances to Amazon EMR jobs. Use Reserved Instances for Amazon Redshift (Spot instances impacts performance)
- Use reduced redundancy storage (RRS) for PDF and CSV data in S3. Add Spot Instances to EMR jobs. Use Spot Instances for Amazon Redshift. (Spot instances impacts performance and Spot instance not available for Redshift)
- is a temporary data repository for messages and provides a reliable, highly scalable, hosted message queuing service for temporary storage and delivery of short (up to 256 KB) text-based data messages.
- supports a virtually unlimited number of queues and supports unordered, at-least-once delivery of messages.
Ideal Usage patterns
- is ideally suited to any scenario where multiple application components must communicate and coordinate their work in a loosely coupled manner particularly producer consumer scenarios
- can be used to coordinate a multi-step processing pipeline, where each message is associated with a task that must be processed.
- enables the number of worker instances to scale up or down, and also enable the processing power of each single worker instance to scale up or down, to suit the total workload, without any application changes.
- Binary or Large Messages
- SQS is suited for text messages with maximum size of 64 KB. If the application requires binary or messages exceeding the length, it is best to use Amazon S3 or RDS and use SQS to store the pointer
- Long Term storage
- SQS stores messages for max 14 days and if application requires storage period longer than 14 days, Amazon S3 or other storage options should be preferred
- High-speed message queuing or very short tasks
- If the application requires a very high-speed message send and receive response from a single producer or consumer, use of Amazon DynamoDB or a message-queuing system hosted on Amazon EC2 may be more appropriate.
- is a distributed queuing system that is optimized for horizontal scalability, not for single-threaded sending or receiving speeds.
- A single client can send or receive Amazon SQS messages at a rate of about 5 to 50 messages per second. Higher receive performance can be achieved by requesting multiple messages (up to 10) in a single call.
Durability & Availability
- are highly durable but temporary.
- stores all messages redundantly across multiple servers and data centers.
- Message retention time is configurable on a per-queue basis, from a minimum of one minute to a maximum of 14 days.
- Messages are retained in a queue until they are explicitly deleted, or until they are automatically deleted upon expiration of the retention time.
- pricing is based on
- number of requests and
- the amount of data transferred in and out (priced per GB per month).
Scalability & Elasticity
- is both highly elastic and massively scalable.
- is designed to enable a virtually unlimited number of computers to read and write a virtually unlimited number of messages at any time.
- supports virtually unlimited numbers of queues and messages per queue for any user.
- is a fast, fully-managed, petabyte-scale data warehouse service that makes it simple and cost-effective to efficiently analyze all your data using your existing business intelligence tools.
- is optimized for datasets that range from a few hundred gigabytes to a petabyte or more.
- manages the work needed to set up, operate, and scale a data warehouse, from provisioning the infrastructure capacity to automating ongoing administrative tasks such as backups and patching.
Ideal Usage Pattern
- is ideal for analyzing large datasets using the existing business intelligence tools
- Common use cases include
- Analyze global sales data for multiple products
- Store historical stock trade data
- Analyze ad impressions and clicks
- Aggregate gaming data
- Analyze social trends
- Measure clinical quality, operation efficiency, and financial
- performance in the health care space
- OLTP workloads
- Redshift is a column-oriented database and more suited for data warehousing and analytics. If application involves online transaction processing, Amazon RDS would be a better choice.
- Blob data
- For Blob storage, Amazon S3 would be a better choice with metadata in other storage as RDS or DynamoDB
- Amazon Redshift allows a very high query performance on datasets ranging in size from hundreds of gigabytes to a petabyte or more.
- It uses columnar storage, data compression, and zone maps to reduce the amount of I/O needed to perform queries.
- It has a massively parallel processing (MPP) architecture that parallelizes and distributes SQL operations to take advantage of all available resources.
- Underlying hardware is designed for high performance data processing that uses local attached storage to maximize throughput.
Durability & Availability
- Amazon Redshift stores three copies of your data—all data written to a node in your cluster is automatically replicated to other nodes within the cluster, and all data is continuously backed up to Amazon S3.
- Snapshots are automated, incremental, and continuous and stored for a user-defined period (1-35 days)
- Manual snapshots can be created and are retained until explicitly deleted.
- Amazon Redshift also continuously monitors the health of the cluster and automatically re-replicates data from failed drives and replaces nodes as necessary.
- has three pricing components:
- data warehouse node hours – total number of hours run across all the compute node
- backup storage – storage cost for automated and manual snapshots
- data transfer
- There is no data transfer charge for data transferred to or from Amazon Redshift outside of Amazon VPC
- Data transfer to or from Amazon Redshift in Amazon VPC accrues standard AWS data transfer charges.
Scalability & Elasticity
- provides push button scaling and the number of nodes can be easily scaled in the data warehouse cluster as the demand changes.
- Redshift places the existing cluster in the read only mode, so the existing queries can continue to run, while is provisions a new cluster with chosen size and copies the data to it. Once the data is copied, it automatically redirects queries to the new cluster