Primary key uniquely identifies each item in a DynamoDB table and can be simple (a partition key only) or composite (a partition key combined with a sort key).
Partition key portion of a table’s primary key determines the logical partitions in which a table’s data is stored, which in turn affects the underlying physical partitions.
Avoid hot keys and hot partitions – a partition key design that doesn’t distribute I/O requests evenly can create “hot” partitions that result in throttling and use the provisioned I/O capacity inefficiently.
Partition key should have many unique values
Distribute reads / writes uniformly across partitions to avoid hot partitions
Store hot and cold data in separate tables
Consider all possible query patterns to eliminate use of scans and filters
Choose sort key depending on the application’s needs
Secondary Indexes
Use indexes based on when the application’s query patterns
LSIs
Use primary key or LSIs when strong consistency is desired
Watch for expanding item collections (10 GB size limit!)
GSIs
Use GSIs for finer control over throughput or when your application needs to query using a different partition key
Can be used for eventually consistent read replicas – set up a global secondary index that has the same key schema as the parent table, with some or all of the non-key attributes projected into it.
Project fewer attributes – As secondary indexes consume storage and provisioned throughput, keep the size of the index as small as possible. as it would provide greater performance.
Keep the number of indexes to a minimum – don’t create secondary indexes on attributes that aren’t queried often. Indexes that are seldom used contribute to increased storage and I/O costs without improving application performance.
Large Items and Attributes
DynamoDB currently limits the size of each item that is stored in a table
Use shorter (yet intuitive!) attribute names
Keep item size small
Use compression (GZIP)
Split large attributes across multiple items
Store metadata in DynamoDB and large BLOBs or attributes in S3
Querying and Scanning Data
Avoid scans and filters – Scan operations are less efficient than other operations in DynamoDB. A Scan operation always scans the entire table or secondary index. It then filters out values to provide the result, essentially adding the extra step of removing data from the result set.
Use eventual consistency for reads
Time Series Data
Use table per day, week, month etc for storing time series data – create one table per period, provisioned with the required read and write capacity and the required indexes.
Before the end of each period, prebuild the table for the next period. Just as the current period ends, direct event traffic to the new table. Assign names to the tables that specify the periods they have recorded.
As soon as a table is no longer being written to, reduce its provisioned write capacity to a lower value (for example, 1 WCU), and provision whatever read capacity is appropriate. Reduce the provisioned read capacity of earlier tables as they age. Archive or delete the tables whose contents are rarely or never needed.
Amazon Aurora Serverless is an on-demand, autoscaling configuration for the MySQL-compatible and PostgreSQL-compatible editions of Aurora.
An Aurora Serverless DB cluster automatically starts up, shuts down, and scales capacity up or down based on the application’s needs.
Aurora Serverless enables running database in the cloud without managing any database instances.
Aurora Serverless provides a relatively simple, cost-effective option for infrequent, intermittent, or unpredictable workloads.
Aurora Serverless Use Cases include
Infrequently-Used Applications
New Applications – where the needs and instance size is yet to be determined.
Variable and Unpredictable Workloads – scale as per the needs
Development and Test Databases
Multi-tenant Applications
Aurora Serverless DB cluster does not have a public IP address and can be accessed only from within a VPC based on the VPC service.
Aurora Architecture
Aurora Serverless separates Storage and Compute, so it can scale down to zero processing and you pay only for storage.
With Aurora Serverless, a database endpoint is created without specifying the DB instance class size.
Minimum and maximum capacity is set in terms of Aurora capacity units (ACUs). Each ACU is a combination of processing and memory capacity.
Database storage automatically scales from 10 GiB to 64 TiB, the same as storage in a standard Aurora DB cluster.
The minimum Aurora capacity unit is the lowest ACU to which the DB cluster can scale down. The maximum Aurora capacity unit is the highest ACU to which the DB cluster can scale up. Based on the settings, Aurora Serverless automatically creates scaling rules for thresholds for CPU utilization, connections, and available memory.
Database endpoint connects to a proxy fleet that routes the workload to a fleet of resources that are automatically scaled.
Aurora Serverless manages the connections automatically.
Proxy fleet enables continuous connections as Aurora Serverless scales the resources automatically based on the minimum and maximum capacity specifications.
Database client applications don’t need to change to use the proxy fleet.
Scaling is rapid because it uses a pool of “warm” resources that are always ready to service requests.
Aurora Serverless supports Automatic Pause where the DB cluster can be paused after a given amount of time with no activity. The default inactvity timeout is five minutes. Pausing the DB cluster can be disabled.
Automatic Pause reduces the compute charges to zero and only storage is charged. If database connections are requested when an Aurora Serverless DB cluster is paused, the DB cluster automatically resumes and services the connection requests.
When the DB cluster resumes activity, it has the same capacity as it had when Aurora paused the cluster. The number of ACUs depends on how much Aurora scaled the cluster up or down before pausing it.
Aurora Serverless and Failover
Aurora Serverless compute layer is placed in a Single AZ
Aurora separates computation capacity and storage, the storage volume for the cluster is spread across multiple AZs. The data remains available even if outages affect the DB instance or the associated AZ.
Aurora Serverless supports automatic multi-AZ failover where if the DB instance for an DB cluster becomes unavailable or the Availability Zone (AZ) it is in fails, Aurora recreates the DB instance in a different AZ.
Aurora Serverless failover mechanism takes longer than for an Aurora Provisioned cluster.
Aurora Serverless failover time is currently undefined because it depends on demand and capacity availability in other AZs within the given AWS Region
Aurora Serverless Auto Scaling
Aurora Serverless automatically scales based on the active database workload ( CPU or connections), in some cases, capacity might not scale fast enough to meet a sudden workload change, such as a large number of new transactions.
Once a scaling operation is initiated, Aurora Serverless attempts to find a scaling point, which is a point in time at which the database can safely complete scaling.
Aurora Serverless might not be able to find a scaling point and will not scale if there are
long-running queries or transactions in progress, or
temporary tables or table locks in use.
Supports cooldown period
After Scale up, Aurora Serverless has a 15 minutes cooldown period for subsequent scale down
After Scale down, Aurora Serverless has a 310 secs cooldown period for subsequent scale down
Aurora Serverless has not cooldown period for scale up activities and scales as and when necessary
AWS Certification Exam Practice Questions
Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
Open to further feedback, discussion and correction.
Multi-AZ deployments provides high availability and automatic failover support for DB instances
Multi-AZ helps improve the durability and availability of a critical system, enhancing availability during planned system maintenance, DB instance failure and Availability Zone disruption.
Multi-AZ is a High Availability feature is not a scaling solution for read-only scenarios; standby replica can’t be used to serve read traffic. To service read-only traffic, use a Read Replica.
Multi-AZ deployments for Oracle, PostgreSQL, MySQL, and MariaDB DB instances use Amazon technology, while SQL Server DB instances use SQL Server Mirroring.
In a Multi-AZ deployment,
RDS automatically provisions and maintains a synchronous standby replica in a different Availability Zone.
Copies of data are stored in different Availability Zones for greater levels of data durability.
Primary DB instance is synchronously replicated across Availability Zones to a standby replica to provide
data redundancy,
eliminate I/O freezes during snapshots and backups
and minimize latency spikes during system backups.
DB instances may have increased write and commit latency compared to a Single AZ deployment, due to the synchronous data replication
Transaction success is returned only if the commit is successful both on the primary and the standby DB
There might be a change in latency if the deployment fails over to the standby replica, although AWS is engineered with low-latency network connectivity between Availability Zones.
When using the BYOL licensing model, a license for both the primary instance and the standby replica is required
For production workloads, it is recommended to use Multi-AZ deployment with Provisioned IOPS and DB instance classes (m1.large and larger), optimized for Provisioned IOPS for fast, consistent performance.
When Single-AZ deployment is modified to a Multi-AZ deployment (for engines other than SQL Server or Amazon Aurora)
RDS takes a snapshot of the primary DB instance from the deployment and restores the snapshot into another Availability Zone.
RDS then sets up synchronous replication between the primary DB instance and the new instance.
This avoids downtime when conversion from Single AZ to Multi-AZ
RDS Multi-AZ Failover Process
In the event of a planned or unplanned outage of the DB instance,
RDS automatically switches to a standby replica in another AZ, if enabled for Multi-AZ.
Time it takes for the failover to complete depends on the database activity and other conditions at the time the primary DB instance became unavailable.
Failover times are typically 60-120 secs. However, large transactions or a lengthy recovery process can increase failover time.
Failover mechanism automatically changes the DNS record of the DB instance to point to the standby DB instance.
Multi-AZ switch is seamless to the applications as there is no change in the endpoint URLs but just needs to re-establish any existing connections to the DB instance.
RDS handles failover automatically so that database operations can be resumed as quickly as possible without administrative intervention.
Primary DB instance switches over automatically to the standby replica if any of the following conditions occur:
An Availability Zone outage
Primary DB instance fails
DB instance’s server type is changed
Operating system of the DB instance is undergoing software patching
A manual failover of the DB instance was initiated using Reboot with failover (also referred to as Forced Failover)
If the Multi-AZ DB instance has failed over, can be determined by
DB event subscriptions can be setup to notify you via email or SMS that a failover has been initiated.
DB events can be viewed via the Amazon RDS console or APIs.
Current state of your Multi-AZ deployment can be viewed via the RDS console and APIs.
AWS Certification Exam Practice Questions
Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
Open to further feedback, discussion and correction.
A company is deploying a new two-tier web application in AWS. The company has limited staff and requires high availability, and the application requires complex queries and table joins. Which configuration provides the solution for the company’s requirements?
MySQL Installed on two Amazon EC2 Instances in a single Availability Zone (does not provide High Availability out of the box)
Amazon RDS for MySQL with Multi-AZ
Amazon ElastiCache (Just a caching solution)
Amazon DynamoDB (Not suitable for complex queries and joins)
What would happen to an RDS (Relational Database Service) multi-Availability Zone deployment if the primary DB instance fails?
IP of the primary DB Instance is switched to the standby DB Instance.
A new DB instance is created in the standby availability zone.
The canonical name record (CNAME) is changed from primary to standby.
The RDS (Relational Database Service) DB instance reboots.
Will my standby RDS instance be in the same Availability Zone as my primary?
Only for Oracle RDS types
Yes
Only if configured at launch
No
Is creating a Read Replica of another Read Replica supported?
Only in certain regions
Only with MySQL based RDS
Only for Oracle RDS types
No
A user is planning to set up the Multi-AZ feature of RDS. Which of the below mentioned conditions won’t take advantage of the Multi-AZ feature?
Availability zone outage
A manual failover of the DB instance using Reboot with failover option
Region outage
When the user changes the DB instance’s server type
When you run a DB Instance as a Multi-AZ deployment, the “_____” serves database writes and reads
secondary
backup
stand by
primary
When running my DB Instance as a Multi-AZ deployment, can I use the standby for read or write operations?
Yes
Only with MSSQL based RDS
Only for Oracle RDS instances
No
Read Replicas require a transactional storage engine and are only supported for the _________ storage engine
OracleISAM
MSSQLDB
InnoDB
MyISAM
A user is configuring the Multi-AZ feature of an RDS DB. The user came to know that this RDS DB does not use the AWS technology, but uses server mirroring to achieve replication. Which DB is the user using right now?
MySQL
Oracle
MS SQL
PostgreSQL
If you have chosen Multi-AZ deployment, in the event of a planned or unplanned outage of your primary DB Instance, Amazon RDS automatically switches to the standby replica. The automatic failover mechanism simply changes the ______ record of the main DB Instance to point to the standby DB Instance.
DNAME
CNAME
TXT
MX
When automatic failover occurs, Amazon RDS will emit a DB Instance event to inform you that automatic failover occurred. You can use the _____ to return information about events related to your DB Instance
FetchFailure
DescriveFailure
DescribeEvents
FetchEvents
The new DB Instance that is created when you promote a Read Replica retains the backup window period.
TRUE
FALSE
Will I be alerted when automatic failover occurs?
Only if SNS configured
No
Yes
Only if Cloudwatch configured
Can I initiate a “forced failover” for my MySQL Multi-AZ DB Instance deployment?
Only in certain regions
Only in VPC
Yes
No
A user is accessing RDS from an application. The user has enabled the Multi-AZ feature with the MS SQL RDS DB. During a planned outage how will AWS ensure that a switch from DB to a standby replica will not affect access to the application?
RDS will have an internal IP which will redirect all requests to the new DB
RDS uses DNS to switch over to standby replica for seamless transition
The switch over changes Hardware so RDS does not need to worry about access
RDS will have both the DBs running independently and the user has to manually switch over
Which of the following is part of the failover process for a Multi-AZ Amazon Relational Database Service (RDS) instance?
The failed RDS DB instance reboots.
The IP of the primary DB instance is switched to the standby DB instance.
The DNS record for the RDS endpoint is changed from primary to standby.
A new DB instance is created in the standby availability zone.
Which of these is not a reason a Multi-AZ RDS instance will failover?
An Availability Zone outage
A manual failover of the DB instance was initiated using Reboot with failover
To autoscale to a higher instance class (Refer link)
Master database corruption occurs
The primary DB instance fails
How does Amazon RDS multi Availability Zone model work?
A second, standby database is deployed and maintained in a different availability zone from master, using synchronous replication. (Refer link)
A second, standby database is deployed and maintained in a different availability zone from master using asynchronous replication.
A second, standby database is deployed and maintained in a different region from master using asynchronous replication.
A second, standby database is deployed and maintained in a different region from master using synchronous replication.
A user is using a small MySQL RDS DB. The user is experiencing high latency due to the Multi AZ feature. Which of the below mentioned options may not help the user in this situation?
Schedule the automated back up in non-working hours
Use a large or higher size instance
Use PIOPS
Take a snapshot from standby Replica
What is the charge for the data transfer incurred in replicating data between your primary and standby?
No charge. It is free.
Double the standard data transfer charge
Same as the standard data transfer charge
Half of the standard data transfer charge
A user has enabled the Multi AZ feature with the MS SQL RDS database server. Which of the below mentioned statements will help the user understand the Multi AZ feature better?
In a Multi AZ, AWS runs two DBs in parallel and copies the data asynchronously to the replica copy
In a Multi AZ, AWS runs two DBs in parallel and copies the data synchronously to the replica copy
In a Multi AZ, AWS runs just one DB but copies the data synchronously to the standby replica
Read replicas enable increased scalability and database availability in the case of an AZ failure.
Read Replicas allow elastic scaling beyond the capacity constraints of a single DB instance for read-heavy database workloads
RDS read replicas can be Multi-AZ i.e. set up with their own standby instances in a different AZ.
Load on the source DB instance can be reduced by routing read queries from applications to the Read Replica.
one or more replicas of a given source DB Instance and serve high-volume application read traffic from multiple copies of your data, thereby increasing aggregate read throughput. Read replicas can also be promoted when needed to become standalone DB instances.
RDS uses DB engines’ built-in replication functionality to create a special type of DB instance called a Read Replica from a source DB instance. It uses the engines’ native asynchronous replication to update the read replica whenever there is a change to the source DB instance.
Read replicas are available in RDS for MySQL, MariaDB, PostgreSQL, Oracle, and SQL Server as well as Aurora.
RDS supports replication between an RDS MySQL or MariaDB DB instance and a MySQL or MariaDB instance that is external to RDS using Binary Log File Position or Global Transaction Identifiers (GTIDs) replication.
Read Replicas creation
Up to five Read Replicas can be created from one source DB instance.
Creation process
Automatic backups must be enabled on the source DB instance by setting the backup retention period to a value other than 0
Existing DB instance needs to be specified as the source.
RDS takes a snapshot of the source instance and creates a read-only instance from the snapshot.
RDS then uses the asynchronous replication method for the DB engine to update the Read Replica for any changes to the source DB instance.
RDS replicates all databases in the source DB instance.
RDS sets up a secure communications channel between the source DB instance and the Read Replica, if that Read Replica is in a different AWS region from the DB instance.
RDS establishes any AWS security configurations, such as adding security group entries, needed to enable the secure channel.
During the Read Replica creation, a brief I/O suspension on the source DB instance can be experienced as the DB snapshot occurs.
I/O suspension typically lasts about one minute and can be avoided if the source DB instance is a Multi-AZ deployment (in the case of Multi-AZ deployments, DB snapshots are taken from the standby).
Read Replica creation time can be slow if any long-running transactions are being executed and should wait for completion
For multiple Read Replicas created in parallel from the same source DB instance, only one snapshot is taken at the start of the first create action.
A Read Replica can be promoted to a new independent source DB, in which case the replication link is broken between the Read Replica and the source DB. However, the replication continues for other replicas using the original source DB as the replication source
Read Replica Deletion & DB Failover
Read Replicas must be explicitly deleted, using the same mechanisms for deleting a DB instance.
If the source DB instance is deleted without deleting the replicas, each replica is promoted to a stand-alone, single-AZ DB instance.
If the source instance of a Multi-AZ deployment fails over to the standby, any associated Read Replicas are switched to use the secondary as their replication source.
Read Replica Storage & Compute requirements
A Read Replica, by default, is created with the same storage type as the source DB instance.
For replication to operate effectively, each Read Replica should have the same amount of compute & storage resources as the source DB instance.
Source DB instance, if scaled, Read Replicas should be scaled accordingly
Cross-Region Read Replicas
Supported for MySQL, PostgreSQL, MariaDB and Oracle
Not supported for SQL Server
helps to improve
disaster recovery capabilities (reduces RTO and RPO),
scale read operations into a region closer to end users,
migration from a data center in one region to another region
A source DB instance can have cross-Region read replicas in multiple AWS Regions.
Cross-Region RDS read replica can be created from a source RDS DB instance that is not a read replica of another RDS DB instance.
Replica lags are higher for Cross-region replicas. This lag time comes from the longer network channels between regional data centers.
RDS can’t guarantee more than five cross-Region read replica instances, due to the limit on the number of access control list (ACL) entries for a VPC
Read replica uses the default DB parameter group and default security group for the specified DB engine.
Deleting the source for a cross-Region read replica will result in
read replica promotion for MariaDB, MySQL, and Oracle DB instances
no read replica promotion for PostgreSQL DB instances and the replication status of the read replica is set to terminated.
Read Replica Features & Limitations
RDS does not support circular replication.
DB instance cannot be configured to serve as a replication source for an existing DB instance; a new Read Replica can be created only from an existing DB instance for e.g., if MyDBInstance replicates to ReadReplica1, ReadReplica1 can’t be configured to replicate back to MyDBInstance. From ReadReplica1, only a new Read Replica can be created, such as ReadRep2.
Read Replica can be created from other Read replicas as well. However, the replica lag is higher for these instances and there cannot be more than four instances involved in a replication chain.
Read Replica Use cases
Scaling beyond the compute or I/O capacity of a single DB instance for read-heavy database workloads, directing excess read traffic to Read Replica(s)
Serving read traffic while the source DB instance is unavailable for e.g. If the source DB instance cannot take I/O requests due to backups I/O suspension or scheduled maintenance, the read traffic can be directed to the Read Replica(s). However, the data might be stale.
Business reporting or data warehousing scenarios where business reporting queries can be executed against a Read Replica, rather than the primary, production DB instance.
Implementing disaster recovery by promoting the read replica to a standalone instance as a disaster recovery solution, if the primary DB instance fails.
AWS Certification Exam Practice Questions
Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
Open to further feedback, discussion and correction.
You are running a successful multi-tier web application on AWS and your marketing department has asked you to add a reporting tier to the application. The reporting tier will aggregate and publish status reports every 30 minutes from user-generated information that is being stored in your web applications database. You are currently running a Multi-AZ RDS MySQL instance for the database tier. You also have implemented ElastiCache as a database caching layer between the application tier and database tier. Please select the answer that will allow you to successfully implement the reporting tier with as little impact as possible to your database.
Continually send transaction logs from your master database to an S3 bucket and generate the reports off the S3 bucket using S3 byte range requests.
Generate the reports by querying the synchronously replicated standby RDS MySQL instance maintained through Multi-AZ (Standby instance cannot be used as a scaling solution)
Launch a RDS Read Replica connected to your Multi-AZ master database and generate reports by querying the Read Replica.
Generate the reports by querying the ElastiCache database caching tier. (ElasticCache does not maintain full data and is simply a caching solution)
Your company is getting ready to do a major public announcement of a social media site on AWS. The website is running on EC2 instances deployed across multiple Availability Zones with a Multi-AZ RDS MySQL Extra Large DB Instance. The site performs a high number of small reads and writes per second and relies on an eventual consistency model. After comprehensive tests you discover that there is read contention on RDS MySQL. Which are the best approaches to meet these requirements? (Choose 2 answers)
Deploy ElastiCache in-memory cache running in each availability zone
Implement sharding to distribute load to multiple RDS MySQL instances (this is only a read contention, the writes work fine)
Increase the RDS MySQL Instance size and Implement provisioned IOPS (not scalable, this is only a read contention, the writes work fine)
Add an RDS MySQL read replica in each availability zone
Your company has HQ in Tokyo and branch offices all over the world and is using logistics software with a multi-regional deployment on AWS in Japan, Europe and US. The logistic software has a 3-tier architecture and currently uses MySQL 5.6 for data persistence. Each region has deployed its own database. In the HQ region you run an hourly batch process reading data from every region to compute cross-regional reports that are sent by email to all offices this batch process must be completed as fast as possible to quickly optimize logistics. How do you build the database architecture in order to meet the requirements?
For each regional deployment, use RDS MySQL with a master in the region and a read replica in the HQ region
For each regional deployment, use MySQL on EC2 with a master in the region and send hourly EBS snapshots to the HQ region
For each regional deployment, use RDS MySQL with a master in the region and send hourly RDS snapshots to the HQ region
For each regional deployment, use MySQL on EC2 with a master in the region and use S3 to copy data files hourly to the HQ region
Use Direct Connect to connect all regional MySQL deployments to the HQ region and reduce network latency for the batch process
Your business is building a new application that will store its entire customer database on a RDS MySQL database, and will have various applications and users that will query that data for different purposes. Large analytics jobs on the database are likely to cause other applications to not be able to get the query results they need to, before time out. Also, as your data grows, these analytics jobs will start to take more time, increasing the negative effect on the other applications. How do you solve the contention issues between these different workloads on the same data?
Enable Multi-AZ mode on the RDS instance
Use ElastiCache to offload the analytics job data
Create RDS Read-Replicas for the analytics work
Run the RDS instance on the largest size possible
If I have multiple Read Replicas for my master DB Instance and I promote one of them, what happens to the rest of the Read Replicas?
The remaining Read Replicas will still replicate from the older master DB Instance
The remaining Read Replicas will be deleted
The remaining Read Replicas will be combined to one read replica
You need to scale an RDS deployment. You are operating at 10% writes and 90% reads, based on your logging. How best can you scale this in a simple way?
Create a second master RDS instance and peer the RDS groups.
Cache all the database responses on the read side with CloudFront.
Create read replicas for RDS since the load is mostly reads.
Create a Multi-AZ RDS installs and route read traffic to standby.
A customer is running an application in US-West (Northern California) region and wants to setup disaster recovery failover to the Asian Pacific (Singapore) region. The customer is interested in achieving a low Recovery Point Objective (RPO) for an Amazon RDS multi-AZ MySQL database instance. Which approach is best suited to this need?
Synchronous replication
Asynchronous replication
Route53 health checks
Copying of RDS incremental snapshots
A user is using a small MySQL RDS DB. The user is experiencing high latency due to the Multi AZ feature. Which of the below mentioned options may not help the user in this situation?
Schedule the automated back up in non-working hours
Use a large or higher size instance
Use PIOPS
Take a snapshot from standby Replica
My Read Replica appears “stuck” after a Multi-AZ failover and is unable to obtain or apply updates from the source DB Instance. What do I do?
You will need to delete the Read Replica and create a new one to replace it.
You will need to disassociate the DB Engine and re associate it.
The instance should be deployed to Single AZ and then moved to Multi- AZ once again
You will need to delete the DB Instance and create a new one to replace it.
A company is running a batch analysis every hour on their main transactional DB running on an RDS MySQL instance to populate their central Data Warehouse running on Redshift. During the execution of the batch their transactional applications are very slow. When the batch completes they need to update the top management dashboard with the new data. The dashboard is produced by another system running on-premises that is currently started when a manually-sent email notifies that an update is required The on-premises system cannot be modified because is managed by another team. How would you optimize this scenario to solve performance issues and automate the process as much as possible?
Replace RDS with Redshift for the batch analysis and SNS to notify the on-premises system to update the dashboard
Replace RDS with Redshift for the batch analysis and SQS to send a message to the on-premises system to update the dashboard
Create an RDS Read Replica for the batch analysis and SNS to notify me on-premises system to update the dashboard
Create an RDS Read Replica for the batch analysis and SQS to send a message to the on-premises system to update the dashboard.
AWS Certified Data Analytics – Specialty (DAS-C01) Exam Summary
AWS Certified Data Analytics – Specialty exam, as its name suggests, covers a lot of Big Data concepts right from data transfer and collection techniques, storage, pre and post processing, analytics, visualization with the added concepts for data security at each layer.
AWS Certified Data Analytics – Specialty exam has 65 questions to be solved within a time limit of 170 minutes
Questions and answer options are pretty long, so need time to read through them to make sense of the requirements and filter out the answers
As the exam was online from home, there was no access to paper and pen but the trick remains the same, read the question and draw a rough architecture and focus on the areas that you need to improve. Trust me, you will be able to eliminate 2 answers for sure and then need to focus on only the other two. Read the other 2 answers to check the difference area and that would help you reach to the right answer or atleast have a 50% chance of getting it right.
Glue supports Job Bookmark that helps track data that has already been processed during a previous run of an ETL job by persisting state information from the job run. Job bookmarks help AWS Glue maintain state information and prevent the reprocessing of old data or duplicate records.
supports SSE-S3, SS3-KMS, CSE-KMS and CSE-Custom encryption for EMRFS
doesn’t support SSE-C encryption
supports LUKS encryption for local disks
supports TLS for data in transit encryption
Know Hive can be externally hosted using RDS, Aurora and AWS Glue Data Catalog
Know also different technologies
Presto is a fast SQL query engine designed for interactive analytic queries over large datasets from multiple sources
Spark is a distributed processing framework and programming model that helps do machine learning, stream processing, or graph analytics using Amazon EMR clusters
Zeppelin/Jupyter as a notebook for interactive data exploration and provides open-source web application that can be used to create and share documents that contain live code, equations, visualizations, and narrative text
Phoenix is used for OLTP and operational analytics, allowing you to use standard SQL queries and JDBC APIs to work with an Apache HBase backing store
Understand Redshift Advanced topics like Workload Management, Distribution Style, Sort key
Understand Redshift Spectrum which allows querying data in S3 without loading existing Redshift cluster. It also helps querying S3 data with Redshift data.
Know Redshift Best Practices w.r.t selection of Distribution style, Sort key, importing/exporting data
COPY command which allows parallelism, and performs better than multiple COPY commands
COPY command can use manifest files to load data
COPY command handles encrypted data
Understand Redshift Resizing cluster options (elastic resize did not support node type changes before, but does now)
Know Redshift views to control access to data.
Athena
serverless, interactive query service to analyze data in S3 using standard SQL
QuickSight is a very fast, easy-to-use, cloud-powered business analytics service that makes it easy to build visualizations, perform ad-hoc analysis, and quickly get business insights from their data, anytime, on any device.
QuickSight enables organizations to scale their business analytics capabilities to hundreds of thousands of users, and delivers fast and responsive query performance by using a robust in-memory engine (SPICE).
QuickSight supports
Excel files and flat files like CSV, TSV, CLF, ELF
on-premises databases like PostgreSQL, SQL Server and MySQL
SaaS applications like Salesforce
and AWS data sources such as Redshift, RDS, Aurora, Athena, and S3
QuickSight supports various functions to format and transform the data.
alias data fields and change data types.
subset the data using built in filters and perform database join operations using drag and drop.
create calculated fields using mathematical operations and built-in functions such conditional statements, string, numerical and date functions
QuickSight supports assorted visualizations that facilitate different analytical approaches:
Comparison and distribution – Bar charts (several assorted variants)
Changes over time – Line graphs, Area line charts
Correlation – Scatter plots, Heat maps
Aggregation – Pie graphs, Tree maps
Tabular – Pivot tables
QuickSight comes with a built-in suggestion engine that provides suggested visualizations based on the properties of the underlying datasets
QuickSight support Stories, that provide guided tours through specific views of an analysis. They are used to convey key points, a thought process, or the evolution of an analysis for collaboration.
QuickSight is built with “SPICE” – a Super-fast, Parallel, In-memory Calculation Engine
SPICE uses a combination of columnar storage, in-memory technologies enabled through the latest hardware innovations and machine code generation to run interactive queries on large datasets and get rapid responses.
SPICE supports rich data discovery and business analytics capabilities to help customers derive valuable insights from their data without worrying about provisioning or managing infrastructure.
SPICE supports rich calculations to help derive valuable insights from the analysis without worrying about provisioning or managing infrastructure.
Data in SPICE is persisted until it is explicitly deleted by the user.
QuickSight can also be configured to keep the data in SPICE up-to-date as the data in the underlying sources change.
SPICE automatically replicates data for high availability and enables QuickSight to scale to hundreds of thousands of users who can all simultaneously perform fast interactive analysis across a wide variety of AWS data sources.
Authors and Readers
QuickSight Author is a user who
can connect to data sources (within AWS or outside), create visuals and analyze data.
can create interactive dashboards using advanced QuickSight capabilities such as parameters and calculated fields, and publish dashboards with other users in the account.
QuickSight Reader is a user who
consumes interactive dashboards.
can log in via their organization’s preferred authentication mechanism (QuickSight username/password, SAML portal or AD auth), view shared dashboards, filter data, drill down to details or export data as a CSV file, using a web browser or mobile app.
Readers do not have any allocated SPICE capacity.
Security and access
QuickSight supports multi-factor authentication (MFA) for the AWS account via the AWS Management console.
For VPC with public connectivity, QuickSight’s IP address range can be added to the database instances’ security group rules to enable traffic flow into the VPC and database instances.
QuickSight supports Row-level security (RLS), which enables dataset owners to control access to data at row granularity based on permissions associated with the user interacting with the data. With RLS, QuickSight users only need to manage a single set of data and apply appropriate row-level dataset rules to it. All associated dashboards and analyses will enforce these rules, simplifying dataset management and removing the need to maintain multiple datasets for users with different data access privileges.
QuickSight supports Private VPC (Virtual Private Cloud) Access, which uses an Elastic Network Interface (ENI) for secure, private communication with data sources in a VPC. It also allows use of AWS Direct Connect to create a secure, private link with the on-premises resources.
AWS Certification Exam Practice Questions
Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
Open to further feedback, discussion and correction.
You are using QuickSight to identify demand trends over multiple months for your top five product lines. Which type of visualization do you choose?
Scatter Plot
Pie Chart
Pivot Table
Line Chart
You need to provide customers with rich visualizations that allow you to easily connect multiple disparate data sources in S3, Redshift, and several CSV files. Which tool should you use that requires the least setup?
AWS Glue is a fully-managed, ETL i.e extract, transform, and load service that automates the time-consuming steps of data preparation for analytics
AWS Glue is serverless and supports pay-as-you-go model. There is no infrastructure to provision or manage. AWS Glue handles provisioning, configuration, and scaling of the resources required to run the ETL jobs on a fully managed, scale-out Apache Spark environment.
AWS Glue makes it simple and cost-effective to categorize the data, clean it, enrich it, and move it reliably between various data stores and streams
AWS Glue automatically discovers and profiles the data via the Glue Data Catalog, recommends and generates ETL code to transform the source data into target schemas, and runs the ETL jobs on a fully managed, scale-out Apache Spark environment to load the data into its destination.
AWS Glue also helps setup, orchestrate, and monitor complex data flows.
AWS Glue consists of a
Data Catalog, which is a central metadata repository,
ETL engine that can automatically generate Scala or Python code,
Flexible scheduler that handles dependency resolution, job monitoring, and retries.
AWS Glue components help automate much of the undifferentiated heavy lifting involved with discovering, categorizing, cleaning, enriching, and moving data, so more time can be spent on analyzing the data.
Glue can automatically discover both structured and semi-structured data stored in the data lake on S3, data warehouse in Redshift, and various databases running on AWS.
Glue provides a unified view of the data via the Glue Data Catalog that is available for ETL, querying and reporting using services like Athena, EMR, and Redshift Spectrum.
MySQL, Oracle, Microsoft SQL Server, and PostgreSQL databases in the Virtual Private Cloud (VPC) running on EC2.
in (beta) data streams from MSK, Kinesis Data Streams, and Apache Kafka.
Glue also supports custom Scala or Python code and import custom libraries and Jar files into the AWS Glue ETL jobs to access data sources not natively supported by AWS Glue.
Glue supports server side encryption for data at rest and SSL for data in motion.
AWS Glue provides development endpoints to edit, debug, and test the code it generates.
AWS Glue Data Catalog
AWS Glue Data Catalog is a central repository and persistent metadata store to store structural and operational metadata for all the data assets.
AWS Glue Data Catalog provides a uniform repository where disparate systems can store and find metadata to keep track of data in data silos, and use that metadata to query and transform the data.
For a given data set, AWS Glue Data Catalog can store its table definition, physical location, add business relevant attributes, as well as track how this data has changed over time.
AWS Glue Data Catalog is Apache Hive Metastore compatible and is a drop-in replacement for the Apache Hive Metastore for Big Data applications running on Amazon EMR
AWS Glue Data Catalog also provides out-of-box integration with Athena, EMR, and Redshift Spectrum.
Table definitions once added to the Glue Data Catalog, are available for ETL and also readily available for querying in Athena, EMR, and Redshift Spectrum to provide a common view of the data between these services.
AWS Glue Data Catalog supports bulk import of the metadata from existing persistent Apache Hive Metastore by using our import script.
Data Catalog provides comprehensive audit and governance capabilities, with schema change tracking and data access controls, which helps ensure that data is not inappropriately modified or inadvertently shared
Each AWS account has one AWS Glue Data Catalog per region.
AWS Glue Crawlers
AWS Glue crawler connects to a data store, progresses through a prioritized list of classifiers to extract the schema of the data and other statistics, and then populates the Glue Data Catalog with this metadata.
Glue crawlers scan various data stores to automatically infer schemas and partition structure to populate the Glue Data Catalog with corresponding table definitions and statistics.
Glue crawlers can be scheduled to run periodically so that the metadata is always up-to-date and in-sync with the underlying data.
Crawlers automatically add new tables, new partitions to existing table, and new versions of table definitions.
Dynamic Frames
AWS Glue is designed to work with semi-structured data and introduces a component called a dynamic frame, which you can use in the ETL scripts
Dynamic frame is a distributed table that supports nested data such as structures and arrays.
Each record is self-describing, designed for schema flexibility with semi-structured data. Each record contains both data and the schema that describes that data.
A Dynamic Frame is similar to an Apache Spark dataframe, which is a data abstraction used to organize data into rows and columns, except that each record is self-describing so no schema is required initially.
Dynamic frames provide schema flexibility and a set of advanced transformations specifically designed for dynamic frames.
Conversion can be done between Dynamic frames and Spark dataframes, to take advantage of both AWS Glue and Spark transformations to do the kinds of analysis needed.
AWS Glue Streaming ETL
AWS Glue enables performing ETL operations on streaming data using continuously-running jobs.
AWS Glue streaming ETL is built on the Apache Spark Structured Streaming engine, and can ingest streams from Kinesis Data Streams and Apache Kafka using Amazon Managed Streaming for Apache Kafka.
Streaming ETL can clean and transform streaming data and load it into S3 or JDBC data stores.
Use Streaming ETL in AWS Glue to process event data like IoT streams, clickstreams, and network logs.
Glue Job Bookmark
AWS Glue Job Bookmark tracks data that has already been processed during a previous run of an ETL job by persisting state information from the job run.
Job bookmarks help AWS Glue maintain state information and prevent the reprocessing of old data.
Job bookmarks help process new data when rerunning on a scheduled interval
Job bookmark is composed of the states for various elements of jobs, such as sources, transformations, and targets. for e.g, a ETL job might read new partitions in an S3 file. AWS Glue tracks which partitions the job has processed successfully to prevent duplicate processing and duplicate data in the job’s target data store.
AWS Certification Exam Practice Questions
Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
Open to further feedback, discussion and correction.
An organization is setting up a data catalog and metadata management environment for their numerous data stores currently running on AWS. The data catalog will be used to determine the structure and other attributes of data in the data stores. The data stores are composed of Amazon RDS databases, Amazon Redshift, and CSV files residing on Amazon S3. The catalog should be populated on a scheduled basis, and minimal administration is required to manage the catalog. How can this be accomplished?
Set up Amazon DynamoDB as the data catalog and run a scheduled AWS Lambda function that connects to data sources to populate the database.
Use an Amazon database as the data catalog and run a scheduled AWS Lambda function that connects to data sources to populate the database.
Use AWS Glue Data Catalog as the data catalog and schedule crawlers that connect to data sources to populate the database.
Set up Apache Hive metastore on an Amazon EC2 instance and run a scheduled bash script that connects to data sources to populate the metastore.
provides low latency and high data transfer speeds for distribution of static, dynamic web or streaming content to web users
delivers the content through a worldwide network of data centers called Edge Locations
keeps persistent connections with the origin servers so that the files can be fetched from the origin servers as quickly as possible.
dramatically reduces the number of network hops that users’ requests must pass through
supports multiple origin server options, like AWS hosted service for e.g. S3, EC2, ELB or an on premise server, which stores the original, definitive version of the objects
single distribution can have multiple origins and Path pattern in a cache behavior determines which requests are routed to the origin
supports Web Download distribution and RTMPStreaming distribution
Web distribution supports static, dynamic web content, on demand using progressive download & HLS and live streaming video content
RTMP supports streaming of media files using Adobe Media Server and the Adobe Real-Time Messaging Protocol (RTMP) ONLY
supports HTTPS using either
dedicated IP address, which is expensive as dedicated IP address is assigned to each CloudFront edge location
Server Name Indication (SNI), which is free but supported by modern browsers only with the domain name available in the request header
For E2E HTTPS connection,
Viewers -> CloudFront needs either self signed certificate, or certificate issued by CA or ACM
CloudFront -> Origin needs certificate issued by ACM for ELB and by CA for other origins
Security
Origin Access Identity (OAI) can be used to restrict the content from S3 origin to be accessible from CloudFront only
supports Geo restriction (Geo-Blocking) to whitelist or blacklist countries that can access the content
Signed URLs
for RTMP distribution as signed cookies aren’t supported
to restrict access to individual files, for e.g., an installation download for your application.
users using a client, for e.g. a custom HTTP client, that doesn’t support cookies
Signed Cookies
provide access to multiple restricted files, for e.g., video part files in HLS format or all of the files in the subscribers’ area of a website.
don’t want to change the current URLs
integrates with AWS WAF, a web application firewall that helps protect web applications from attacks by allowing rules configured based on IP addresses, HTTP headers, and custom URI strings
supports GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE to get object & object headers, add, update, and delete objects
only caches responses to GET and HEAD requests and, optionally, OPTIONS requests
does not cache responses to PUT, POST, PATCH, DELETE request methods and these requests are proxied back to the origin
object removal from cache
would be removed upon expiry (TTL) from the cache, by default 24 hrs
can be invalidatedexplicitly, but has a cost associated, however might continue to see the old version until it expires from those caches
objects can be invalidated only for Web distribution
change object name, versioning, to serve different version
supports adding or modifying custom headers before the request is sent to origin which can be used to
validate if user is accessing the content from CDN
identifying CDN from which the request was forwarded from, in case of multiple CloudFront distribution
for viewers not supporting CORS to return the Access-Control-Allow-Origin header for every request
supports Partial GET requests using range header to download object in smaller units improving the efficiency of partial downloads and recovery from partially failed transfers
supports compression to compress and serve compressed files when viewer requests include Accept-Encoding: gzip in the request header
supports different price class to include all regions, to include only least expensive regions and other regions to exclude most expensive regions
supports access logs which contain detailed information about every user request for both web and RTMP distribution
AWS Database Migration Service enables quick and secure data migration with minimal to zero downtime.
AWS Database Migration Service helps migration to AWS with virtually no downtime. Source database remains fully operational during the migration, minimizing downtime to applications that rely on the database.
AWS Database Migration Service can migrate
relational databases, data warehouses, NoSQL databases, and other types of data stores
data to and from most widely used commercial and open-source databases.
AWS DMS supports homogeneous migrations such as Oracle to Oracle, as well as heterogeneous migrations (using SCT) between different database platforms, such as Oracle or Microsoft SQL Server to Aurora.
AWS DMS enables both one time migration and continuous data replication with high availability and consolidate databases into a petabyte-scale data warehouse by streaming data to Redshift and S3.
AWS DMS continually monitors source and target databases, network connectivity, and the replication instance.
AWS DMS is highly resilient and self–healing. If the primary replication server fails for any reason, a backup replication server can take over with little or no interruption of service.
AWS DMS automatically manages all of the infrastructure that supports the migration server, including hardware and software, software patching, and error reporting.
In case of interruption, DMS automatically restarts the process and continues the migration from where it was halted.
AWS DMS supports Multi-AZ option to provide high-availability for database migration and continuous data replication by enabling redundant replication instances.
AWS DMS ensures that the data migration is secure. Data at rest is encrypted with AWS Key Management Service (AWS KMS) encryption. During migration, Secure Socket Layers (SSL) can be used to encrypt the in-flight data as it travels from source to target.
DMS Components
DMS Replication Instance
A DMS replication instance performs the actual data migration between source and target.
DMS replication instance is a managed EC2 instance that hosts one or more replication tasks.
Replication instance also caches the transaction logs during the migration
CPU and memory capacity of the replication instance influences the overall time required for the migration.
DMS can provide high availability and failover support using a Multi-AZ deployment.
In a Multi-AZ deployment, AWS DMS automatically provisions and maintains a standby replica of the replication instance in a different AZ
Primary replication instance is synchronously replicated to the standby replica.
If the primary replication instance fails or becomes unresponsive, the standby resumes any running tasks with minimal interruption.
Because the primary is constantly replicating its state to the standby, Multi-AZ deployment does incur some performance overhead.
Endpoints
AWS DMS uses an endpoint to access the source or target data store.
Replication tasks
AWS DMS replication task helps move a set of data from the source endpoint to the target endpoint.
Replication task required Replication instance, source and target endpoints
Replication task supports following migration type options
Full load (Migrate existing data) – Migrate the data from the source to the target database as a one time migration.
CDC only (Replicate data changes only) – Replicate only changes, while using native export tools for performing bulk data load.
Full load + CDC (Migrate existing data and replicate ongoing changes) – Performs a full data load while capturing changes on the source. After the full load is complete, captured changes are applied to the target. Once the changes reach a steady state, the applications can be switched over to the target.
LOB mode options
Don’t include LOB columns – LOB columns are excluded
Full LOB mode – Migrate complete LOBs regardless of size. AWS DMS migrates LOBs piecewise in chunks controlled by the Max LOB Size parameter. This mode is slower than using limited LOB mode.
Limited LOB mode – Truncate LOBs to the value specified by the Max LOB Size parameter. This mode is faster than using full LOB mode.
Data validation – If the data validation needs to be performed, once the migration has been completed.
AWS Schema Conversion Tool
AWS Schema Conversion Tool makes heterogeneous database migrations by automatically converting the source database schema and a majority of the database code objects, including views, stored procedures, and functions, to a format compatible with the target database.
DMS and SCT work in conjunction to both migrate databases and support ongoing replication for a variety of uses such as populating datamarts, synchronizing systems, etc.
SCT can copy database schemas for homogeneous migrations and convert them for heterogeneous migrations.
SCT clearly marks any objects that cannot be automatically converted so that they can be manually converted to complete the migration.
SCT can scan the application source code for embedded SQL statements and convert them as part of a database schema conversion project.
SCT performs cloud native code optimization by converting legacy Oracle and SQL Server functions to their equivalent AWS service thus helping modernize the applications at the same time of database migration.
Once schema conversion is complete, SCT can help migrate data from a range of data warehouses to Redshift using built-in data migration agents.
DMS Best Practices
DMS Performance
In full load, multiple tables are loaded in parallel and it is recommended to drop primary key indexes, secondary indexes, referential integrity constraints, and data manipulation language (DML) triggers.
For a full load + CDC task, it is recommended to add secondary indexes before the CDC phase. Because AWS DMS uses logical replication, secondary indexes that support DML operations should be in-place to prevent full table scans.
Replication task can be paused before the CDC phase to build indexes, create triggers, and create referential integrity constraints
Use multiple tasks for a single migration to improve performance
Disable backups and Multi-AZ on the target until ready to cut over.
Migration LOBs
DMS migrates LOBS in a two step process
creates a new row in the target table and populates the row with all data except the associated LOB value.
Update the row in the target table with the LOB data.
All LOB columns on the target table must be nullable
Limited LOB mode
default for all migration tasks
migrates all LOB values up to a user-specified size limit, default 32K
LOB values larger than the size limit must be manually migrated. typically provides the best performance.
Ensure that the Max LOB size parameter setting is set to the largest LOB size for all the tables.
Full LOB mode
migrates all LOB data in the tables, regardless of size.
provides the convenience of moving all LOB data in the tables, but the process can have a significant impact on performance.
Migrating Large Tables
Break the migration into more than one task.
Using row filtering, use a key or a partition key to create multiple tasks
Convert schema
Use SCT to convert the source objects, table, indexes, views, triggers, and other system objects into the target DDL format
DMS doesn’t perform schema or code conversion
Replication
Enable Multi-AZ for ongoing replication (for high availability and failover support)
DMS can read/write from/to encrypted DBs
AWS Certification Exam Practice Questions
Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
Open to further feedback, discussion and correction.
Which AWS service would simplify migration of a database to AWS?
AWS IoT Core is a managed cloud platform that lets connected devices easily and securely interact with cloud applications and other devices.
AWS IoT Core can support billions of devices and trillions of messages, and can process and route those messages to AWS endpoints and to other devices reliably and securely.
AWS IoT Core allows the applications to keep track of and communicate with all the devices, all the time, even when they aren’t connected.
AWS IoT Core offers
Connectivity between devices and the AWS cloud.
AWS IoT Core allows communication with connected devices securely, with low latency and with low overhead.
Communication can scale to as many devices as needed.
AWS IoT Core supports standard communication protocols (HTTP, MQTT, and WebSockets are supported currently).
Communication is secured using TLS.
Processing data sent from connected devices.
AWS IoT Core can continuously ingest, filter, transform, and route the data streamed from connected devices.
Actions can be taken based on the data and route it for further processing and analytics.
It serves as an easy to use interface for applications running in the cloud and on mobile devices to access data sent from connected devices, and send data and commands back to the devices.
AWS IoT Core Works
Connected devices, such as sensors, actuators, embedded devices, smart appliances, and wearable devices, connect to AWS IoT Core over HTTPS, WebSockets, or secure MQTT.
Communication with AWS IoT Core is secure.
HTTPS and WebSockets requests sent to AWS IoT Core are authenticated using AWS IAM or AWS Cognito, both of which support the AWS SigV4 authentication.
HTTPS requests can also be authenticated using X.509 certificates.
MQTT messages to AWS IoT Core are authenticated using X.509 certificates.
With AWS IoT Core allows using AWS IoT Core generated certificates, as well as those signed by your preferred Certificate Authority (CA).
AWS IoT Core also offers fine-grained authorization to isolate and secure communication among authenticated clients.
Device Gateway
Device Gateway forms the backbone of communication between connected devices and the cloud capabilities such as the Rules Engine, Device Shadow, and other AWS and 3rd-party services.
Device Gateway allows secure, low-latency, low-overhead, bi-directional communication between connected devices, cloud and mobile application
Device Gateway supports the pub/sub messaging pattern, which involves involves clients publishing messages on logical communication channels called ‘topics’ and clients subscribing to topics to receive messages
Device gateway enables communication between publishers and subscribers
Device Gateway scales automatically as per the demand, without any operational overhead
Rules Engine
Rules Engine enables continuous processing of data sent by connected devices.
Rules can be configured to filter and transform the data using an intuitive, SQL-like syntax.
Rules can be configured to route the data to other AWS services such as DynamoDB, Kinesis, Lambda, SNS, SQS, CloudWatch, Elasticsearch Service with built-in Kibana integration, as well as to non-AWS services, via Lambda for further processing, storage, or analytics.
Registry
Registry allows registering devices and keeping track of devices connected to AWS IoT Core, or devices that may connect in the future.
Device Shadow
Device Shadow enables cloud and mobile applications to query data sent from devices and send commands to devices, using a simple REST API, while letting AWS IoT Core handle the underlying communication with the devices.
Device Shadow accelerates application development by providing
a uniform interface to devices, even when they use one of the several IoT communication and security protocols with which the applications may not be compatible.
an always available interface to devices even when the connected devices are constrained by intermittent connectivity, limited bandwidth, limited computing ability or limited power.
Device and its Device Shadow Lifecycle
A device (such as a light bulb) is registered in the Registry.
Connected device is programmed to publish a set of its property values or ‘state (“I am ON and my color is RED”) to the AWS IoT Core service.
Device Shadow also stores the last reported state in the in AWS IoT Core.
An application (such as a mobile app controlling the light bulb) uses a RESTful API to query AWS IoT Core for the last reported state of the light bulb, without the complexity of communicating directly with the light bulb
When a user wants to change the state (such as turning the light bulb from ON to OFF), the application uses a RESTful API to request an update, i.e. sets a ‘desired’ state for the device in AWS IoT Core. AWS IoT Core takes care of synchronizing the desired state to the device.
Application gets notified when the connected device updates its state to the desired state.
AWS Certification Exam Practice Questions
Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
Open to further feedback, discussion and correction.
You need to filter and transform incoming messages coming from a smart sensor you have connected with AWS. Once messages are received, you need to store them as time series data in DynamoDB. Which AWS service can you use?
IoT Device Shadow Service (maintains device state)
Redshift
Kinesis (While Kinesis could technically be used as an intermediary between different sources, it isn’t a great way to get data into DynamoDB from an IoT device.)
Read Replica Use cases
