DynamoDB Advanced – Streams, Global Tables, TTL & Capacity Patterns

DynamoDB Advanced Patterns — Overview

DynamoDB advanced topics appear in 167 questions across DBS-C01 (129) and DAS-C01/DOP-C02. Beyond basic CRUD, the exam tests DynamoDB Streams, Global Tables, TTL, capacity modes, GSI/LSI design, single-table design, and DAX caching patterns.

DynamoDB Advanced Architecture
Streams + CDC
Change Data Capture
Trigger Lambda
Cross-region replication
Event-driven patterns
24-hour retention
Global Tables
Multi-region active-active
Eventual consistency cross-region
Conflict resolution (last writer)
Auto-replication
Local reads everywhere
Capacity & Cost
On-Demand (pay per request)
Provisioned + Auto Scaling
Reserved Capacity
Burst (300s saved)
Adaptive capacity
Access Patterns
GSI (alternate PK)
LSI (alternate SK)
Single-table design
Sparse indexes
TTL auto-delete

DynamoDB Streams

Feature Details
What Ordered, time-sequenced log of all item-level changes (insert, update, delete)
Views KEYS_ONLY, NEW_IMAGE, OLD_IMAGE, NEW_AND_OLD_IMAGES
Retention 24 hours (then deleted). Process with Lambda or Kinesis Data Streams.
Consumers Lambda trigger (most common), KCL application, Kinesis Data Streams adapter
Use cases Trigger Lambda on changes, sync to OpenSearch, aggregate analytics, cross-region replication (Global Tables use this internally)

Global Tables — Multi-Region

  • Active-active: Read AND write in any region. Changes replicate to all other regions (typically <1 second).
  • Conflict resolution: Last-writer-wins (based on timestamp). No custom conflict resolution.
  • Consistency: Eventually consistent cross-region. Strongly consistent within a single region.
  • Requirements: Streams must be enabled, same table name in all regions, On-Demand or provisioned with auto-scaling.
  • Exam pattern: “Multi-region low-latency reads AND writes” → DynamoDB Global Tables. “Multi-region reads only” → Global Tables or cross-region read replicas.

Capacity Modes — Decision

Mode Pricing Best For
On-Demand Pay per request ($1.25/M writes, $0.25/M reads) Unknown/unpredictable traffic, spiky workloads, new tables
Provisioned Pay per RCU/WCU provisioned per hour Predictable traffic, consistent utilization, cost optimization
Provisioned + Auto Scaling Scales within min/max bounds based on target utilization (default 70%) Variable but predictable patterns (scales up/down)
  • Burst capacity: DynamoDB saves unused capacity for up to 300 seconds. Short spikes use burst before throttling.
  • Adaptive capacity: Redistributes throughput across partitions to handle hot partitions (automatic).
  • Switch modes: Can switch between On-Demand and Provisioned once per 24 hours.

GSI vs LSI

Feature GSI (Global Secondary Index) LSI (Local Secondary Index)
Key Different partition key + optional sort key Same partition key + different sort key
Create Anytime (add/remove after table creation) Table creation time only (cannot add later)
Capacity Separate provisioned capacity from table Shares table’s capacity
Consistency Eventually consistent only Strongly or eventually consistent
Limit 20 per table 5 per table

TTL (Time to Live)

  • What: Automatically delete items after a specified timestamp (epoch seconds)
  • Cost: Free — no WCU consumed for TTL deletions
  • Timing: Items typically deleted within 48 hours of expiration (not instant)
  • Streams: TTL deletions appear in DynamoDB Streams (for audit/archival)
  • Use cases: Session expiration, temporary data, log retention, GDPR data deletion
  • Pattern: Set TTL → item expires → Stream captures deletion → Lambda archives to S3 (audit trail)

Single-Table Design

  • Concept: Store multiple entity types in one table using composite keys (PK: USER#123, SK: ORDER#456)
  • Why: Reduce table count, enable single-query access patterns that span entities (get user + orders in one query)
  • GSI overloading: GSI with generic attribute names (GSI1PK, GSI1SK) used for multiple access patterns
  • When NOT to: When entities have vastly different access patterns or RCU/WCU needs (hot partition risk)

Exam Tips

Exam Key Points
DBS-C01 “Trigger on item change” → DynamoDB Streams + Lambda. “Multi-region writes” → Global Tables. “Auto-delete expired sessions” → TTL. “Query by non-key attribute” → GSI. “Unpredictable traffic” → On-Demand. “Reduce read latency to microseconds” → DAX. “Hot partition” → better key design or adaptive capacity. “Export to S3” → DynamoDB Export to S3 (PITR-based, no RCU).

AWS Certification Exam Practice Questions

Question 1:

A company needs their DynamoDB table accessible for reads and writes in 3 regions (us-east-1, eu-west-1, ap-southeast-1) with single-digit millisecond latency for local users. What is the simplest solution?

  1. DynamoDB Streams + Lambda to replicate items to tables in other regions
  2. DynamoDB Global Tables — enable replication to all 3 regions for active-active access
  3. DynamoDB with DAX in each region + cross-region read replicas
  4. S3 cross-region replication with DynamoDB in each region synced manually
Show Answer

Answer: B — Global Tables provide fully managed, active-active multi-region replication. Write anywhere, read anywhere. Replication typically <1 second. No custom Lambda needed (unlike A). Built-in conflict resolution (last-writer-wins). Enable with a few clicks. DAX (C) is for read caching within a region, not cross-region replication.

Question 2:

A web application stores user sessions in DynamoDB. Sessions should automatically expire after 24 hours without the application explicitly deleting them. The company also needs to know which sessions were cleaned up (for audit). Which features achieve this?

  1. CloudWatch Events scheduled to scan and delete old items every hour
  2. TTL attribute set to creation_time + 24 hours. DynamoDB Streams captures TTL deletions → Lambda writes to audit log.
  3. DynamoDB Auto Scaling to remove old partitions
  4. Glue job that scans for expired items and deletes them nightly
Show Answer

Answer: B — TTL automatically deletes expired items at no WCU cost. Set a TTL attribute (e.g., expireAt = current_epoch + 86400). DynamoDB Streams captures TTL deletions with the full item image — Lambda processes the stream to write audit records. This is zero-maintenance (no scheduled jobs) and free (no WCU for TTL deletes). Scanning solutions (A, D) consume RCUs and add complexity.

Question 3:

A table’s provisioned capacity is set to 1000 WCU evenly across 4 partitions (250 WCU each). One partition receives 80% of writes (hot partition) and gets throttled. The table has burst capacity available. How does DynamoDB handle this?

  1. All writes to the hot partition are throttled — must redesign the key
  2. Adaptive capacity automatically redistributes throughput to the hot partition, using unused capacity from other partitions
  3. Burst capacity is only table-level, not partition-level
  4. Auto Scaling increases total capacity until the hot partition is satisfied
Show Answer

Answer: B — Adaptive capacity (enabled by default) redistributes unused throughput from idle partitions to hot partitions. If partitions 2-4 use only 50 WCU each, the remaining 600 WCU can be used by partition 1. This handles moderate imbalances. Extreme hot keys (single item getting all writes) still need key redesign. Adaptive capacity works within total table throughput.

Related Posts

References

Frequently Asked Questions

On-Demand vs Provisioned — which is cheaper?

On-Demand is cheaper when traffic is unpredictable or very spiky (utilization <18% of would-be provisioned capacity). Provisioned is cheaper for steady-state traffic with predictable patterns. Rule of thumb: if your table consistently uses >18% of peak capacity, provisioned + auto-scaling is cheaper. New tables or uncertain patterns → start On-Demand, analyze, switch later.

When should I use Global Tables vs cross-region read replicas?

Global Tables: When you need writes in multiple regions (active-active). Both regions accept writes, replication is automatic. No native cross-region read replicas in DynamoDB — Global Tables is the only option for multi-region. For read-only multi-region, Global Tables still works (just write to one region, read from any). For relational databases with read-only needs, Aurora Global is the alternative.

CloudFormation Advanced – StackSets, Drift Detection, Custom Resources & Guard

CloudFormation Advanced — Overview

CloudFormation is the #1 DOP-C02 topic (113 questions) and appears on SOA-C03 and SAP-C02. The exam goes deep on StackSets (multi-account), drift detection, custom resources, change sets, nested stacks, and CloudFormation Guard for policy-as-code validation.

CloudFormation Advanced Patterns
StackSets
Deploy to multiple accounts/regions
Organizations integration
Automatic new account deployment
Failure tolerance settings
Drift Detection
Detect manual changes
Compare actual vs template
Resource-level detail
Import resources to fix
Custom Resources
Lambda-backed
Extend CFN capabilities
Non-AWS resources
Complex provisioning logic
Guard / Validation
Policy-as-code
Pre-deployment validation
cfn-lint (syntax)
cfn-nag (security)

StackSets — Multi-Account Deployment

Feature Details
Deployment targets Specific accounts, OUs, or entire Organization
Regions Deploy to multiple regions simultaneously or sequentially
Auto-deployment Automatically deploy to new accounts added to an OU
Failure tolerance Set max failures per region/account before rollback
Concurrency Max concurrent accounts (percentage or fixed number)
Permission model Self-managed (IAM roles) or service-managed (Organizations trust)

Exam pattern: “Deploy security baseline to all accounts” → StackSets with Organization auto-deployment to target OU.

Nested Stacks vs Cross-Stack References

Pattern How Best For
Nested Stacks Parent stack references child templates (AWS::CloudFormation::Stack) Modular templates, reusable components (VPC module, DB module)
Cross-Stack (Exports) Stack A exports values, Stack B imports them (Fn::ImportValue) Independent lifecycle stacks that share values (VPC ID used by App stack)
  • Nested: Updated together as one unit. Parent manages lifecycle. Can’t update child independently.
  • Cross-Stack: Independent lifecycles. Can update one without touching the other. But: can’t delete exporting stack while imported.

Change Sets

  • What: Preview changes before executing an update (like terraform plan)
  • Shows: Resources to be added, modified, or deleted. Whether replacement is needed.
  • Replacement types: None (in-place), Conditional (may replace), Always (will delete and recreate)
  • Exam note: “Preview impact before deploying” → Create Change Set, review, then execute

Drift Detection

  • What: Detects if actual resource configuration differs from template (someone changed manually)
  • Shows: Expected vs actual values for each drifted property
  • Fix options: Update template to match actual (accept drift) OR import resource to bring under CFN management
  • Limitation: Not all resources support drift detection. Some properties are not checked.
  • Exam pattern: “Someone manually changed a resource, now updates fail” → detect drift, resolve

Custom Resources (Lambda-backed)

  • When: CFN doesn’t support a resource natively, or you need custom logic during provisioning
  • How: Define AWS::CloudFormation::CustomResource → triggers Lambda function on Create/Update/Delete
  • Lambda responds: Sends SUCCESS/FAILED + response data to CFN-provided signed URL
  • Use cases: Populate DynamoDB table on deploy, get latest AMI ID, configure third-party service, run DB migration
  • Exam trap: Lambda must respond to the signed URL. If it doesn’t (timeout/error without response), stack hangs for 1 hour.

CloudFormation Guard

  • What: Policy-as-code tool that validates templates against rules BEFORE deployment
  • Language: Domain-specific language (DSL) for writing rules
  • Example rule: “All S3 buckets must have encryption enabled” or “No Security Group allows 0.0.0.0/0 on port 22”
  • CI/CD integration: Run in CodeBuild/CodePipeline as a gate — fail deployment if rules violated
  • vs Config: Guard prevents deployment (shift-left). Config detects after deployment (reactive).

Other Validation Tools

Tool What Checks
cfn-lint Linter for CFN templates Syntax errors, invalid property values, deprecated features
cfn-nag Security-focused static analysis Overly permissive IAM, public resources, missing encryption
TaskCat Template testing (deploy to multiple regions) Actually deploys and validates, then cleans up
CloudFormation Guard Policy validation (custom rules) Compliance policies, organizational standards, security baselines

Deployment Strategies with CFN

  • Stack policies: Prevent accidental updates/deletions of critical resources (e.g., deny Update on production RDS)
  • Termination protection: Prevent accidental stack deletion (must explicitly disable first)
  • DeletionPolicy: Retain (keep resource after stack delete), Snapshot (create snapshot), Delete (default)
  • UpdateReplacePolicy: What happens when a resource needs replacement — Retain old, Snapshot old, or Delete old

Exam Tips

Exam Key Points
DOP-C02 “Deploy to all org accounts” → StackSets. “Preview before update” → Change Set. “Manual change broke stack” → Drift Detection. “Unsupported resource” → Custom Resource (Lambda). “Validate security before deploy” → cfn-nag or Guard. “Modular reusable templates” → Nested Stacks. “Share VPC ID between stacks” → Cross-Stack Exports. “Prevent RDS deletion” → DeletionPolicy: Retain.

AWS Certification Exam Practice Questions

Question 1:

A company needs to deploy a security baseline (CloudTrail, Config, GuardDuty) to all 200 accounts in their Organization. When new accounts are created, the baseline must deploy automatically. Which approach achieves this?

  1. Create individual stacks in each account using a script
  2. CloudFormation StackSets with service-managed permissions, targeting the Organization root OU, with auto-deployment enabled
  3. Deploy via AWS Config conformance packs to each account
  4. Use Control Tower account factory blueprints
Show Answer

Answer: B — StackSets with service-managed permissions uses Organizations trust (no manual IAM role setup per account). Targeting the root OU deploys to all accounts. Auto-deployment ensures new accounts added to the OU automatically receive the stack. This is the standard pattern for organization-wide baseline deployment. Control Tower (D) also works but StackSets gives more template flexibility.

Question 2:

A CloudFormation stack update fails with “Resource X has been modified outside of CloudFormation.” The team needs to understand what changed and resolve the issue. Which steps should they take?

  1. Delete the stack and recreate it from the template
  2. Run drift detection to identify the differences, then either update the template to match actual state or manually revert the resource to match the template
  3. Force the update with –continue-update-rollback
  4. Import the resource into a new stack
Show Answer

Answer: B — Drift detection shows exactly which properties were changed manually (expected vs actual values). Then either: (1) Update your template to accept the manual change (if valid), or (2) manually revert the resource to match the template. Both resolve the conflict. Deleting (A) destroys resources. Force continue (C) may skip the conflicting resource without fixing it.

Question 3:

A team needs to populate a DynamoDB table with seed data every time their CloudFormation stack is deployed. CloudFormation doesn’t natively support writing items to DynamoDB. How can this be achieved within the same stack?

  1. Use a CloudFormation Init script on an EC2 instance
  2. Create a Custom Resource backed by a Lambda function that writes items to DynamoDB on the Create event
  3. Add a UserData script to an EC2 instance in the stack
  4. Use CloudFormation macros to transform the template
Show Answer

Answer: B — Custom Resources invoke a Lambda function during stack operations (Create/Update/Delete). The Lambda writes seed data to DynamoDB on Create, and optionally cleans up on Delete. This runs within the stack lifecycle — downstream resources can depend on it. Lambda responds with SUCCESS to the CFN signed URL. No EC2 needed (A, C add unnecessary compute).

Related Posts

References

Frequently Asked Questions

StackSets vs Control Tower — when to use which?

StackSets: Deploy any CloudFormation template to multiple accounts/regions. Full template flexibility. Use for custom baselines, application infrastructure, or anything CFN can define. Control Tower: Deploys a prescriptive AWS landing zone with guardrails (SCPs + Config rules). Use for initial account governance setup. Many teams use both: Control Tower for governance + StackSets for custom deployments.

What is the difference between cfn-lint, cfn-nag, and Guard?

cfn-lint: Syntax/schema validation (is the template valid?). cfn-nag: Security static analysis (are there security issues?). Guard: Custom policy validation (does it meet OUR organizational rules?). Use all three in CI/CD: cfn-lint first (catch syntax), then cfn-nag (catch security), then Guard (enforce org policies). All run before deployment — shift-left validation.

RDS & Aurora Performance – Read Replicas, Proxy, Global & Performance Insights

RDS & Aurora Performance Architecture — Overview

Database performance is the #1 DBS-C01 topic (218 questions) and appears heavily on SAA-C03 and SAP-C02. The exam tests Read Replicas, Provisioned IOPS, Performance Insights, RDS Proxy, Aurora Serverless, and Aurora Global Database patterns.

RDS/Aurora Performance Optimization Stack
Read Scaling
Read Replicas (up to 15 Aurora)
Reader endpoint (LB)
Cross-region replicas
ElastiCache for hot data
Connection Management
RDS Proxy (pooling)
Connection multiplexing
Lambda integration
Faster failover
Storage Performance
gp3 (3000 IOPS baseline)
io2 (up to 256K IOPS)
Aurora auto-grows (128TB)
Optimized Reads (local NVMe)
Monitoring
Performance Insights
Enhanced Monitoring (OS)
CloudWatch metrics
Slow query log

Aurora vs RDS — Performance Differences

Feature RDS (MySQL/PostgreSQL) Aurora
Read Replicas Up to 5, async replication (seconds lag) Up to 15, shared storage (milliseconds lag)
Storage EBS-based (gp3, io2), manual sizing Auto-growing (10GB to 128TB), 6 copies across 3 AZs
Failover 60-120 seconds ~30 seconds (with RDS Proxy: seconds)
Throughput Standard MySQL/PostgreSQL performance Up to 5x MySQL, 3x PostgreSQL (AWS claims)
Global Cross-region read replicas (manual promotion) Global Database (<1s replication, managed failover)

Read Replica Patterns

  • Reader endpoint (Aurora): Load-balances across all replicas automatically. Application uses reader endpoint for reads.
  • Custom endpoints: Route specific queries to specific replicas (e.g., analytics queries to larger replicas)
  • Cross-region replicas: For global reads and DR. Aurora Global: <1s replication. RDS: async (minutes possible)
  • Replica priority tiers: Control which replica promotes on failover (tier 0 = highest priority)
  • Read/write splitting: Application sends writes to writer endpoint, reads to reader endpoint. Frameworks: ProxySQL, Spring read-replica routing.

RDS Proxy — Connection Pooling

  • Problem: Lambda/serverless creates hundreds of connections. Each connection consumes DB memory. Connection storms on scale-out.
  • Solution: RDS Proxy maintains persistent connection pool to DB. Lambda connects to Proxy (fast), Proxy multiplexes to DB (few connections).
  • Failover: RDS Proxy detects failover and redirects connections to new primary without application reconnection (~66% faster failover).
  • IAM Auth: Proxy supports IAM authentication (no password in code). Manages secrets from Secrets Manager.
  • Pinning: Some session state pins connections (prepared statements, temp tables). Minimize pinning for best multiplexing.

Storage Performance

Type IOPS Throughput Use Case
gp3 3,000 baseline (up to 16,000) 125 MiB/s (up to 1,000) Most workloads (default, balanced cost/performance)
io2 Block Express Up to 256,000 Up to 4,000 MiB/s Extreme IOPS: OLTP, SAP HANA, high-frequency trading
Aurora Storage Managed (auto-optimizes) Managed No provisioning — Aurora manages storage performance

Aurora Serverless v2

  • Auto-scales: ACUs (Aurora Capacity Units) scale up/down in seconds based on load
  • Range: Min 0.5 ACU to max 256 ACU. Set min/max bounds.
  • Mixed clusters: Combine provisioned writer + Serverless v2 readers (scale reads on demand)
  • Use case: Variable workloads, dev/test, infrequent access databases that need instant scaling
  • vs Provisioned: Serverless costs more per ACU-hour but saves when utilization is low/variable

Aurora Global Database

  • Replication: Storage-level replication, <1 second lag (not logical replication)
  • Regions: 1 primary + up to 5 secondary regions (16 read replicas per region)
  • Failover: Managed cross-region failover (RTO <1 minute). Secondary promotes to primary.
  • Use case: Global low-latency reads + DR across regions
  • Write forwarding: Secondary region can forward writes to primary (application doesn’t need to know which is primary)

Performance Insights & Monitoring

  • Performance Insights: Visual dashboard showing DB load by wait events, SQL statements, hosts, users. Identifies bottleneck queries.
  • Enhanced Monitoring: OS-level metrics (CPU per process, memory, swap, I/O) at 1-second granularity. More detail than CloudWatch.
  • Slow query log: Log queries exceeding a time threshold. Analyze for optimization.
  • CloudWatch: Standard metrics (CPU, connections, IOPS, replica lag). Set alarms.

Exam Tips

Exam Key Points
DBS-C01 “Reduce read load” → Read Replicas + reader endpoint. “Lambda connection issues” → RDS Proxy. “High IOPS” → io2 or Aurora. “Global low-latency reads” → Aurora Global Database. “Variable workload DB” → Aurora Serverless v2. “Identify slow queries” → Performance Insights. “Faster failover” → RDS Proxy (66% faster). “Scale reads without provisioning” → Aurora Serverless v2 readers.

AWS Certification Exam Practice Questions

Question 1:

An Aurora MySQL database serves a global application. Users in Europe experience 200ms read latency because the database is in us-east-1. The company needs European reads under 20ms without application changes. Which solution achieves this?

  1. Create a cross-region read replica in eu-west-1
  2. Configure Aurora Global Database with a secondary region in eu-west-1. European application uses the reader endpoint in eu-west-1.
  3. Use ElastiCache in eu-west-1 to cache query results
  4. Deploy CloudFront to cache database responses
Show Answer

Answer: B — Aurora Global Database provides <1 second storage-level replication to secondary regions. European users connect to the reader endpoint in eu-west-1 (local reads, <20ms). No application changes — just point reads to the local endpoint. Cross-region read replica (A) uses logical replication (higher lag). ElastiCache (C) requires code changes for caching logic.

Question 2:

A serverless application (Lambda + API Gateway) connects to an RDS PostgreSQL database. During traffic spikes, the database reaches max_connections (200) and Lambda functions fail. The database CPU is only at 30%. What is the root cause and fix?

  1. Database too small — upgrade instance size for more connections
  2. Connection exhaustion from Lambda — add RDS Proxy for connection pooling and multiplexing
  3. Increase max_connections parameter in the parameter group
  4. Add read replicas to distribute connection load
Show Answer

Answer: B — Lambda creates a new DB connection per invocation (no connection reuse across invocations). 200 concurrent Lambdas = 200 connections = maxed out. CPU at 30% confirms the DB isn’t overloaded — it’s connection count. RDS Proxy pools connections: hundreds of Lambdas share a small pool of DB connections (e.g., 20). Increasing max_connections (C) helps temporarily but wastes DB memory and doesn’t scale.

Question 3:

A DBA needs to identify which SQL queries are causing high DB load during peak hours. They need to see which queries consume the most CPU and I/O, and which wait events are occurring. Which tool provides this visibility?

  1. CloudWatch CPU Utilization metric with anomaly detection
  2. RDS Performance Insights — shows DB load breakdown by wait events, top SQL statements, and contributing sessions
  3. Enhanced Monitoring for OS-level process details
  4. AWS X-Ray tracing through the application
Show Answer

Answer: B — Performance Insights shows Average Active Sessions (AAS) broken down by wait events (CPU, I/O, locks, network), top SQL statements consuming resources, and contributing hosts/users. You can identify exactly which query causes load and what it’s waiting on. CloudWatch (A) shows CPU but not SQL-level detail. Enhanced Monitoring (C) shows OS processes but not SQL queries.

Related Posts

References

Frequently Asked Questions

When should I use Aurora vs standard RDS?

Aurora: When you need higher performance (5x MySQL), more read replicas (15 vs 5), faster failover (30s vs 120s), auto-growing storage, or global database. Standard RDS: When you need specific engine versions not yet on Aurora, lowest cost for small workloads, or Oracle/SQL Server (Aurora only supports MySQL/PostgreSQL). Aurora costs ~20% more but offers significantly better performance and availability.

RDS Proxy vs connection pooling in application (PgBouncer)?

RDS Proxy: Managed, integrates with IAM/Secrets Manager, faster failover (redirects connections seamlessly), no infrastructure to manage. Self-managed (PgBouncer): Full control, cheaper (EC2 cost only), more configuration options. Use RDS Proxy for Lambda (must-have), or when you want managed failover. Use PgBouncer when you need specific pooling configurations or cost sensitivity.

AWS EMR & Spark Architecture – Cluster Modes, EMRFS & Optimization

AWS EMR & Spark Architecture — Overview

EMR is the largest topic on DAS-C01 (274 questions) and appears on DEA-C01 and MLA-C01 for data processing. The exam tests cluster modes (transient vs persistent), Spark optimization, EMRFS (S3 integration), instance selection, and EMR Serverless vs managed clusters.

EMR Architecture Options
EMR on EC2
Full cluster control
Master + Core + Task nodes
Spot for Task nodes
HDFS on Core nodes
Bootstrap actions
Custom AMIs
Best: Complex jobs, custom config
EMR on EKS
Spark on Kubernetes
Shared EKS cluster
Multi-tenant
Fine-grained resource
Container-based
Consolidate infra
Best: K8s teams, multi-tenant
EMR Serverless
No cluster management
Auto-scales workers
Pay per vCPU-hour used
Pre-initialized capacity
Spark, Hive, Presto
Submit jobs only
Best: Variable workloads, no ops

EMR on EC2 — Cluster Architecture

Node Type Role Scaling Instance Strategy
Primary (Master) YARN ResourceManager, NameNode, job tracking Single node (HA: 3 for multi-master) On-Demand only (critical)
Core HDFS DataNode + task execution Can scale out (not in easily) On-Demand or mix (stores HDFS data)
Task Task execution only (no HDFS) Freely scale in/out Spot instances (no data loss risk)

Transient vs Persistent Clusters

Pattern How Best For
Transient Launch cluster → run job → terminate. Data on S3 (EMRFS). Batch ETL, scheduled jobs, cost optimization (pay only during execution)
Persistent Long-running cluster. Interactive queries, notebooks. Ad-hoc analysis, Jupyter notebooks, interactive Spark/Presto/Hive

Exam rule: “Minimize cost for batch ETL” → transient cluster + S3 (EMRFS). “Interactive analysis” → persistent cluster.

EMRFS — S3 as Primary Storage

  • What: EMR’s implementation of HDFS file system interface on top of S3
  • Why: Decouple storage from compute. Cluster terminates, data persists in S3. Enables transient clusters.
  • Consistent view: EMRFS provides read-after-write consistency (S3 now natively consistent)
  • S3 Committer: EMRFS S3-optimized committer avoids the rename problem (faster job completion)
  • vs HDFS: Use HDFS only for intermediate shuffle data or when sub-second latency needed. S3 for input/output data.

Spark Optimization

  • Partitioning: Match parallelism to data size. Too few partitions = underutilized cluster. Too many = scheduling overhead.
  • Memory: spark.executor.memory + spark.executor.memoryOverhead. OOM = increase memory or reduce partition size.
  • Shuffle: Expensive operation (data crosses network). Reduce with: broadcast joins, coalesce, partition-wise operations.
  • Dynamic Allocation: Automatically scales executors based on workload. Enable for variable job sizes.
  • Instance Fleet: Mix instance types in a fleet (Spot diversification). EMR picks cheapest available. Reduces Spot interruptions.

EMR Serverless vs Glue vs EMR on EC2

Feature EMR on EC2 EMR Serverless Glue
Management You manage cluster Serverless (no cluster) Serverless (no cluster)
Frameworks Spark, Hive, Presto, HBase, Flink, Pig Spark, Hive, Presto Spark (PySpark), Python Shell
Custom config Full (bootstrap, custom AMI, libs) Limited (custom JARs, spark-submit args) Limited (Glue libraries)
Data Catalog Can use Glue Data Catalog as metastore Glue Data Catalog Native integration
Best for Complex multi-framework, custom dependencies Spark/Hive without cluster ops ETL-focused, visual designer, bookmarks

Exam Tips

Exam Key Points
DAS-C01 “Cost-effective batch” → transient cluster + EMRFS (S3). “Spot instances” → Task nodes only (no HDFS data). “Interactive queries” → persistent cluster + Presto/Hive. “Cluster terminates data lost” → was using HDFS, switch to EMRFS. “Scale processing not storage” → EMRFS. “Spark OOM” → increase executor memory or reduce partition size. “Cheapest Spark” → EMR Serverless for variable loads.

AWS Certification Exam Practice Questions

Question 1:

A company runs nightly Spark ETL jobs that process 2TB of data. The cluster is provisioned 24/7 but only runs jobs for 3 hours each night. How can they reduce costs by 80%+?

  1. Use Reserved Instances for the cluster
  2. Switch to transient cluster pattern — launch cluster before job, use EMRFS (S3) for data, terminate after job completes
  3. Reduce cluster size and run jobs longer
  4. Switch to EMR on EKS
Show Answer

Answer: B — Transient clusters run only during job execution (3 hours/day instead of 24). Store data in S3 (EMRFS), not HDFS — data persists after cluster terminates. Pay for 3 hours instead of 24 = 87.5% cost reduction. Add Spot instances for Task nodes for additional savings. This is the standard cost optimization pattern for batch ETL.

Question 2:

A Spark job on EMR occasionally fails because Spot instances in the Task node fleet are reclaimed. The job must complete reliably while still using Spot for cost savings. How should this be configured?

  1. Use On-Demand instances only (no Spot)
  2. Use Instance Fleets with multiple instance types for Task nodes (Spot diversification), keep Core nodes On-Demand, enable Spark checkpointing
  3. Increase the number of Task nodes to compensate for lost instances
  4. Use EMR Serverless instead
Show Answer

Answer: B — Instance Fleets with diverse instance types reduce Spot interruption probability (EMR picks from available pools). Core nodes On-Demand ensures HDFS/shuffle data isn’t lost. Spark checkpointing allows recovery from interrupted tasks without restarting entire job. This is the best-practice pattern for reliable Spot usage on EMR.

Question 3:

An analytics team needs to run interactive Presto queries on data in S3 alongside scheduled Spark ETL jobs. They want a single cluster for both workloads. Which EMR configuration supports this?

  1. Two separate transient clusters — one for Presto, one for Spark
  2. Persistent cluster with YARN capacity scheduler — allocate queues for Presto (interactive) and Spark (batch) with resource limits
  3. EMR Serverless for both workloads
  4. Glue for Spark + Athena for queries (no EMR needed)
Show Answer

Answer: B — Persistent cluster with YARN capacity scheduler allows resource sharing between workloads. Presto gets a dedicated queue with guaranteed resources (low-latency interactive). Spark batch jobs use remaining capacity. Both share the same cluster infrastructure and EMRFS data. Option D is valid but doesn’t use “a single cluster” as specified.

Related Posts

References

Frequently Asked Questions

EMR vs Glue — when to use which?

Glue: Simpler, fully serverless, Data Catalog native, visual designer, job bookmarks. Best for ETL pipelines. EMR: Full Hadoop ecosystem (Spark, Hive, Presto, HBase, Flink), custom configurations, better for complex multi-framework workloads, interactive analysis. Use Glue for standard ETL. Use EMR when you need frameworks beyond Spark or custom cluster configuration.

When should I use EMRFS vs HDFS?

EMRFS (S3): Primary storage for input/output data. Survives cluster termination. Enables transient clusters. Use for most data. HDFS: Only for intermediate data needing lowest latency (shuffle data, temporary computation). HDFS is lost when cluster terminates. Modern best practice: EMRFS for everything, HDFS only as automatic intermediate storage.

ALB vs NLB vs GWLB – Load Balancer Advanced Decision Guide

ALB vs NLB vs GWLB — Load Balancer Decision Guide

Load balancer selection appears in 67 ANS-C01 questions plus heavily on SAA-C03 and SAP-C02. The exam tests when to use ALB vs NLB, GWLB for transparent inspection, target group types, cross-zone behavior, and advanced features like connection draining, slow start, and sticky sessions.

Load Balancer Decision Tree
What does your application need?
ALB (Layer 7)
HTTP/HTTPS/gRPC
Path-based routing
Host-based routing
Header/query routing
WebSocket
Lambda targets
Cognito auth
Redirect/fixed response
Use: Web apps, microservices, APIs
NLB (Layer 4)
TCP/UDP/TLS
Static IPs (1 per AZ)
Millions of req/sec
Ultra-low latency
Preserve source IP
PrivateLink compatible
Long-lived connections
Non-HTTP protocols
Use: Gaming, IoT, financial, TCP
GWLB (Layer 3)
Transparent inspection
GENEVE encapsulation
Bumps-in-the-wire
Preserves all headers
Third-party appliances
Firewall/IDS/IPS
Scales appliance fleet
Gateway LB Endpoint
Use: Security appliances

Detailed Comparison

Feature ALB NLB GWLB
Layer 7 (HTTP/HTTPS) 4 (TCP/UDP/TLS) 3 (IP packets)
Static IP No (use Global Accelerator for static IPs) Yes (1 per AZ, EIP assignable) No (uses GWLBe)
Performance Millions req/sec Millions req/sec, ultra-low latency (~100μs) Millions pps
Source IP X-Forwarded-For header Preserved natively (target sees real client IP) Preserved (transparent)
TLS termination Yes (ACM certificate) Yes (TLS listener) or passthrough No (transparent)
Health checks HTTP/HTTPS (path, status code) TCP, HTTP, HTTPS TCP, HTTP, HTTPS
PrivateLink Not directly (needs NLB in front) Yes (expose as VPC Endpoint Service) Yes (GWLBe)
Cross-zone Enabled by default (free) Disabled by default (costs when enabled) Disabled by default

Target Group Types

Type Targets Use Case
Instance EC2 instance IDs Standard EC2 workloads, ASG integration
IP Private IP addresses ECS (awsvpc mode), on-premises targets (via DX/VPN), cross-VPC
Lambda Lambda function ARN Serverless backends (ALB only)
ALB ALB ARN (NLB → ALB chain) Static IPs (NLB) + Layer 7 routing (ALB) combined

Advanced Features

  • Connection draining (deregistration delay): When target deregisters, existing connections complete (default 300s). Set lower for faster deployments.
  • Slow start: New targets gradually receive traffic (avoids overwhelming cold instance). Ramps up over configurable duration.
  • Sticky sessions: ALB uses cookie (application or duration-based). NLB uses source IP hash. Breaks even distribution — avoid if possible.
  • Cross-zone LB: ALB: enabled by default (distributes evenly across all targets in all AZs). NLB: disabled by default (each AZ node only routes to targets in its AZ).
  • mTLS: ALB supports mutual TLS (verify client certificates). Requires trust store configuration.

When to Chain NLB → ALB

  • Problem: Need static IPs (NLB) + path-based routing (ALB)
  • Solution: NLB (static IPs, PrivateLink compatible) → ALB target group → application targets
  • Also for: Exposing ALB-backed services via PrivateLink (PrivateLink requires NLB/GWLB)
  • Alternative: ALB + Global Accelerator (provides static IPs without NLB)

Exam Tips

Exam Key Points
ANS-C01 “Static IP” → NLB or NLB→ALB. “Path-based routing” → ALB. “TCP/UDP non-HTTP” → NLB. “Preserve client IP without headers” → NLB. “Third-party firewall appliance” → GWLB. “PrivateLink” → NLB or GWLB endpoint. “Ultra-low latency” → NLB. “Lambda target” → ALB. “Cross-zone disabled by default” → NLB (gotcha). “mTLS client certs” → ALB.

AWS Certification Exam Practice Questions

Question 1:

A financial services API requires: static IP addresses for client allowlisting, Layer 7 path-based routing for different API versions (/v1/* vs /v2/*), and TLS termination. No single AWS load balancer provides all three. What architecture satisfies all requirements?

  1. ALB with Global Accelerator (provides static IPs + Layer 7 routing)
  2. NLB (static IPs) with ALB as target (provides Layer 7 routing + TLS termination)
  3. NLB with TLS listener and path-based routing
  4. GWLB with ALB targets
Show Answer

Answer: B — NLB provides static IPs (one per AZ, EIP assignable) for client allowlisting. ALB as target group provides path-based routing (/v1/*, /v2/*) and TLS termination. The chain NLB→ALB combines both capabilities. Global Accelerator (A) also provides static IPs but the question implies direct IP allowlisting which NLB handles natively. NLB alone (C) doesn’t support path-based routing.

Question 2:

A gaming company’s servers handle millions of concurrent TCP connections on a custom protocol (not HTTP). They need the load balancer to preserve the client’s source IP address so game servers can identify players. Which load balancer type preserves source IP natively for TCP?

  1. ALB with X-Forwarded-For header
  2. NLB — preserves source IP natively for TCP targets (no proxy protocol or header needed)
  3. GWLB with GENEVE encapsulation
  4. Classic Load Balancer with proxy protocol
Show Answer

Answer: B — NLB operates at Layer 4 and preserves the source IP address natively for instance and IP targets. The game server sees the actual client IP without any headers or protocol modifications. ALB (A) replaces source IP with its own — requires X-Forwarded-For header parsing (only works for HTTP). For non-HTTP TCP protocols, NLB is the only option that preserves source IP without modification.

Question 3:

After deploying new instances into an ALB target group, the new instances are immediately overwhelmed with traffic and fail health checks. Existing instances are fine. How can the new instances be gradually introduced?

  1. Increase the health check interval to give instances more time
  2. Enable slow start mode on the target group — new targets receive gradually increasing traffic over a configurable duration
  3. Deploy one instance at a time with manual monitoring
  4. Reduce the connection timeout on the ALB
Show Answer

Answer: B — Slow start mode gradually increases the proportion of traffic sent to newly registered targets over a configurable duration (30-900 seconds). This gives the application time to warm caches, initialize connections, and reach steady state before receiving full load. Without slow start, new targets immediately receive their equal share — which can overwhelm cold instances.

Question 4:

A company has 4 instances: 3 in AZ-A, 1 in AZ-B. With NLB’s default cross-zone disabled, the single instance in AZ-B is overwhelmed (receiving 50% of traffic) while AZ-A instances are underutilized. How should this be resolved?

  1. Add more instances to AZ-B to handle 50% of traffic
  2. Enable cross-zone load balancing on the NLB — traffic distributes evenly across all 4 targets regardless of AZ
  3. Remove AZ-B from the NLB to stop routing there
  4. Switch to ALB which has cross-zone enabled by default
Show Answer

Answer: B — NLB cross-zone disabled means each AZ node routes only to targets in its AZ. With 2 AZ nodes: AZ-A node sends 50% to 3 instances (17% each), AZ-B node sends 50% to 1 instance (50% alone). Enabling cross-zone distributes across all 4 targets evenly (25% each). Note: NLB cross-zone has a data transfer charge. ALB (D) would also work but changes the load balancer type entirely.

Question 5:

A company wants to expose an internal ALB-backed microservice to partner accounts via PrivateLink. However, PrivateLink VPC Endpoint Services only support NLB or GWLB as the load balancer. How can they expose the ALB service via PrivateLink?

  1. PrivateLink now supports ALB directly
  2. Create an NLB with the ALB as a target (ALB-type target group), then create a VPC Endpoint Service pointing to the NLB
  3. Replace ALB with NLB for the microservice
  4. Use VPC Peering instead of PrivateLink
Show Answer

Answer: B — NLB supports ALB as a target type. Create NLB → target group type “ALB” → point to existing ALB. Then create VPC Endpoint Service backed by the NLB. Partners create Interface Endpoints to access the service. This preserves all ALB Layer 7 features (path routing, auth) while providing PrivateLink access. No need to replace the ALB (C).

Related Posts

References

Frequently Asked Questions

ALB vs NLB — how do I choose?

ALB: When you need HTTP-aware features — path/host routing, header inspection, Lambda targets, WebSocket, Cognito auth, redirects. NLB: When you need TCP/UDP (non-HTTP), static IPs, ultra-low latency, source IP preservation, PrivateLink, or millions of connections. Rule of thumb: HTTP/HTTPS application → ALB. Everything else → NLB.

Why is NLB cross-zone disabled by default?

Two reasons: (1) Cost — cross-zone NLB incurs data transfer charges between AZs. ALB cross-zone is free. (2) Latency — without cross-zone, traffic stays in the same AZ (lower latency). For most applications, enable cross-zone for even distribution. Disable when cost-sensitive or when you want AZ-local routing (each AZ is independent).

Route 53 Resolver & Hybrid DNS – DNSSEC, Forwarding & Split-Horizon

Route 53 Resolver & Hybrid DNS — Overview

Hybrid DNS is a key ANS-C01 topic (85 questions) that tests Resolver endpoints (inbound/outbound), conditional forwarding, DNSSEC, split-horizon DNS, and private hosted zone associations. Understanding DNS resolution flow between on-premises and AWS is critical.

Hybrid DNS Architecture
On-Premises DNS
(Active Directory DNS)
Resolves: corp.internal
→ Inbound Endpoint →
(on-prem queries AWS)
← Outbound Endpoint ←
(AWS queries on-prem)
Route 53 Resolver
(VPC .2 address)
Resolves: aws.internal
+ public DNS
Forwarding Rules: *.corp.internal → forward to on-prem DNS (outbound) | *.aws.internal → private hosted zone (local)

Resolver Endpoints

Endpoint Direction Use Case
Inbound On-premises → AWS (queries FROM on-prem resolved by Route 53) On-prem servers need to resolve AWS private hosted zones (e.g., rds.internal.aws)
Outbound AWS → On-premises (queries FROM VPC forwarded to on-prem DNS) EC2 instances need to resolve on-prem domains (e.g., ldap.corp.internal)
  • Inbound endpoint: Creates ENIs in your VPC with IP addresses. On-prem DNS forwards to these IPs. Requires DX/VPN connectivity.
  • Outbound endpoint: Creates ENIs that forward DNS queries to on-prem DNS servers. Uses forwarding rules to determine which domains go where.
  • Multi-AZ: Always deploy endpoints in 2+ AZs for availability (min 2 ENIs per endpoint).

Forwarding Rules

  • Conditional forwarding: “If query matches *.corp.internal → forward to 10.0.1.53 (on-prem DNS)”
  • System rules: Auto-created for VPC CIDR reverse DNS and private hosted zones
  • Recursive: Default behavior — Route 53 Resolver recursively resolves public domains
  • Rule sharing: Share forwarding rules via RAM to other accounts (centralized DNS management)
  • Rule priority: Most specific domain match wins (corp.internal beats .internal beats .)

Private Hosted Zones

  • What: DNS zone accessible only from associated VPCs (not public internet)
  • Cross-account: Associate private hosted zone with VPCs in other accounts (via CLI/API authorization)
  • Split-horizon: Same domain name (example.com) resolves differently: public hosted zone → internet users, private hosted zone → VPC users. Private takes precedence inside VPC.
  • Overlap: If multiple private hosted zones match, most specific wins (a.b.example.com > b.example.com)

DNSSEC

  • What: Cryptographic signing of DNS records to prevent spoofing/tampering
  • Route 53 supports: DNSSEC signing for public hosted zones (KMS-backed signing key)
  • Validation: Route 53 Resolver validates DNSSEC signatures on responses (enable per VPC)
  • Chain of trust: Root → TLD → domain. All must have DNSSEC for full validation.
  • Exam note: “Prevent DNS spoofing” → DNSSEC signing + validation. “DNS poisoning protection” → DNSSEC.

Route 53 Resolver DNS Firewall

  • What: Filter DNS queries from your VPC (block queries to malicious domains)
  • Rules: ALLOW, BLOCK, or ALERT on domain lists (managed or custom)
  • Managed lists: AWS provides threat intelligence domain lists (malware, botnet C2)
  • Use case: Prevent DNS-based data exfiltration, block known malicious domains, compliance
  • Priority: DNS Firewall → Forwarding Rules → Resolver. Firewall evaluates first.

Exam Tips

Exam Key Points
ANS-C01 “On-prem resolve AWS private zones” → Inbound Resolver endpoint. “AWS resolve on-prem domains” → Outbound + forwarding rules. “Centralize DNS rules across accounts” → share rules via RAM. “Prevent DNS spoofing” → DNSSEC. “Block malicious DNS queries” → DNS Firewall. “Same domain different answers internal vs external” → split-horizon (public + private hosted zone). “Cross-account private hosted zone” → associate + authorize.

AWS Certification Exam Practice Questions

Question 1:

A company has on-premises Active Directory DNS resolving corp.internal. EC2 instances in AWS need to resolve hostnames in corp.internal (e.g., ldap.corp.internal). The VPCs are connected to on-premises via Direct Connect. Which configuration enables this?

  1. Create a Route 53 private hosted zone named corp.internal with manual records
  2. Create a Route 53 Resolver outbound endpoint + forwarding rule for corp.internal pointing to on-premises DNS server IPs
  3. Configure EC2 instances to use on-premises DNS servers directly (change DHCP options)
  4. Create an inbound resolver endpoint for on-premises to query
Show Answer

Answer: B — Outbound endpoint forwards DNS queries from VPC to on-premises DNS. Forwarding rule specifies: if query matches *.corp.internal → forward to on-prem DNS IPs (10.0.1.53, 10.0.2.53). EC2 instances use the VPC Resolver (.2 address) as normal — it transparently forwards matching queries. DHCP option (C) works but loses Route 53 features and breaks AWS service DNS. Inbound (D) is the reverse direction.

Question 2:

On-premises servers need to resolve private hostnames in AWS (e.g., mydb.internal.aws which points to an RDS instance in a private hosted zone). Which configuration allows on-premises DNS to resolve these AWS-hosted names?

  1. Make the private hosted zone public
  2. Create a Route 53 Resolver inbound endpoint in the VPC. Configure on-premises DNS to forward *.internal.aws queries to the inbound endpoint IPs.
  3. Create a public hosted zone with the same records
  4. Add the records to on-premises DNS manually
Show Answer

Answer: B — Inbound endpoint creates ENIs in your VPC that accept DNS queries from outside the VPC. On-premises DNS is configured with a conditional forwarder: *.internal.aws → inbound endpoint IPs. Queries travel over DX/VPN to the endpoint, Route 53 Resolver resolves from the private hosted zone, returns the answer. Making it public (A) exposes internal records to the internet.

Question 3:

A company wants the domain api.example.com to resolve to a public ALB IP for internet users, but to a private ALB IP for users inside the VPC (avoiding internet round-trip). How should this be configured?

  1. Create two A records in the public hosted zone with geolocation routing
  2. Create a public hosted zone with public ALB record + a private hosted zone (same name) with private ALB record associated with the VPC
  3. Use CloudFront to route differently based on source
  4. Configure the ALB to return different IPs based on source IP
Show Answer

Answer: B — Split-horizon DNS: public hosted zone (api.example.com → public ALB) serves internet queries. Private hosted zone (api.example.com → private ALB IP) serves VPC queries. When a VPC instance queries api.example.com, the private zone takes precedence (VPC Resolver checks private zones first). Internet users hit the public zone. Same domain, different answers based on source.

Related Posts

References

Frequently Asked Questions

Inbound vs Outbound endpoint — which do I need?

Inbound: When on-premises needs to resolve AWS domains (on-prem → AWS). On-prem DNS forwards to inbound endpoint IPs. Outbound: When AWS instances need to resolve on-premises domains (AWS → on-prem). Forwarding rules send matching queries to on-prem DNS. Most hybrid environments need BOTH: inbound for on-prem→AWS resolution, outbound for AWS→on-prem resolution.

What is split-horizon DNS?

Same domain name resolves to different IPs depending on where the query originates. Public hosted zone serves internet users (public IPs). Private hosted zone serves VPC users (private IPs). VPC Resolver checks private zones first — if a match exists, it returns the private answer without querying the public zone. Common for internal vs external access to the same service.

VPC Advanced Networking – Secondary CIDRs, ENI, Prefix Lists & IPv6

VPC Advanced Networking — CIDR, ENI, Prefix Lists & IPv6

VPC advanced topics appear in 134 ANS-C01 questions. Beyond basic VPC setup, the exam tests secondary CIDRs, ENI management, managed prefix lists, IPv6 dual-stack, BYOIP, and VPC endpoint policies.

VPC Advanced Features
CIDR Management
Secondary CIDRs (up to 5)
Non-overlapping requirement
RFC 1918 + public ranges
100.64.0.0/10 (shared)
ENI (Elastic Network Interface)
Multiple per instance
Move between instances (failover)
Secondary private IPs
Different SGs per ENI
Prefix Lists
Managed IP ranges
Reuse across SGs/routes
AWS-managed (CloudFront, S3)
Share via RAM
IPv6 / Dual-Stack
All IPv6 are public
Egress-only IGW (outbound only)
NAT64 + DNS64
Dual-stack subnets

Secondary CIDRs

  • Why: VPC runs out of IP addresses, need to expand without recreating
  • Add: Up to 5 IPv4 CIDR blocks per VPC (original + 4 secondary)
  • Rules: Cannot overlap with existing CIDRs or peered VPC CIDRs. Can be from different RFC 1918 ranges.
  • 100.64.0.0/10: Shared address space — useful when RFC 1918 ranges exhausted across organization
  • Exam pattern: “VPC running out of IPs” → add secondary CIDR + new subnets in that range

ENI — Advanced Patterns

Pattern How Use Case
Dual-homed instance ENI in public subnet + ENI in private subnet Management interface separate from data interface
Failover Detach ENI from failed instance → attach to standby IP address moves with ENI (no DNS/client changes)
Multiple SGs Different security groups per ENI on same instance Different access rules for management vs application traffic
Secondary IPs Multiple private IPs on one ENI Hosting multiple SSL sites, container networking

Managed Prefix Lists

  • What: Named set of CIDR blocks that can be referenced in security groups and route tables
  • AWS-managed: com.amazonaws.region.s3 (S3 IPs), com.amazonaws.global.cloudfront.origin-facing (CloudFront IPs)
  • Customer-managed: Create your own (e.g., corporate office IPs, partner networks)
  • Sharing: Share via RAM across accounts — update once, all accounts get updated rules
  • SG reference: Instead of listing 50 IPs in a SG rule, reference a prefix list (single entry)
  • Route table: Use prefix list as destination in route table entries

IPv6 in AWS

Feature IPv4 IPv6
Address type Private (RFC 1918) + optional public (EIP) All globally unique (public), no NAT needed
Outbound-only NAT Gateway Egress-Only Internet Gateway (outbound, blocks inbound)
Dual-stack Subnets have both IPv4 + IPv6 CIDRs
IPv4-only services NAT64 + DNS64 allows IPv6 instances to reach IPv4 endpoints
  • Egress-Only IGW: Like NAT for IPv6 — allows outbound connections, blocks unsolicited inbound. Free.
  • NAT64: Translates IPv6 packets to IPv4 for reaching IPv4-only destinations. Configured on NAT Gateway.
  • DNS64: Route 53 Resolver synthesizes AAAA records for IPv4-only domains (so IPv6 clients can reach them).

BYOIP (Bring Your Own IP)

  • What: Use your own public IPv4/IPv6 address ranges in AWS (instead of AWS-provided)
  • Why: IP reputation, allowlisting by partners, regulatory requirements
  • Requirements: /24 minimum (IPv4), must own the range (RIR registration), create ROA (Route Origin Authorization)
  • Use with: EIPs, EC2, NLB, Global Accelerator

Exam Tips

Exam Key Points
ANS-C01 “VPC out of IPs” → secondary CIDR. “Move IP on failover” → ENI detach/attach. “Restrict SG to CloudFront only” → AWS-managed prefix list. “IPv6 outbound only” → Egress-Only IGW. “IPv6 instance reach IPv4 service” → NAT64 + DNS64. “Share IP list across accounts” → Prefix list + RAM. “Use own public IPs” → BYOIP.

AWS Certification Exam Practice Questions

Question 1:

A VPC with CIDR 10.0.0.0/16 has exhausted all available IP addresses. The company needs more IPs without disrupting existing workloads or recreating the VPC. What should they do?

  1. Delete unused ENIs to free up IPs
  2. Add a secondary CIDR block (e.g., 10.1.0.0/16) to the VPC and create new subnets in that range
  3. Resize the VPC CIDR from /16 to /8
  4. Create a new VPC and peer it to the existing one
Show Answer

Answer: B — You can add up to 4 secondary CIDR blocks to an existing VPC. Create new subnets in the secondary range. Existing workloads continue using original CIDRs — no disruption. VPC CIDRs cannot be resized (C is wrong). New VPC with peering (D) works but adds complexity and doesn’t extend the existing network seamlessly.

Question 2:

A company needs their ALB security group to only allow traffic from CloudFront. CloudFront has hundreds of IP ranges that change frequently. How should this be maintained without manual updates?

  1. Create a Lambda function that updates the SG every hour from the AWS IP ranges JSON file
  2. Reference the AWS-managed prefix list for CloudFront (com.amazonaws.global.cloudfront.origin-facing) in the security group rule
  3. Use WAF on the ALB to validate CloudFront headers instead of IP restriction
  4. Open the SG to 0.0.0.0/0 and rely on CloudFront custom headers for security
Show Answer

Answer: B — AWS-managed prefix lists are automatically updated by AWS when IP ranges change. Reference it in your SG rule — one entry covers all CloudFront IPs, always current. Lambda (A) works but is unnecessary complexity when the managed prefix list exists. Custom headers (C, D) are defense-in-depth but should complement IP restriction, not replace it.

Question 3:

A company is transitioning to IPv6. Their instances need to initiate outbound connections to the internet (software updates) but must NOT be reachable from the internet on IPv6. No NAT is desired. Which gateway provides this for IPv6?

  1. NAT Gateway configured for IPv6
  2. Egress-Only Internet Gateway — allows outbound IPv6 traffic, blocks unsolicited inbound
  3. Internet Gateway with restrictive security groups
  4. VPN Gateway with IPv6 routing
Show Answer

Answer: B — Egress-Only Internet Gateway is the IPv6 equivalent of NAT Gateway (for outbound-only). It allows instances to initiate connections to the internet but blocks unsolicited inbound connections. No address translation (all IPv6 is public). Free to use. Regular Internet Gateway (C) allows both directions — SGs help but the gateway itself doesn’t block inbound at the network level.

Related Posts

References

Frequently Asked Questions

When should I use secondary CIDRs vs a new VPC?

Secondary CIDR: When you need more IPs in the same logical network (same route tables, same security groups apply). Simpler. New VPC: When you need network isolation (different security domain, different team ownership). Secondary CIDRs keep everything in one VPC. New VPC + peering/TGW creates separate networks that must be explicitly connected.

What are managed prefix lists useful for?

Prefix lists simplify rule management. Instead of maintaining 50+ IP rules in multiple security groups across accounts, create one prefix list → reference it everywhere. When IPs change, update the prefix list once — all referencing SGs update automatically. AWS-managed lists (CloudFront, S3) are maintained by AWS automatically. Share your own lists via RAM across accounts.

AWS Direct Connect Deep Dive – DX Gateway, LAG & Resiliency Patterns

AWS Direct Connect Deep Dive — Overview

Direct Connect is the second-largest ANS topic (150 questions). The exam goes far deeper than basic DX setup — testing DX Gateway, hosted vs dedicated connections, LAG (Link Aggregation Groups), resiliency models, MACsec encryption, and failover patterns with VPN backup.

Direct Connect Architecture
On-Premises
Router/Firewall
BGP peering
—DX—
DX Location
AWS cage + customer cage
Cross-connect (fiber)
Virtual Interface (VIF)
Private VIF → VPC
Public VIF → AWS services
Transit VIF → TGW
DX Gateway
Connect to multiple VPCs
Cross-region access
Works with TGW or VGW

Connection Types

Type Speed Who Owns Lead Time
Dedicated Connection 1 Gbps, 10 Gbps, 100 Gbps You request from AWS, physical port allocated Weeks to months (physical install)
Hosted Connection 50 Mbps to 10 Gbps DX Partner provisions on their connection Days (partner has existing infrastructure)

Exam note: Hosted connections are single VIF (one VLAN). Dedicated connections support multiple VIFs (up to 50 private/transit + 1 public).

Virtual Interfaces (VIFs)

VIF Type Connects To Use Case
Private VIF VGW (single VPC) or DX Gateway (multiple VPCs) Access private VPC resources (EC2, RDS, etc.)
Public VIF AWS public services (S3, DynamoDB, STS) Access AWS APIs over DX instead of internet
Transit VIF Transit Gateway (via DX Gateway) Access multiple VPCs via TGW (scalable)

DX Gateway

  • What: Global resource that connects your DX connection to VPCs in ANY region (not just the DX location’s region)
  • Associations: DX Gateway → up to 20 VGWs (private VIF) OR 3 TGWs (transit VIF)
  • Cross-region: Single DX in us-east-1 DX location → DX Gateway → VPCs in eu-west-1, ap-southeast-1, etc.
  • Cannot: Route traffic between associated VPCs through DX Gateway (not transitive)
  • Exam trap: DX Gateway + VGW associations don’t allow VPC-to-VPC routing. For that, use TGW.

LAG (Link Aggregation Group)

  • What: Bundle multiple DX connections into single logical connection (LACP 802.3ad)
  • Requirements: All connections must be same speed, same DX location, same AWS device
  • Max: 4 connections per LAG (e.g., 4 × 10 Gbps = 40 Gbps logical)
  • Minimum links: Configure minimum active links (e.g., LAG fails if <2 of 4 links active)
  • Use case: Higher aggregate bandwidth + link-level redundancy (if one fiber fails, LAG continues)
  • NOT resiliency: All links go to same device/location. If location fails, entire LAG fails. For resiliency, use separate locations.

Resiliency Models (AWS Recommended)

Model Architecture Protects Against
Maximum Resiliency 2 DX connections at 2 separate DX locations, each with separate devices Device failure, connection failure, location failure
High Resiliency 2 DX connections at 1 DX location (separate devices) Device failure, connection failure (NOT location failure)
DX + VPN Backup Primary: DX connection. Backup: Site-to-Site VPN over internet DX failure (VPN takes over, lower bandwidth, higher latency)

Failover Patterns

  • DX + VPN backup: BGP configures DX as preferred (shorter AS path or higher local preference). VPN advertises same routes with longer AS path. Failover is automatic via BGP.
  • Active/Active DX: Both connections active, traffic load-balanced. If one fails, other carries all traffic.
  • BFD (Bidirectional Forwarding Detection): Faster failure detection (sub-second) compared to BGP hold timer (90s default). Enable for faster failover.
  • MACsec: Layer 2 encryption on 10 Gbps and 100 Gbps dedicated connections. Encrypts traffic between your router and AWS device at DX location.

Exam Tips

Exam Key Points
ANS-C01 “Access VPCs in multiple regions from one DX” → DX Gateway. “Higher bandwidth than single 10G” → LAG (up to 4×10G). “Survive DX location failure” → Maximum Resiliency (2 locations). “Fast provisioning” → Hosted connection (days). “DX + backup” → Site-to-Site VPN with longer BGP AS path. “Encrypt DX traffic” → MACsec (L2) for 10/100G. “Access S3 over DX” → Public VIF. “Connect to TGW via DX” → Transit VIF + DX Gateway.

AWS Certification Exam Practice Questions

Question 1:

A company has a Direct Connect connection in us-east-1 DX location. They need to access VPCs in us-east-1, eu-west-1, and ap-southeast-1 from their on-premises data center using this single DX connection. Which configuration enables this?

  1. Create private VIFs for each VPC directly on the DX connection
  2. Create a DX Gateway, associate it with VGWs in each region’s VPC, create a private VIF to the DX Gateway
  3. Create a Transit Gateway in us-east-1 and peer it to the other regions
  4. Create separate DX connections in each region
Show Answer

Answer: B — DX Gateway is a global resource that allows a single DX connection to reach VPCs in any region. Associate VGWs from VPCs in multiple regions with the DX Gateway. Create one private VIF from your DX connection to the DX Gateway. Traffic routes to the correct region automatically. Private VIFs directly (A) only work for same-region VPCs. Separate DX per region (D) is expensive and unnecessary.

Question 2:

A company requires their Direct Connect to survive a complete DX location failure. Their compliance requires maintaining connectivity even if an entire facility goes offline. Which resiliency architecture meets this requirement?

  1. Two DX connections in the same DX location on separate AWS devices
  2. LAG with 4 connections at one DX location
  3. Two DX connections at two geographically separate DX locations with separate devices
  4. Single DX connection with Site-to-Site VPN backup
Show Answer

Answer: C — Maximum Resiliency requires connections at 2 separate physical locations. If one entire location fails (fire, power outage, fiber cut), the other location maintains connectivity. Same-location redundancy (A, B) doesn’t survive location failure. VPN backup (D) provides connectivity but at much lower bandwidth and higher latency — may not meet SLA requirements.

Question 3:

A company needs to access both private VPC resources AND public AWS services (S3 API) over their Direct Connect connection. How should this be configured?

  1. Single private VIF — all AWS traffic routes over it
  2. Create both a private VIF (for VPC resources) and a public VIF (for AWS public services) on the same DX connection
  3. Use a transit VIF which provides access to both private and public services
  4. Route all traffic through a NAT Gateway in the VPC
Show Answer

Answer: B — Private VIF routes to VPC private IP space. Public VIF routes to AWS public IP ranges (S3, DynamoDB, STS, etc.) over DX instead of internet. Both can coexist on the same DX connection as separate VLANs. A dedicated connection supports up to 50 private/transit VIFs + 1 public VIF. Transit VIF (C) connects to TGW for VPC access but doesn’t provide public service access.

Question 4:

A company has a 10 Gbps Direct Connect but needs 30 Gbps aggregate bandwidth for a data migration. They want a single logical connection for simplified management. All connections must be at the same DX location. Which solution provides this?

  1. Upgrade to a 100 Gbps dedicated connection
  2. Create a LAG with 3 × 10 Gbps connections (must be same speed, same location, same AWS device)
  3. Create 3 separate DX connections and use ECMP
  4. Use multiple VIFs on a single 10 Gbps connection
Show Answer

Answer: B — LAG bundles multiple physical connections into one logical connection (LACP). Requirements: same speed, same location, same AWS device. 3 × 10 Gbps = 30 Gbps aggregate. Managed as single connection (one set of VIFs). 100 Gbps (A) works but is more expensive. Multiple VIFs on one connection (D) don’t increase bandwidth beyond 10 Gbps. Note: LAG max is 4 connections.

Question 5:

A company uses DX as primary connectivity. They need a backup that automatically activates if DX fails, accepting lower performance during failover. The backup must require no manual intervention. Which design provides this?

  1. Second DX at another location (active/passive)
  2. Site-to-Site VPN as backup — configure BGP to prefer DX (shorter AS path) and VPN as fallback (longer AS path). BGP automatically fails over.
  3. AWS Cloud WAN with automatic failover
  4. Internet-based IPSec tunnel manually activated during outage
Show Answer

Answer: B — DX + VPN backup with BGP: DX advertises routes with shorter AS path (preferred). VPN advertises same routes with longer AS path. When DX fails, BGP removes DX routes → VPN routes become active automatically. No manual intervention. Enable BFD on DX for faster detection (sub-second vs 90s BGP hold timer). VPN bandwidth is lower but maintains connectivity. Second DX (A) works but costs significantly more than VPN.

Related Posts

References

Frequently Asked Questions

DX Gateway vs Transit Gateway — when to use which with DX?

DX Gateway + Private VIFs: Connect DX to multiple VPCs (up to 20 VGWs) across regions. Simple, no transitive routing. DX Gateway + Transit VIF + TGW: Connect DX to TGW, which connects to VPCs. Provides transitive routing (VPC-to-VPC through TGW), centralized management, and scales to thousands of VPCs. Use TGW when you need inter-VPC routing or 20+ VPCs.

What is the difference between dedicated and hosted connections?

Dedicated: You request a physical port directly from AWS (1/10/100 Gbps). You own the connection. Supports multiple VIFs (up to 50). Takes weeks-months to provision. Hosted: A DX Partner provisions a connection on their existing infrastructure and gives you access. Single VIF only. 50 Mbps to 10 Gbps. Provisions in days. Choose hosted for: faster setup, sub-1Gbps needs, or when you don’t have presence at a DX location.

AWS Network Performance – ENA, EFA, Jumbo Frames & Placement Groups

AWS Network Performance Architecture — Overview

Network performance is the #1 topic on ANS-C01/C02 (160+ questions) and appears on SAP-C02 for HPC and high-throughput scenarios. The exam tests Enhanced Networking (ENA/EFA), placement groups, jumbo frames, bandwidth allocation, and network optimization patterns.

Network Performance Stack
Network Interface
ENA: Up to 200 Gbps
EFA: HPC/ML inter-node
SR-IOV: Bypass hypervisor
Intel 82599 VF (legacy)
Placement Groups
Cluster: Low latency (same rack)
Spread: HA (distinct hardware)
Partition: Large distributed (HDFS, Kafka)
Frame Size
Standard: 1500 MTU
Jumbo: 9001 MTU
Within VPC/peering/DX
NOT over internet/VPN
Bandwidth
Instance type determines max
Baseline vs burst
Multi-flow: aggregate BW
Single-flow: 5-10 Gbps cap

Enhanced Networking — ENA vs EFA

Feature ENA (Elastic Network Adapter) EFA (Elastic Fabric Adapter)
Purpose High-performance general networking HPC inter-node communication (MPI, NCCL)
Speed Up to 200 Gbps Up to 200 Gbps + OS-bypass
Latency Low (microseconds) Ultra-low (bypasses OS kernel for node-to-node)
Protocol TCP/UDP Libfabric (MPI, NCCL) + TCP/UDP
Use case All workloads (enabled by default on Nitro) Tightly-coupled HPC (weather modeling, CFD), distributed ML training
Placement Any Cluster placement group required for best performance

Placement Groups

Type How Use Case Limitation
Cluster Instances on same rack, single AZ HPC, low-latency node-to-node (10 Gbps between instances) Single AZ = single point of failure. Capacity errors if rack full.
Spread Each instance on distinct hardware (max 7 per AZ) Critical instances that must not fail together (HA for small groups) Max 7 instances per AZ per group.
Partition Instances grouped into partitions on separate racks (up to 7 partitions/AZ) Large distributed systems (HDFS, HBase, Kafka, Cassandra) Topology-aware apps needed to leverage partition isolation.

Jumbo Frames (MTU 9001)

  • What: Larger Ethernet frames (9001 bytes vs standard 1500 bytes). Less overhead per byte transferred → higher throughput.
  • Supported: Within VPC, across VPC peering (same region), over Direct Connect, between instances in same placement group
  • NOT supported: Over the internet, over VPN, over TGW (TGW reduces to 8500 MTU), across inter-region peering
  • Path MTU Discovery: ICMP “Fragmentation Needed” messages. If blocked by NACL/SG, causes packet drops (exam trap!)
  • Configuration: Set on OS level (Linux: ip link set dev eth0 mtu 9001). Both sender AND receiver must support.

Instance Bandwidth

  • Determined by instance type: t3.micro = 5 Gbps burst, c5.18xlarge = 25 Gbps, c5n.18xlarge = 100 Gbps
  • Single-flow limit: Single TCP connection maxes at ~5-10 Gbps (use multi-flow/parallel connections for full bandwidth)
  • Baseline vs burst: Smaller instances have baseline + burst (like EBS bandwidth). Larger instances have consistent bandwidth.
  • “n” instances: c5n, m5n, r5n = network-optimized variants with higher bandwidth (100 Gbps)
  • Cross-AZ: ~$0.01/GB data transfer charge. Same performance as intra-AZ but costs money.

Network Optimization Patterns

Problem Solution
HPC needs ultra-low latency between nodes EFA + Cluster placement group + jumbo frames
Single TCP flow not reaching instance bandwidth Use multiple parallel TCP connections (multi-flow aggregates)
Packet drops on large transfers Check MTU mismatch (Path MTU Discovery), enable jumbo frames end-to-end
Distributed DB needs fault isolation Partition placement group (HDFS, Kafka rack-aware)
Critical services can’t share hardware failure Spread placement group (max 7/AZ)

Exam Tips

Exam Key Points
ANS-C01 “Lowest latency between instances” → Cluster placement group + ENA. “HPC MPI workload” → EFA + Cluster placement. “Jumbo frames not working” → check TGW (8500), VPN (1500), internet (1500). “HDFS rack awareness” → Partition placement. “Max network bandwidth” → check instance type (“n” variant). “Single flow limited” → use multiple parallel connections. “PMTUD blocked” → check NACL allows ICMP type 3 code 4.

AWS Certification Exam Practice Questions

Question 1:

A company runs an HPC simulation that requires ultra-low latency communication between 100 compute instances using MPI. The instances need to communicate at the highest possible bandwidth with OS-bypass for inter-node traffic. Which configuration provides this?

  1. ENA-enabled instances in a spread placement group with jumbo frames
  2. EFA-enabled instances in a cluster placement group with jumbo frames (MTU 9001)
  3. ENA-enabled instances with enhanced networking across 3 AZs
  4. Standard networking with NLB distributing traffic between instances
Show Answer

Answer: B — EFA provides OS-bypass for MPI/NCCL (libfabric), eliminating kernel overhead for inter-node communication. Cluster placement group places all instances on the same rack for lowest latency and highest bandwidth (up to 200 Gbps between instances). Jumbo frames reduce per-packet overhead. ENA (A) doesn’t provide OS-bypass for MPI. Spread (A) separates instances — opposite of what HPC needs.

Question 2:

Instances in a VPC communicate with jumbo frames (MTU 9001). After adding a Transit Gateway to route traffic between VPCs, large packet transfers start failing with fragmentation. What is the cause and fix?

  1. TGW doesn’t support jumbo frames — reduce MTU to 1500 on all instances
  2. TGW supports 8500 byte MTU (not 9001). Enable Path MTU Discovery and ensure NACLs allow ICMP type 3 code 4
  3. Enable jumbo frame support on the TGW attachment
  4. Increase the TGW bandwidth allocation
Show Answer

Answer: B — Transit Gateway supports a maximum MTU of 8500 bytes (not 9001). When instances send 9001-byte frames through TGW, packets need fragmentation. Path MTU Discovery (PMTUD) sends ICMP “Fragmentation Needed” messages back to the sender to reduce MTU. If NACLs block ICMP type 3 code 4, the sender never learns to reduce size → packets silently drop. Fix: allow ICMP in NACLs, or set instance MTU to 8500 for TGW paths.

Question 3:

A company’s c5.xlarge instances are achieving only 5 Gbps throughput when transferring large files between instances, despite the instance type supporting 10 Gbps. The transfer uses a single TCP connection. How can they achieve full 10 Gbps?

  1. Enable enhanced networking (ENA) on the instances
  2. Use multiple parallel TCP connections (multi-flow) to aggregate bandwidth beyond the single-flow limit
  3. Move instances to a cluster placement group
  4. Increase instance type to c5.2xlarge
Show Answer

Answer: B — Single TCP flow (5-tuple: src IP, dst IP, src port, dst port, protocol) is limited to ~5 Gbps on AWS. To reach the instance’s full bandwidth (10 Gbps), use multiple parallel connections (different source ports = different flows). Tools like iperf3 with -P flag or multi-threaded file transfer. ENA (A) is already enabled on c5 by default. Placement group (C) helps latency, not single-flow bandwidth.

Question 4:

A distributed Kafka cluster on EC2 needs rack-awareness for partition replicas (replicas should be on different physical racks for fault tolerance). The cluster has 30 brokers across 3 AZs. Which placement strategy provides rack isolation information to Kafka?

  1. Cluster placement group (all on same rack)
  2. Spread placement group (max 7 per AZ — insufficient for 10 per AZ)
  3. Partition placement group — Kafka maps partitions to placement partitions for rack-awareness
  4. No placement group — use AZ as rack identifier
Show Answer

Answer: C — Partition placement groups provide up to 7 partitions per AZ. Each partition runs on a separate rack. Kafka (or HDFS) uses the partition number as the “rack” identifier for replica placement. This ensures replicas are on physically separate racks. Spread (B) maxes at 7 instances per AZ total — not enough for 10 brokers/AZ. Cluster (A) puts everything on ONE rack (worst for HA).

Question 5:

A company’s application transfers data between instances in the same VPC and also to instances in a peered VPC in another region. They want to use jumbo frames (MTU 9001) for maximum throughput. Where will jumbo frames work?

  1. Both intra-VPC and inter-region peering support MTU 9001
  2. Only intra-VPC supports MTU 9001. Inter-region peering is limited to MTU 1500.
  3. Neither supports jumbo frames without special configuration
  4. Both work but require enabling jumbo frames on the peering connection
Show Answer

Answer: B — Jumbo frames (9001 MTU) work within a VPC and across same-region VPC peering. Inter-region VPC peering reduces MTU to 1500 bytes. Traffic over the internet, VPN, or inter-region peering does NOT support jumbo frames. If you set 9001 MTU and send cross-region, packets will be fragmented (or dropped if DF bit set without PMTUD). Always verify MTU support per path.

Related Posts

References

Frequently Asked Questions

ENA vs EFA — when do I need EFA?

ENA is sufficient for 99% of workloads (web, databases, containers, general compute). It provides up to 200 Gbps with low latency. EFA adds OS-bypass (kernel bypass) specifically for tightly-coupled parallel computing: MPI-based HPC simulations, distributed ML training (NCCL). If your workload doesn’t use MPI or NCCL, you don’t need EFA. EFA instances also have ENA for standard traffic.

Why does jumbo frame performance matter?

Standard frames carry 1500 bytes per packet with ~40 bytes header overhead (2.6%). Jumbo frames carry 9001 bytes with the same ~40 bytes overhead (0.4%). For large data transfers, jumbo frames: reduce CPU interrupt rate (fewer packets per GB), reduce header overhead (more payload per packet), and increase throughput. Most impactful for bulk data transfer, database replication, and storage traffic.

AWS Glue ETL & Data Pipeline Architecture – Catalog, Crawlers & Data Quality

AWS Glue ETL & Data Pipeline Architecture — Overview

AWS Glue is the core ETL service tested on DEA-C01 (34 questions) and MLA-C01 (36 questions). The exam tests Glue jobs (Spark/Python), Glue Data Catalog (metadata), crawlers (schema discovery), data quality, and orchestration patterns for data engineering pipelines.

Data Pipeline Architecture with Glue
Sources
S3, RDS, DynamoDB
JDBC, Kinesis
On-premises DBs
Crawlers
Auto-discover schema
Populate Data Catalog
Classify formats
Glue ETL Jobs
Spark/Python/Ray
Transform, clean
DQ checks
Targets
S3 (Parquet/Iceberg)
Redshift, RDS
OpenSearch
Consume
Athena queries
Redshift analytics
QuickSight dashboards
Orchestration: Glue Workflows | Step Functions | MWAA (Airflow) | EventBridge schedules

Glue Components

Component What Key Details
Data Catalog Central metadata repository (databases, tables, schemas) Shared across Athena, Redshift Spectrum, EMR, Lake Formation. Hive-compatible metastore.
Crawlers Auto-discover data schema and populate Catalog Runs on schedule or on-demand. Classifiers detect format (Parquet, CSV, JSON, Avro).
ETL Jobs (Spark) Distributed data transformation using Apache Spark Glue 4.0 (Spark 3.3). DPU-based pricing. Auto-scaling. Bookmarks for incremental processing.
ETL Jobs (Python Shell) Lightweight Python scripts (no Spark overhead) Cheaper for simple transforms. Max 1 DPU. Good for API calls, small files.
Data Quality Define and monitor data quality rules DQDL rules (completeness, uniqueness, freshness). Alert or stop pipeline on failure.
Glue Studio Visual ETL job designer (no-code/low-code) Drag-and-drop transforms. Generates Spark code. Good for data engineers.
Glue DataBrew Visual data preparation (profiling, cleaning) 250+ transforms without code. Profile datasets for quality. Recipes for reuse.

Job Bookmarks — Incremental Processing

  • Problem: Daily ETL job should only process NEW data (not re-process everything)
  • Solution: Job bookmarks track what was already processed (S3 paths, JDBC timestamps)
  • How: Enable bookmark on job. Glue remembers last processed file/row. Next run starts from where it left off.
  • Exam note: “Process only new files in S3” → Enable Glue job bookmarks

Glue Connections & VPC

  • JDBC sources: Glue jobs connect to RDS/Redshift via JDBC connections (require VPC, subnet, SG configuration)
  • ENI: Glue creates ENIs in your VPC subnet to access private data sources
  • NAT Gateway: If Glue job needs internet access (e.g., API call) from private subnet, requires NAT Gateway
  • S3 endpoint: Always add VPC Gateway Endpoint for S3 (Glue reads/writes S3 heavily — avoid NAT charges)

Orchestration Patterns

Orchestrator Best For Key Feature
Glue Workflows Simple Glue-only pipelines (crawler → job → crawler) Built-in, triggers on schedule/event/condition
Step Functions Complex multi-service pipelines with error handling Visual workflow, retry/catch, parallel execution, human approval
MWAA (Airflow) Teams already using Airflow, complex DAG dependencies Managed Apache Airflow, Python DAGs, rich ecosystem
EventBridge Scheduler Simple scheduled triggers (cron/rate) Serverless, triggers Glue/Step Functions/Lambda on schedule

Performance Optimization

  • Partitioning: Partition output by date/region/category. Athena/Redshift Spectrum prunes partitions for faster queries.
  • File format: Convert CSV/JSON → Parquet/ORC (columnar, compressed, 10-100x faster queries)
  • File size: Avoid small files (coalesce to 128MB-1GB per file). Small files kill query performance.
  • Auto-scaling: Glue 4.0 auto-scales DPUs based on workload (no over-provisioning)
  • Pushdown predicates: Filter data at source (pushdown to JDBC, S3 Select) — less data to process

Exam Tips

Exam Key Points
DEA-C01 “Discover schema” → Crawler. “Process only new files” → Job Bookmarks. “Convert CSV to Parquet” → Glue ETL job. “Central metadata” → Data Catalog. “Visual ETL” → Glue Studio. “Data quality rules” → Glue Data Quality (DQDL). “Orchestrate pipeline” → Step Functions (complex) or Glue Workflows (simple). “Access RDS from Glue” → JDBC connection in VPC.

AWS Certification Exam Practice Questions

Question 1:

A company receives daily CSV files in S3. They need to transform them to Parquet format, partition by date, and make them queryable by Athena. The pipeline should only process new files each day. Which solution achieves this?

  1. Lambda triggered by S3 event → convert each file individually
  2. Glue ETL job with bookmarks enabled, scheduled daily. Reads new CSVs, transforms to Parquet, writes partitioned output. Crawler updates Catalog.
  3. Athena CTAS query to convert CSV to Parquet
  4. EMR Spark cluster running 24/7 processing files as they arrive
Show Answer

Answer: B — Glue ETL job handles the transformation (CSV → Parquet with partitioning). Job bookmarks ensure only new files are processed each day (no reprocessing). Crawler updates the Data Catalog so Athena sees new partitions. Serverless — no infrastructure to manage. Lambda (A) works for small files but doesn’t handle large-scale transformations well. EMR (D) is over-provisioned for a daily batch job.

Question 2:

A data team needs a central metadata repository that Athena, Redshift Spectrum, and EMR can all query against. They want schema discovered automatically from S3 data. Which combination provides this?

  1. Custom Hive metastore on EC2 with manual schema creation
  2. Glue Data Catalog populated by Glue Crawlers — automatically discovered schemas accessible by Athena, Redshift Spectrum, and EMR
  3. S3 bucket tags as metadata with Lambda reading them
  4. DynamoDB table storing schema information
Show Answer

Answer: B — Glue Data Catalog is the central metastore that integrates natively with Athena (uses it by default), Redshift Spectrum (external schema), and EMR (as Hive-compatible metastore). Crawlers auto-discover schemas from S3 data. No manual schema management needed. This is the AWS-native data catalog solution — replaces external Hive metastore.

Question 3:

A Glue ETL job needs to read from an RDS PostgreSQL database in a private subnet. The job also writes output to S3. The job fails with connection timeout errors to both RDS and S3. What is the most likely configuration issue?

  1. Glue job doesn’t have IAM permissions to access RDS
  2. Glue connection not configured with VPC, subnet, and security group. Also missing S3 VPC Gateway Endpoint.
  3. RDS instance is encrypted and Glue can’t decrypt
  4. Glue job DPU count is too low for the data volume
Show Answer

Answer: B — Glue jobs accessing private resources need: (1) Connection with VPC, subnet, and SG configured (creates ENI in your VPC). (2) SG must allow Glue ENI → RDS on port 5432. (3) S3 VPC Gateway Endpoint needed because Glue in a private subnet can’t reach S3 over the internet. This is the #1 Glue troubleshooting question on the exam. Without VPC config, Glue can’t reach private resources.

Question 4:

A data pipeline produces Parquet files in S3 that are later queried by Athena. Queries are slow because the pipeline creates thousands of small files (1-5 MB each). How should the pipeline be optimized?

  1. Switch from Parquet to CSV for faster reading
  2. Configure the Glue job to coalesce output files to 128MB-1GB using repartition/coalesce, and partition output by date for pruning
  3. Increase Athena query timeout to allow more time
  4. Add more Glue DPUs to write faster
Show Answer

Answer: B — Small files are the #1 performance killer for S3-based analytics. Each file requires a separate S3 GET request and has metadata overhead. Coalescing to 128MB-1GB optimal size + date partitioning allows Athena to: (1) read fewer, larger files (less I/O overhead), and (2) skip irrelevant partitions. CSV (A) is worse than Parquet. More DPUs (D) don’t fix the output file size issue.

Question 5:

A data team needs to validate that incoming data meets quality rules (no null values in required fields, dates in correct format, values within expected ranges) before loading into the data warehouse. If rules fail, the pipeline should stop and alert the team. Which Glue feature provides this?

  1. Glue Crawler with classification rules
  2. Glue Data Quality with DQDL rules configured to halt the job on failure and send SNS notification
  3. Custom Lambda function that samples and validates data before Glue runs
  4. Athena query that checks constraints after loading
Show Answer

Answer: B — Glue Data Quality uses DQDL (Data Quality Definition Language) to define rules: Completeness, Uniqueness, Freshness, ColumnValues ranges. Rules run within the ETL job. On failure: stop pipeline (prevent bad data loading) + trigger CloudWatch alarm → SNS notification. Built-in, no custom code needed. Athena (D) is post-load (too late). Lambda (C) requires custom implementation.

Related Posts

References

Frequently Asked Questions

Glue ETL vs EMR — when to use which?

Glue: Serverless, auto-scaling, pay per DPU-hour. Best for scheduled ETL jobs, data catalog integration, and teams wanting managed Spark without cluster management. EMR: Full Hadoop/Spark ecosystem, persistent or transient clusters, custom configurations (Hive, Presto, HBase). Best for complex big data workloads needing fine-tuned Spark configs, long-running clusters, or non-Spark frameworks.

What is the Glue Data Catalog used for?

Central metadata repository storing table definitions (schema, location, format, partitions). Shared across AWS analytics services: Athena queries tables from the Catalog, Redshift Spectrum references external tables, EMR uses it as Hive metastore, Lake Formation controls access to Catalog tables. It’s the “phone book” for your data lake — services look up where data lives and what format it’s in.