AWS Serverless API Architecture — Overview

The serverless API pattern is the most common modern application architecture on AWS — combining API Gateway for routing, Lambda for compute, and DynamoDB or Aurora+RDS Proxy for data. It provides zero server management, automatic scaling, and pay-per-request pricing. This architecture appears in 20%+ of SAP-C02 questions and is foundational for SAA-C03.

API Gateway — REST vs HTTP API vs WebSocket

Exam tip: Default to HTTP API unless you specifically need REST API features (caching, usage plans, API keys, request/response transformation). HTTP API is 70% cheaper with lower latency.

Lambda — Patterns & Pitfalls

Connection Pooling Problem (Lambda + RDS)

The #1 tested Lambda anti-pattern: Lambda scales to hundreds of concurrent executions, each opening a database connection → overwhelms RDS connection limit → 5xx errors.

• Solution: RDS Proxy — Sits between Lambda and RDS. Maintains a connection pool. Hundreds of Lambda instances share a few dozen actual DB connections. Handles connection multiplexing transparently.
• Alternative: DynamoDB — HTTP-based connections, no connection limit. Truly serverless-native.

Cold Starts

• Problem: First invocation after idle period requires container initialization (100ms-10s depending on runtime/dependencies)
• Mitigation:
  • Provisioned Concurrency — Pre-warm N instances (eliminates cold starts, costs money)
  • SnapStart (Java only) — Snapshot after init, restore from snapshot on cold start (~10x faster)
  • Keep functions warm — CloudWatch scheduled rule pings function every 5 min (hacky, not recommended)
  • Smaller packages — Reduce deployment package size and dependencies
  • ARM (Graviton) — 20% cheaper + 10% faster startup on arm64

Lambda Best Practices

• Initialize outside handler — DB connections, SDK clients, config loaded once and reused across invocations
• Environment variables — Store config (not secrets) in env vars. Secrets → Secrets Manager with caching.
• Use Powertools — AWS Lambda Powertools provides structured logging, tracing, metrics, idempotency out of the box
• Right-size memory — More memory = more CPU. Use AWS Lambda Power Tuning to find optimal setting.

Database Layer — DynamoDB vs Aurora + RDS Proxy

Authentication & Authorization

Scaling & Limits

Handling the 29-Second Timeout

API Gateway has a hard 29-second timeout. For longer operations:

• Async pattern: API → SQS → Lambda (no timeout). Return 202 + task ID immediately. Client polls a status endpoint.
• WebSocket: Client opens WebSocket connection → receives result when ready (push-based).
• Step Functions: API → Step Functions (async) → callback when complete.

Cost Optimization

• HTTP API over REST API: 70% cheaper ($1/M vs $3.50/M). Use unless you need REST-specific features.
• Lambda ARM (Graviton): 20% cheaper, 10% faster. Switch with one config change.
• API Gateway caching: Reduces Lambda invocations for repeated requests (REST API only). $0.02-$3.80/hr depending on cache size.
• DynamoDB on-demand: Best for unpredictable traffic. Switch to provisioned + auto-scaling for predictable patterns to save 50%+.
• Lambda right-sizing: Use Power Tuning to find optimal memory. Over-provisioned memory wastes money; under-provisioned increases duration (= higher cost).
• CloudFront caching: Cache static responses and semi-dynamic content at edge. Each cache hit avoids API GW + Lambda invoke.
• Reserved Concurrency: Free (limits max scale). Provisioned Concurrency costs money (use only where cold starts matter).

Exam Tips

AWS Certification Exam Practice Questions

Question 1:

A serverless application uses Lambda functions and an Amazon RDS PostgreSQL database. Under peak load, the application returns intermittent 5xx errors. CloudWatch shows Lambda concurrency reaching 500 while RDS shows max connections exhausted. What is the MOST operationally efficient fix?

1. Increase the RDS instance size to allow more connections
2. Add Amazon RDS Proxy between Lambda and the database
3. Reduce Lambda reserved concurrency to 50
4. Replace RDS with DynamoDB

Question 2:

A company’s API Gateway REST API costs $10,000/month due to high request volume. Most requests (80%) hit the same 5 endpoints with identical responses for any given 5-minute window. The team wants to reduce costs without changing the application. What should they do?

1. Enable API Gateway caching with a 300-second TTL
2. Switch from REST API to HTTP API
3. Add CloudFront in front of API Gateway with caching
4. Move to Application Load Balancer with Lambda targets

Question 3:

A serverless API has a Lambda function that takes 45 seconds to process complex report generation requests. Clients receive timeout errors. The API must return results to the client. Which architecture solves this?

1. Increase API Gateway timeout to 60 seconds
2. Use API Gateway → SQS → Lambda (async). Return 202 with task ID. Client polls a separate status endpoint for results.
3. Use Lambda Provisioned Concurrency to speed up processing
4. Switch to Application Load Balancer (120s timeout) with Lambda target

Question 4:

A mobile app needs user authentication with social login (Google, Facebook), MFA, and email verification. After authentication, users access an API Gateway API. Which is the SIMPLEST authentication setup?

1. Lambda Authorizer that validates OAuth tokens from social providers
2. Amazon Cognito User Pool with social identity providers, configured as API Gateway authorizer
3. IAM authentication with Cognito Identity Pool for temporary credentials
4. Custom authentication microservice on ECS behind the same API Gateway

Question 5:

A DynamoDB table serves a serverless API. Traffic is highly spiky — near zero at night, 50,000 WCU during flash sales (lasting 30 minutes, 2-3 times per month). The table currently uses provisioned capacity with auto-scaling but users experience throttling during the first minutes of each flash sale. What change eliminates the throttling?

1. Pre-scale the table manually before each flash sale
2. Switch to DynamoDB on-demand capacity mode
3. Increase the auto-scaling target utilization to 90%
4. Add DynamoDB Accelerator (DAX) for write caching

Related Posts

• SQS Standard vs FIFO – Differences, Ordering & When to Use
• Event-Driven Serverless Architecture
• SNS vs SQS vs EventBridge
• Lambda Durable Functions
• Container Platform Architecture
• Aurora vs RDS vs DynamoDB

References

• Lambda Best Practices — AWS Docs
• Lambda Performance Optimization — AWS Compute Blog
• Using RDS Proxy with Lambda — AWS Blog
• HTTP API vs REST API — API Gateway Docs
• Well-Architected Serverless Applications — AWS Blog

Frequently Asked Questions

When should I use serverless vs containers?

Use serverless (Lambda) for event-driven, short-duration tasks (<15 min), variable traffic with periods of zero load, and when you want zero infrastructure management. Use containers (ECS/EKS) for long-running processes, consistent high throughput, specific runtime requirements, or when you need persistent connections (WebSocket servers, gRPC).

What is the difference between HTTP API and REST API?

HTTP API is 70% cheaper, lower latency, and simpler — supports JWT auth, Lambda/HTTP integrations, and CORS. REST API adds caching, usage plans/API keys, request/response transformation, WAF integration, and resource policies. Use HTTP API unless you specifically need a REST API feature.

How do I handle tasks longer than 29 seconds?

API Gateway’s maximum timeout is 29 seconds (hard limit). For longer tasks: (1) Accept request and return 202 immediately, (2) Process asynchronously via SQS → Lambda or Step Functions, (3) Client polls a status endpoint or receives a WebSocket/callback notification when complete. This is the standard async API pattern.

AWS Multi-Region Active-Active Architecture — Overview

A multi-region active-active architecture serves traffic from multiple AWS regions simultaneously, providing near-zero RTO/RPO, global low-latency access, and resilience against full region failures. This is the most advanced availability pattern and is heavily tested on SAP-C02 for mission-critical workloads.

Data Layer — Active-Active Options

When to Use Which Database

• DynamoDB Global Tables: True active-active (write in any region), key-value/document workloads, can tolerate eventual consistency and last-writer-wins
• Aurora Global: Relational workloads needing SQL/joins, single-writer acceptable (reads globally), fast managed failover
• Aurora DSQL: Need strong consistency with active-active writes across regions (newest option, distributed SQL)

Global Traffic Routing

Route 53 vs Global Accelerator for Failover

• Route 53: DNS-based — failover speed depends on TTL (typically 60s). Client caches DNS. Good for most web applications.
• Global Accelerator: Network-layer — failover in seconds (no DNS caching issue). Static anycast IPs never change. Better for latency-sensitive or non-HTTP applications.

Key Challenges & Solutions

1. Data Consistency

• Problem: Two regions writing to the same record simultaneously → conflict
• DynamoDB solution: Last-writer-wins (timestamp-based). Application must tolerate this. Use conditional writes for critical operations.
• Aurora solution: Single writer region eliminates write conflicts. Use write-forwarding for reads-everywhere, writes-to-primary.
• Aurora DSQL: Distributed transactions with strong consistency (no conflicts, correctness guaranteed)

2. Session Management

• Problem: User session created in Region A, next request routes to Region B
• Solutions:
  • ElastiCache Global Datastore for cross-region session replication
  • DynamoDB Global Tables for session storage (available everywhere)
  • Sticky sessions (Route 53 geo/latency keeps user in same region)
  • Stateless JWT tokens (no server-side session needed)

3. Split-Brain Prevention

• Problem: Network partition between regions — both think they’re the primary
• Solution: DynamoDB Global Tables handle this natively (multi-master, no split-brain). Aurora uses managed failover with fencing (old writer is demoted). For custom logic: use DynamoDB conditional writes as a distributed lock.

4. Deployment Consistency

• Problem: Application versions must be identical across regions
• Solution: CI/CD pipeline deploys to all regions simultaneously (CodePipeline cross-region actions). Use CloudFormation StackSets for infrastructure. Container images replicated via ECR cross-region replication.

Cost Considerations

• Compute: Full production capacity in 2+ regions (2x baseline compute cost)
• Data transfer: Cross-region replication incurs data transfer fees ($0.02/GB between regions)
• DynamoDB Global Tables: Pay for write capacity in each replica region (replicated writes charged)
• Aurora Global: Read replicas in secondary region are cheaper than full writer clusters
• Optimization: Use CloudFront to reduce origin requests (cache hit → no cross-region call). Compress data before replication.

Active-Active vs Active-Passive

Exam Tips

AWS Certification Exam Practice Questions

Question 1:

A global financial platform requires users in US and Europe to have read-write access to the same data with sub-5ms latency. The system must continue operating with zero downtime even if an entire AWS region fails. Which database architecture supports this?

1. Aurora Global Database with write-forwarding from European reader
2. DynamoDB Global Tables with replicas in us-east-1 and eu-west-1
3. RDS Multi-AZ with cross-region read replicas and manual promotion
4. ElastiCache Global Datastore with application-level write routing

Question 2:

A company uses Route 53 latency-based routing for multi-region active-active. During a regional outage, they notice some users still route to the failed region for up to 60 seconds due to DNS TTL caching. How can they achieve faster failover?

1. Reduce Route 53 TTL to 1 second
2. Replace Route 53 with AWS Global Accelerator
3. Add CloudFront in front with origin failover group
4. Use Route 53 health checks with faster interval (10s)

Question 3:

A multi-region active-active application using DynamoDB Global Tables discovers that two regions occasionally write to the same item simultaneously, causing data inconsistency. The business logic requires that only the FIRST write wins (not last). How should they handle this?

1. Switch to Aurora Global Database (single writer prevents conflicts)
2. Use DynamoDB conditional writes with version attribute (reject if version changed)
3. Configure DynamoDB Global Tables to use first-writer-wins instead of last-writer-wins
4. Add SQS queue to serialize all writes through a single region

Question 4:

A company wants to deploy a multi-region active-active web application. They need users globally to experience <50ms response time for static content and <200ms for dynamic API calls. Which combination of services achieves this?

1. Route 53 latency routing → ALB in each region (both static and dynamic)
2. CloudFront (static + dynamic caching) + Route 53 latency routing → ALB for API + Global Accelerator for WebSocket
3. Global Accelerator for everything (static + dynamic)
4. CloudFront with Lambda@Edge processing all requests

Question 5:

An active-active application stores user sessions in ElastiCache Redis. When a user’s requests route to a different region (due to Route 53 latency changes), they lose their session and must re-authenticate. How should this be fixed?

1. Use sticky sessions in Route 53 (geolocation routing to keep user in one region)
2. Replace ElastiCache with DynamoDB Global Tables for session storage
3. Use ElastiCache Global Datastore to replicate sessions cross-region
4. Store sessions in client-side JWT tokens (stateless, no server session)

Related Architecture Patterns

• Disaster Recovery – All 4 Strategies
• Hybrid Cloud Networking
• Data Lake & Analytics Architecture
• All Architecture Patterns – Hub

Related Posts

• Global Traffic Management – Route 53, GA & CloudFront
• Disaster Recovery Architecture
• Hybrid Cloud Networking

References

• DR Architecture Part IV: Multi-Site Active/Active — AWS Blog
• DynamoDB Global Tables Developer Guide
• Aurora Global Database for HA — AWS Blog
• Global Accelerator Performance — AWS Blog

Frequently Asked Questions

Does active-active mean both regions handle writes?

It depends on the database. With DynamoDB Global Tables: yes, both regions accept writes (multi-master). With Aurora Global: only one region writes (single-master), other regions serve reads with fast failover to become writer. True multi-region active-active writes require DynamoDB Global Tables or Aurora DSQL.

How much does active-active cost compared to single-region?

Roughly 2x for compute (full capacity in both regions) + cross-region data transfer fees ($0.02/GB). DynamoDB Global Tables charge for replicated write capacity. The cost premium is justified only for mission-critical workloads where any downtime has severe business impact. Consider Warm Standby (1.3x cost) as a cheaper alternative if minutes of RTO is acceptable.

What is the difference between Global Accelerator and CloudFront for multi-region?

CloudFront is a CDN that caches content at edge locations — best for HTTP/HTTPS with cacheable content. Global Accelerator is a network-layer accelerator with static anycast IPs — best for non-cacheable traffic, TCP/UDP applications, or when you need instant failover without DNS delays. Use both together: CloudFront for static/cacheable, Global Accelerator for real-time APIs/WebSocket.

AWS Config vs CloudTrail vs CloudWatch — Overview

These three services are frequently confused because they all relate to “monitoring.” However, each serves a distinct purpose: CloudWatch monitors performance metrics and logs, CloudTrail audits API activity (who did what), and Config tracks resource configuration compliance (is it configured correctly). This is a common SAP-C02 and SCS-C03 question.

Detailed Comparison

When to Use Which

How They Work Together

A common scenario illustrating all three:

1. CloudWatch alarm fires: RDS CPU at 95% for 10 minutes
2. Investigation via CloudTrail: Who modified the RDS instance? → Shows IAM user changed instance type from db.r5.xlarge to db.t3.micro 2 hours ago
3. Config shows: The RDS instance is now non-compliant with the “minimum-rds-instance-size” Config rule → Configuration timeline shows exact before/after
4. Resolution: Config auto-remediation triggers SSM Automation to resize back to compliant instance type

CloudWatch Deep Dive

• Metrics: Standard (5-min, free) + Detailed (1-min, paid) + Custom (put your own). Up to 1-second resolution with high-resolution metrics.
• Logs: Log Groups → Log Streams. Metric filters extract metrics from log data. Logs Insights for SQL-like queries.
• Alarms: Metric alarms (threshold), composite alarms (combine multiple), anomaly detection (ML-based).
• Synthetics: Canary scripts that test endpoints on a schedule (availability monitoring).
• Application Signals: APM — auto-discovery of services, SLO tracking, correlated metrics/traces/logs.

CloudTrail Deep Dive

• Management events: Control plane operations (CreateBucket, RunInstances, AttachRolePolicy). Enabled by default.
• Data events: Data plane operations (GetObject, PutObject, InvokeFunction). Must be explicitly enabled (high volume).
• Insights events: Detect unusual API activity patterns (spike in API calls, error rate anomalies).
• Organization Trail: Single trail covers all accounts in the organization → centralized to Log Archive S3 bucket.
• CloudTrail Lake: SQL-based querying of trail events (up to 7 years retention in Lake).

AWS Config Deep Dive

• Configuration Recorder: Records the configuration of supported resources whenever changes occur.
• Config Rules: Managed rules (180+) or custom Lambda rules. Evaluate on change or on schedule.
• Auto-Remediation: Link a rule to an SSM Automation document → auto-fix non-compliant resources.
• Conformance Packs: Collection of rules deployed as a unit (e.g., “PCI-DSS pack” with 30 rules).
• Aggregator: Aggregate compliance data from multiple accounts/regions into single dashboard.
• Advanced Queries: SQL-like queries across all recorded resource configurations.

Exam Tips

AWS Certification Exam Practice Questions

Question 1:

A security team needs to answer: “Which IAM user terminated the production EC2 instance last night at 11 PM?” Which service provides this information?

1. CloudWatch Logs
2. AWS CloudTrail
3. AWS Config
4. VPC Flow Logs

Question 2:

A compliance requirement states: “All RDS instances must have encryption enabled. Any unencrypted RDS instance must be automatically flagged and the team notified.” Which service implements this?

1. CloudWatch alarm on RDS metrics
2. CloudTrail monitoring CreateDBInstance events
3. AWS Config rule (rds-storage-encrypted) with SNS notification on non-compliance
4. GuardDuty RDS Protection

Question 3:

An architect needs to view how a Security Group was configured 3 weeks ago, before someone changed it and broke the application. Which service provides this historical configuration view?

1. CloudTrail (shows the API call that changed it)
2. AWS Config (shows the actual configuration at any point in time)
3. CloudWatch (shows metrics from 3 weeks ago)
4. VPC Flow Logs (shows network traffic history)

Question 4:

A company wants to detect when someone disables CloudTrail logging in any account (a potential indicator of compromise). What is the FASTEST way to detect and alert on this?

1. AWS Config rule checking CloudTrail status hourly
2. EventBridge rule matching the CloudTrail StopLogging API event → SNS alert
3. CloudWatch metric on CloudTrail event count dropping to zero
4. GuardDuty Stealth finding for CloudTrail disabled

Question 5:

A company needs a unified view of resource compliance across 100 accounts and 4 regions. They want to see compliance percentages for CIS benchmarks, PCI-DSS, and custom rules in a single dashboard. Which approach is MOST efficient?

1. AWS Config Aggregator in a central account collecting from all accounts/regions
2. Security Hub with compliance standards enabled across all accounts
3. Custom dashboard querying CloudWatch metrics from each account
4. CloudTrail Lake with compliance-focused SQL queries

Related Architecture Patterns

• Security Services Architecture
• Multi-Account Architecture
• CI/CD Pipeline
• All Architecture Patterns – Hub

Related Posts

• AWS Security Services Architecture – GuardDuty, Security Hub, Inspector
• Multi-Account Architecture – Control Tower & Governance
• Zero Trust Security Architecture
• Event-Driven Architecture (EventBridge patterns)

References

• AWS Config Developer Guide
• Monitor and Remediate Non-Compliant Resources — AWS Blog
• CloudTrail Concepts — AWS Docs
• CloudWatch Anomaly Detection for Security — AWS Blog

Frequently Asked Questions

Can CloudTrail replace CloudWatch Logs?

No — they serve different purposes. CloudTrail logs AWS API calls (control plane: who created/deleted/modified resources). CloudWatch Logs stores application logs, VPC Flow Logs, and custom log data (data plane: what happened inside your application). You need both for complete observability.

AWS Config vs CloudTrail — both track changes, what’s the difference?

CloudTrail answers “WHO made the API call and WHEN” (event-focused). Config answers “WHAT is the resource’s configuration NOW and BEFORE” (state-focused). CloudTrail gives you the action; Config gives you the resulting state. Config also evaluates compliance against rules — CloudTrail doesn’t.

Do I need all three services?

For production workloads: yes. CloudWatch for operational health and alerting. CloudTrail for security audit and accountability. Config for compliance and configuration drift detection. Most organizations enable all three as baseline. They’re complementary, not alternatives.

AWS Security Services Architecture — Overview

AWS provides a layered security services ecosystem for threat detection, vulnerability management, investigation, and centralized security posture management. Understanding how GuardDuty, Security Hub, Inspector, Detective, and Macie work together is essential for SCS-C03 and frequently tested on SAP-C02.

Service Comparison

How They Work Together

1. GuardDuty detects a compromised EC2 instance (cryptocurrency mining activity)
2. Security Hub receives the finding, correlates with Inspector findings (unpatched CVE on same instance)
3. EventBridge triggers automated response Lambda (isolates instance via Security Group change)
4. Detective investigates — shows what the compromised instance communicated with, what IAM calls it made, timeline of suspicious behavior
5. Config records the Security Group change for audit trail

Multi-Account Security Architecture

• Delegated Administrator: Designate a Security Tooling account as delegated admin for GuardDuty, Security Hub, Inspector, Macie (not the management account)
• Auto-enable: New member accounts automatically enrolled in all security services
• Cross-region aggregation: Security Hub can aggregate findings from all regions into a single region
• Centralized logging: All findings → Security Hub → EventBridge → S3 (Log Archive account) for SIEM integration

Automated Response Patterns

Exam Tips

AWS Certification Exam Practice Questions

Question 1:

A security team receives a GuardDuty finding that an EC2 instance is communicating with a known command-and-control (C2) server. They need to immediately contain the threat AND preserve forensic evidence. What should the automated response do?

1. Terminate the instance immediately
2. Change the Security Group to deny all traffic (isolate) but keep the instance running
3. Take an AMI snapshot and then terminate the instance
4. Remove the instance from its Auto Scaling Group

Question 2:

A company needs to ensure all EC2 instances across 50 accounts are scanned for software vulnerabilities (CVEs) automatically, with findings centralized in one place. Which architecture achieves this with LEAST operational overhead?

1. Install vulnerability scanning agents on all EC2 instances manually
2. Amazon Inspector with delegated admin in Security Tooling account, auto-enable for all org accounts, findings to Security Hub
3. AWS Config rules checking for unpatched software
4. GuardDuty Malware Protection scanning all instances

Question 3:

A compliance team needs to verify that no S3 bucket in the organization contains unencrypted PII (Social Security numbers, credit card numbers). They need ongoing monitoring, not just a one-time scan. Which service provides this?

1. AWS Config rule checking S3 encryption settings
2. Amazon Macie with automated sensitive data discovery
3. S3 access logging analysis with Athena
4. GuardDuty S3 protection

Question 4:

After GuardDuty detects suspicious API calls from an IAM user’s access keys, a security analyst needs to understand: what other resources did this user access in the last 30 days? What was their normal behavior pattern vs current anomaly? Which service provides this investigation capability?

1. CloudTrail Lake with SQL queries
2. Amazon Detective with entity behavior graphs
3. Security Hub finding details
4. CloudWatch Logs Insights

Question 5:

A company uses Security Hub with CIS AWS Foundations Benchmark. They want non-compliant findings to be automatically remediated (e.g., disable unused access keys, enable S3 encryption). What is the recommended automation architecture?

1. Security Hub → SNS → manual review → manual fix
2. Security Hub → EventBridge rules (filter by finding type) → SSM Automation documents (remediation)
3. AWS Config auto-remediation rules for each check
4. Lambda scheduled to scan Security Hub findings hourly and fix them

Related Architecture Patterns

• Zero Trust Security
• Multi-Account Architecture
• Config vs CloudTrail vs CloudWatch
• All Architecture Patterns – Hub

Related Posts

• AWS Zero Trust Architecture
• Multi-Account Architecture – Organizations & Control Tower
• Network Firewall vs WAF vs Security Groups vs NACLs
• Responsible AI – Guardrails & Governance
• WAF vs Shield vs Firewall Manager

References

• AWS Security Reference Architecture
• Automate Remediation with GuardDuty and Security Hub — AWS Blog
• Amazon Inspector EC2 Scanning — AWS Blog
• Amazon Detective Simplifies Investigations — AWS Blog

Frequently Asked Questions

What is the difference between GuardDuty and Inspector?

GuardDuty detects active threats and malicious behavior (someone attacking you NOW — C2 communication, brute force, cryptocurrency mining). Inspector finds vulnerabilities that COULD be exploited (unpatched CVEs, misconfigured software). GuardDuty = “you’re being attacked.” Inspector = “you’re vulnerable to attack.”

Do I need both Security Hub and GuardDuty?

Yes — they serve different roles. GuardDuty is a detection engine that generates findings. Security Hub is an aggregation and management platform that collects findings from GuardDuty AND Inspector, Macie, Config, Firewall Manager, and third-party tools. Security Hub provides the unified dashboard, compliance scoring, and automated action triggers.

What is the delegated administrator pattern?

Instead of managing security services from the management account (which should have minimal workloads), you designate a Security Tooling account as delegated admin for GuardDuty, Security Hub, Inspector, and Macie. This account manages security across all org members without needing access to the management account.

Kinesis vs MSK (Managed Kafka) — Overview

AWS offers two managed streaming platforms: Amazon Kinesis Data Streams (AWS-native, serverless option) and Amazon MSK (managed Apache Kafka). Both handle real-time data streaming but differ in complexity, ecosystem, pricing, and operational model. This comparison is relevant for SAP-C02 (architecture selection) and MLA-C01 (data ingestion for ML).

Feature Comparison

Decision Guide

Architecture Patterns

Pattern 1: Kinesis — Real-time Analytics Pipeline

• Flow: IoT devices → Kinesis Data Streams → Managed Flink (windowed aggregation) → DynamoDB/Redshift
• Why Kinesis: Serverless (on-demand mode), Lambda integration for lightweight processing, Firehose for direct S3/Redshift delivery

Pattern 2: MSK — Event Backbone for Microservices

• Flow: 20 microservices produce → MSK topics → Consumer groups per team → Each team processes independently
• Why MSK: Kafka Connect pulls from databases (CDC), consumer groups allow unlimited independent readers, Kafka Streams for stateful processing

Pattern 3: MSK → Kinesis Firehose (Best of Both)

• Flow: MSK (high-throughput streaming backbone) → Firehose (managed delivery to S3/Redshift/OpenSearch)
• Why: Kafka ecosystem for producers/consumers + Firehose’s auto-batching, compression, and format conversion for lake/warehouse delivery

Pricing Comparison

Rule of thumb: Kinesis is cheaper below ~5 MB/s. MSK becomes more cost-effective above ~50 MB/s with provisioned brokers + Graviton instances.

Exam Tips

AWS Certification Exam Practice Questions

Question 1:

A company is migrating from self-managed Apache Kafka on EC2 to a managed service. They have 30+ microservices using Kafka producers/consumers with custom Kafka Connect connectors. They want to minimize code changes. Which service should they use?

1. Amazon Kinesis Data Streams (rewrite producers to use AWS SDK)
2. Amazon MSK (drop-in Kafka replacement, same client code)
3. Amazon SQS FIFO (provides ordering like Kafka)
4. Amazon EventBridge (replace Kafka topics with event bus rules)

Question 2:

A startup needs real-time clickstream processing. They want the simplest, most serverless approach: data arrives → process in real-time → deliver to S3 in Parquet format. No Kafka expertise on the team. Which combination is simplest?

1. MSK Serverless → Kafka Streams → S3 sink connector
2. Kinesis Data Streams (on-demand) → Lambda → S3
3. Kinesis Data Streams → Kinesis Data Firehose → S3 (Parquet conversion built-in)
4. MSK → Managed Flink → S3

Question 3:

A financial platform needs exactly-once processing guarantees for payment events. Events must be processed exactly one time, even if producers retry. Which streaming option provides this natively?

1. Kinesis Data Streams with deduplication in Lambda consumer
2. Amazon MSK with Kafka transactions (exactly-once semantics)
3. SQS FIFO queue with deduplication ID
4. EventBridge with archive ensuring no duplicates

Question 4:

A company streams 500 MB/s of log data from 1000 servers. They need 30 days retention and 5 independent consumer applications reading the same data. Which is MOST cost-effective?

1. Kinesis with 500 shards + Enhanced Fan-Out for 5 consumers
2. MSK Provisioned with Graviton brokers + tiered storage + 5 consumer groups
3. Kinesis on-demand mode with 5 Lambda consumers
4. 5 separate Kinesis streams (one per consumer)

Question 5:

A team uses Kinesis Data Streams but finds that one consumer (analytics) is falling behind because the shared shard read limit (2MB/s) is consumed by another faster consumer (alerting). How should they fix this without increasing shards?

1. Increase the consumer’s batch size
2. Register the analytics consumer as an Enhanced Fan-Out (EFO) consumer
3. Move the analytics consumer to a separate Kinesis stream
4. Switch to MSK for consumer group isolation

Related Architecture Patterns

• SNS vs SQS vs EventBridge
• Event-Driven Serverless
• Data Lake & Analytics
• All Architecture Patterns – Hub

Related Posts

• SQS Standard vs FIFO – Differences, Ordering & When to Use
• Kinesis Streams vs Firehose vs Managed Flink
• SNS vs SQS vs EventBridge
• Event-Driven Serverless Architecture
• Data Lake & Analytics Architecture

References

• Best Practices for Kafka on AWS — AWS Blog
• Amazon MSK Serverless — AWS Blog
• Kinesis Enhanced Fan-Out — AWS Docs
• MSK Tiered Storage — AWS Blog

Frequently Asked Questions

Can I use Kinesis with existing Kafka consumers?

Not directly — Kinesis uses a proprietary API (AWS SDK / KCL), not the Kafka protocol. If you have existing Kafka consumers, MSK allows them to work with zero changes. To use Kinesis, you’d need to rewrite consumers using the KCL or Lambda event source mapping.

What is MSK Serverless vs MSK Provisioned?

MSK Serverless auto-scales partitions and throughput — no broker management, pay per data. MSK Provisioned gives you dedicated broker instances (you choose types, scaling is manual). Serverless is simpler; Provisioned is cheaper at high sustained throughput and gives more configuration control.

When should I use Managed Flink instead of Lambda for stream processing?

Use Lambda for simple, stateless per-record transformations (filter, enrich, route). Use Managed Flink for complex stateful processing: windowed aggregations, joins across streams, pattern detection, CEP (Complex Event Processing). Flink maintains state across records; Lambda processes each batch independently.

SNS vs SQS vs EventBridge — Overview

AWS offers three core messaging/eventing services that are often confused. Each serves a distinct purpose: SQS for queuing (point-to-point buffering), SNS for pub/sub (fan-out notifications), and EventBridge for event routing (content-based filtering from many sources). Choosing the right one — or combining them — is a frequent SAP-C02 and DOP-C02 exam question.

Feature Comparison

Decision Guide — When to Use Which

Common Architecture Patterns

Pattern 1: SNS + SQS Fan-out (Most Common)

One event → multiple independent processors, each with their own retry and failure handling.

• Flow: Order Service → SNS Topic → SQS-Inventory + SQS-Email + SQS-Analytics (each with Lambda consumer)
• Why not just SNS → Lambda directly? SQS adds buffering, retry with backoff, DLQ, and prevents Lambda throttling from affecting other consumers.

Pattern 2: EventBridge + SNS + SQS (Full Decoupling)

Many producers → EventBridge routes by type → SNS fans out → SQS buffers each consumer.

• Flow: 10 microservices emit events → EventBridge (rules filter by event type) → SNS per domain → SQS per consumer
• Why: Producers don’t know about consumers. New consumers subscribe without changing producers.

Pattern 3: SQS as Lambda Buffer (Load Leveling)

Protect downstream from traffic spikes.

• Flow: API Gateway → SQS → Lambda (controlled concurrency)
• Why: API Gateway has unlimited throughput but Lambda has concurrency limits. SQS absorbs spikes and Lambda processes at a controlled rate.

Pattern 4: EventBridge for AWS Service Events

React to infrastructure changes automatically.

• Flow: EC2 instance terminates → EventBridge rule → Lambda (cleanup DNS/LB) + SNS (alert ops team)
• Why: EventBridge receives AWS service events natively without any configuration on the source side.

Pricing Comparison

Cost tip: AWS service events on EventBridge (EC2 state changes, S3 events via EventBridge, etc.) are FREE. Only custom/partner events incur the $1/M charge.

Exam Tips

AWS Certification Exam Practice Questions

Question 1:

An order processing system needs to trigger three downstream services (inventory, billing, shipping) independently. If billing fails, inventory and shipping should still succeed. Each service needs its own retry logic. Which architecture is BEST?

1. SNS topic → 3 Lambda subscribers directly
2. SNS topic → 3 SQS queues (each with own Lambda consumer and DLQ)
3. EventBridge → 3 Lambda targets on one rule
4. SQS queue → Lambda → calls 3 services sequentially

Question 2:

A company receives events from 15 microservices. They need to route “payment” events to the payment processor, “inventory” events to the warehouse system, and “audit” events to the compliance system — based on an “event_type” field in the JSON body. Which service provides this content-based routing?

1. SNS with message attribute filter policies
2. SQS with multiple queues and client-side routing
3. EventBridge with rules filtering on event body content
4. Kinesis with partition key-based routing

Question 3:

A bug in a consumer caused it to silently discard events for 6 hours before being discovered. The events were published to SNS. Can they be recovered?

1. Yes — replay from SNS message retention
2. Yes — if the SQS subscriber had a DLQ configured
3. No — SNS does not persist messages; once delivered they are gone
4. Yes — retrieve from CloudWatch Logs

Question 4:

A company wants to trigger a Lambda function when ANY EC2 instance in their account changes state (running, stopped, terminated). They want this with zero custom code on the producer side. Which approach works?

1. CloudWatch Events (same as EventBridge) rule for EC2 state change → Lambda target
2. SNS topic that EC2 publishes to on state change → Lambda subscription
3. SQS queue polled by Lambda checking EC2 DescribeInstances
4. CloudTrail log with S3 event notification triggering Lambda

Question 5:

A team uses SNS to fan out order events to 5 subscribers. One subscriber (analytics) only needs “premium” orders (order_value > $100). Currently it receives ALL orders and filters client-side, wasting Lambda invocations. How should they reduce unnecessary invocations?

1. Create a separate SNS topic for premium orders
2. Add an SNS subscription filter policy matching the “order_tier” message attribute = “premium”
3. Replace SNS with EventBridge and use content-based filtering on order_value
4. Add a Lambda@Edge function to filter before delivery

Related Architecture Patterns

• Event-Driven Serverless
• Kinesis vs MSK
• Container Platform (ECS/EKS)
• All Architecture Patterns – Hub

Related Posts

• SQS Standard vs FIFO Queue – Complete Comparison
• AWS Event-Driven Serverless Architecture
• Lambda vs Step Functions vs EventBridge
• Kinesis Streams vs Firehose vs Managed Flink
• GenAI Application Architecture

References

• Choosing Between Messaging Services for Serverless — AWS Blog
• Let’s Architect: Event-Driven Architectures — AWS Blog
• Amazon EventBridge User Guide
• Building Event-Driven Applications with EventBridge — AWS Blog
• Filtering Messages with Amazon SNS — AWS Blog

Frequently Asked Questions

When should I use EventBridge vs SNS?

Use EventBridge when you need content-based filtering on the event body, integration with AWS service events or SaaS partners, schema discovery, or event replay. Use SNS for simple fan-out where attribute-based filtering is sufficient, multi-protocol delivery (email, SMS), or when you need FIFO ordering with topics.

Can I use SQS without SNS?

Yes. Use SQS alone when you have a single producer and single consumer (point-to-point). The producer sends directly to the queue, and one consumer processes messages. Add SNS in front only when you need the same message delivered to multiple independent consumers (fan-out).

What happens if an SNS subscriber is down?

For Lambda/HTTP subscribers: SNS retries with backoff (up to 23 days for HTTP). For SQS subscribers: messages are delivered to the SQS queue immediately (SQS is always “up”) and wait until the consumer processes them. This is why SNS→SQS is more reliable than SNS→Lambda directly — SQS persists messages even if the consumer is temporarily unavailable.

AWS Database Migration Architecture — Overview

Database migration is one of the most common workload migration patterns on AWS. The architecture combines AWS Database Migration Service (DMS) for data replication, AWS Schema Conversion Tool (SCT) for schema transformation, and various strategies depending on source/target compatibility and downtime tolerance. This is a core topic for SAP-C02 (migration scenarios) and DOP-C02 (automation).

AWS Database Migration Service (DMS)

DMS replicates data from source to target with minimal downtime. It runs on a managed replication instance in your VPC.

How DMS Works

1. Full Load: Migrates all existing data from source tables to target tables
2. Change Data Capture (CDC): Continuously replicates ongoing changes (inserts, updates, deletes) after full load completes
3. Full Load + CDC: Combines both — migrates existing data, then keeps target in sync until cutover

DMS Components

Supported Sources and Targets

AWS Schema Conversion Tool (SCT)

SCT converts database schemas and code from one engine to another (heterogeneous migrations only):

• Schema conversion: Tables, indexes, views, constraints → target engine syntax
• Code conversion: Stored procedures, functions, triggers → target equivalent
• Assessment Report: Identifies what can be auto-converted vs what needs manual effort (red/yellow/green)
• Data Extraction Agents: For large-scale migrations, SCT agents extract data in parallel to S3, then load into target

Not needed for homogeneous migrations (e.g., MySQL → Aurora MySQL) since the schema is compatible.

Migration Strategies (The 7 Rs)

Zero-Downtime Migration Pattern

1. Provision target — Create Aurora/RDS in target region/account
2. SCT (if heterogeneous) — Convert and apply schema to target
3. DMS Full Load + CDC — Start migration task: initial load + ongoing replication
4. Validation — Enable DMS data validation to compare source vs target row counts and checksums
5. Application testing — Point test/staging app at target DB, validate functionality
6. Cutover — Stop writes to source → Wait for CDC to catch up (seconds) → Redirect application to target
7. Decommission — Monitor target, then retire source after confidence period

Downtime: Only during step 6 (seconds to minutes — time for last CDC events to apply + DNS/connection string update).

Large-Scale Migration (TB to PB)

• Problem: Migrating 50TB+ over network takes weeks even on Direct Connect
• Solution: AWS Snowball Edge for initial bulk transfer + DMS CDC for changes during transit
• Pattern: Start CDC from Day 1 → Ship Snowball (days) → Load Snowball data to target → Apply accumulated CDC → Cutover
• Alternative: SCT Data Extraction Agents — parallel extraction to S3 via Direct Connect, then load to target

Target Database Selection

Exam Tips by Certification

AWS Certification Exam Practice Questions

Question 1:

A company needs to migrate a 10TB Oracle database to Aurora PostgreSQL with less than 1 hour of downtime. The application must continue running during migration. Which approach achieves this?

1. Use AWS SCT to convert schema, then pg_dump/pg_restore for data
2. Use AWS SCT for schema + DMS with Full Load and CDC, then cutover after CDC catches up
3. Create an Oracle RDS instance first, then use Aurora read replica promotion
4. Use AWS Snowball Edge to transfer the data, then apply schema with SCT

Question 2:

A company has a 50TB data warehouse they want to migrate to AWS. Network bandwidth to AWS is 1 Gbps (would take ~5 days for full transfer). They need the target ready within 2 weeks and cannot tolerate data loss. What is the MOST efficient approach?

1. DMS Full Load over Direct Connect for 5 days, then CDC
2. AWS Snowball Edge for bulk data + DMS CDC running in parallel from Day 1
3. S3 Transfer Acceleration for faster upload
4. Multiple DMS replication instances in parallel

Question 3:

A company is migrating from Oracle to Aurora PostgreSQL. The SCT assessment report shows 60% of stored procedures can be auto-converted, but 40% require manual intervention. The migration has a 3-month deadline. What should they prioritize?

1. Rewrite all stored procedures in the application layer and remove database logic
2. Convert the 60% automatically with SCT, manually convert the critical 40%, begin DMS replication in parallel
3. Stay on Oracle and rehost to Oracle on EC2 instead
4. Use AWS Lambda to emulate Oracle stored procedures

Question 4:

After starting a DMS migration task, a company notices some tables have row count mismatches between source and target. What DMS feature helps identify and resolve this?

1. CloudWatch metrics for table statistics
2. DMS data validation (automatic source-target comparison)
3. DMS premigration assessment
4. AWS Config compliance rules

Question 5:

A company wants to migrate their on-premises MySQL database to RDS MySQL. The source and target use the same engine version. What is the SIMPLEST migration approach?

1. SCT for schema conversion + DMS for data migration
2. DMS only (Full Load + CDC) — no SCT needed for homogeneous migration
3. mysqldump export → S3 → RDS import
4. Create RDS read replica from on-premises using binary log replication

Related Architecture Patterns

• Data Lake & Analytics
• Disaster Recovery
• Hybrid Cloud Networking
• All Architecture Patterns – Hub

Related Posts

• AWS DMS – Database Migration Service, CDC & Replication
• AWS RDS Replication – Multi-AZ & Read Replicas
• Aurora vs RDS vs DynamoDB – Selection Guide
• AWS Data Lake & Analytics Architecture
• AWS Disaster Recovery Architecture
• Hybrid Cloud Networking – Direct Connect & Transit Gateway
• Migration Architecture – 7Rs & Tools
• RDS & Aurora Performance

References

• AWS Database Migration Service User Guide
• Migrate Oracle to Aurora – AWS Database Blog
• Migrating Large Databases to AWS – Prescriptive Guidance
• Best Practices for Migrating to Amazon Aurora – AWS Blog
• Database Migration Strategy – AWS Prescriptive Guidance

Frequently Asked Questions

What is the difference between DMS and SCT?

DMS migrates the data (rows) from source to target with optional ongoing replication (CDC). SCT converts the schema and code (stored procedures, functions, triggers) from one database engine to another. For homogeneous migrations (same engine), only DMS is needed. For heterogeneous migrations (different engines), both SCT and DMS are required.

Can DMS migrate to a different AWS region?

Yes. DMS supports cross-region migration. The replication instance can be in either the source or target region. For on-premises to AWS, place the replication instance in the target region. For cross-region AWS migrations, place it in the target region for better performance (closer to where data is written).

How do I achieve zero-downtime migration?

Use DMS Full Load + CDC: the full load migrates existing data while the application continues running. CDC captures all changes made during migration. At cutover, briefly stop application writes (seconds), wait for CDC to apply final changes, then redirect the application to the new target. Total downtime: seconds to minutes.

AWS Architecture Patterns — Reference Diagrams for Certification

58 architecture pattern posts covering all 12 AWS certification exams. Each post includes HTML/CSS diagrams, comparison tables, design decisions, exam tips, and 5 practice questions with expandable answers.

🏗️ Core Architecture Patterns (SAP-C02 / SAA-C03)

🌐 Networking (ANS-C01 / SAP-C02)

🔒 Security (SCS-C03 / SAP-C02)

🤖 AI/ML & GenAI (AIP-C01 / AIF-C01 / MLA-C01)

📊 Data & Analytics (DAS-C01 / DEA-C01 / DBS-C01)

⚙️ DevOps & Operations (DOP-C02 / SOA-C03)

🚀 Migration & Storage (SAP-C02 / SAA-C03)

AWS Container Platform Architecture — Overview

Containers are the standard deployment unit for microservices on AWS. The platform spans image management (ECR), orchestration (ECS/EKS), compute (Fargate/EC2), networking (ALB/NLB/Service Connect), and observability (Container Insights/X-Ray). This architecture is heavily tested on DOP-C02 (deployment and operations) and SAP-C02 (architectural decisions).

ECS vs EKS — Decision Guide

Compute — Fargate vs EC2

Cost Decision Rule of Thumb

• Fargate cheaper when: Utilization is <50% (bursty workloads, dev/test, batch jobs)
• EC2 cheaper when: Utilization is >50% consistently (use Reserved Instances or Savings Plans + Spot for non-critical)
• Mixed strategy: Fargate for baseline + EC2 Spot capacity providers for spikes

Networking Patterns

Ingress (North-South)

• ALB: Layer 7 — path/host routing, gRPC, WebSocket, authentication integration. Best for HTTP/HTTPS microservices.
• NLB: Layer 4 — ultra-low latency, static IPs, TCP/UDP. Best for non-HTTP protocols, high throughput.
• API Gateway: For serverless-first architectures integrating with containers via VPC Link → NLB → ECS/EKS.

Service Discovery (East-West)

Auto-Scaling

ECS Scaling

• Service Auto Scaling: Target tracking (CPU/Memory utilization), step scaling, or scheduled scaling on desired task count
• Capacity Providers: ECS manages EC2 ASG capacity automatically — scales nodes when tasks need capacity
• Fargate: No node scaling needed — each task gets its own compute instantly

EKS Scaling

• Horizontal Pod Autoscaler (HPA): Scales pod replicas based on CPU/memory/custom metrics
• Karpenter: Provisions optimal EC2 nodes in seconds based on pending pod requirements (replaces Cluster Autoscaler)
• KEDA: Event-driven scaling (scale on SQS queue depth, Kafka lag, custom metrics)
• Fargate on EKS: Pods run serverless — no node scaling, instant per-pod provisioning

Deployment Strategies

Security

Observability Stack

• Container Insights: CPU, memory, network, disk metrics per task/pod + cluster-level dashboards (CloudWatch)
• X-Ray: Distributed tracing across services — inject X-Ray daemon as sidecar or use ADOT (OpenTelemetry)
• FireLens: ECS log router (Fluent Bit sidecar) — route logs to CloudWatch, S3, OpenSearch, Datadog, Splunk
• Prometheus + Grafana: Native for EKS — Amazon Managed Prometheus (AMP) + Amazon Managed Grafana (AMG)
• EKS audit logs: Control plane API audit logs → CloudWatch Logs (detect unauthorized access attempts)

Exam Tips by Certification

AWS Certification Exam Practice Questions

Question 1:

A company runs 50 microservices on EKS. Their cluster nodes are often under-utilized (30% average CPU) because the Cluster Autoscaler provisions nodes too conservatively. They want faster scaling with better bin-packing. What should they use?

1. Increase HPA target utilization from 50% to 80%
2. Replace Cluster Autoscaler with Karpenter
3. Switch all workloads to Fargate
4. Use larger EC2 instance types for better packing

Question 2:

An ECS service needs to connect to other ECS services by name (e.g., http://payment-service:8080) with automatic load balancing, health checking, and traffic metrics — without managing an ALB for each internal service. Which feature provides this?

1. Cloud Map DNS-based discovery
2. ECS Service Connect
3. VPC Lattice
4. Application Load Balancer with multiple target groups

Question 3:

An EKS pod needs to access an S3 bucket and DynamoDB table. The security team requires that ONLY this specific pod gets these permissions — not other pods on the same node. What is the recommended approach?

1. Attach the permissions to the node’s EC2 instance profile
2. Use IAM Roles for Service Accounts (IRSA) or EKS Pod Identity
3. Store access keys as K8s secrets and mount in the pod
4. Use a VPC endpoint policy to restrict access to this pod’s security group

Question 4:

A company wants zero-downtime deployments for their ECS Fargate service with the ability to instantly roll back if health checks fail. The deployment should create a new task set, validate it, then switch traffic. Which configuration achieves this?

1. ECS rolling update with minimumHealthyPercent=100, maximumPercent=200
2. CodeDeploy Blue/Green deployment with ECS, ALB listener rules, and CloudWatch alarm-based rollback
3. Manual task count adjustment — double tasks, then drain old
4. ECS service with multiple task definitions running simultaneously

Question 5:

A startup runs a variable workload on ECS — 2 tasks during business hours and 0 tasks at night. They want minimum cost with no node management overhead. On weekdays, tasks need to scale up to 20 within seconds during marketing campaigns. Which configuration fits?

1. EC2 launch type with Reserved Instances and an ASG scheduled action
2. Fargate launch type with ECS Service Auto Scaling (target tracking on CPU)
3. EC2 Spot instances with capacity provider managed scaling
4. Fargate Spot for all tasks to minimize cost

Related Architecture Patterns

• CI/CD Pipeline Architecture
• Event-Driven Serverless
• Zero Trust Security
• All Architecture Patterns – Hub

Related Posts

• AWS EKS vs ECS – Decision Guide
• AWS ALB vs NLB vs GWLB
• AWS CI/CD Pipeline Architecture
• AWS Event-Driven Serverless Architecture
• AWS Zero Trust Architecture

References

• AWS Containers Blog
• EKS Best Practices Guide
• ECS Service Connect — AWS Blog
• Karpenter — Node Autoscaling for Kubernetes
• ECS Deployment Circuit Breaker — AWS Blog

📖 Related: Launch Template vs Launch Configuration

Frequently Asked Questions

Is Fargate always more expensive than EC2?

No. Fargate is cheaper for variable, bursty workloads with <50% average utilization because you pay nothing when tasks aren’t running. EC2 becomes cheaper at >50% sustained utilization, especially with Reserved Instances or Savings Plans. Most production workloads benefit from a mix: Fargate for base + EC2 Spot for peaks.

ECS vs EKS for a new project — which should I pick?

If your team has no Kubernetes experience and you’re AWS-only, choose ECS — less operational overhead, tighter AWS integration, no control plane cost. If you need Kubernetes portability, have K8s expertise, or want ecosystem tools (Helm, ArgoCD, Istio, OPA), choose EKS. Both support Fargate.

Can I run sidecars on Fargate?

Yes. ECS Fargate supports multiple containers per task definition (native sidecars with lifecycle dependencies since 2023). EKS Fargate also supports sidecar containers in pods. Common sidecars: Envoy (Service Connect), X-Ray daemon, Fluent Bit (logs), ADOT (OpenTelemetry).

AWS Data Lake & Analytics Architecture — Overview

A data lake centralizes structured, semi-structured, and unstructured data in S3, enabling analytics at scale without upfront schema design. Combined with AWS analytics services, it forms a complete pipeline from ingestion through transformation to visualization. This architecture is critical for SAP-C02 (architectural decisions) and MLA-C01 (data preparation for ML).

Data Lake Zones

A well-designed data lake organizes data into zones (S3 prefixes) representing data maturity:

Key principle: Raw zone is immutable — never modify source data. All transformations produce new objects in curated/enriched zones. This enables reprocessing if transformation logic changes.

Ingestion Layer

Catalog & Governance — Lake Formation

AWS Lake Formation is the governance layer that sits on top of the Glue Data Catalog:

• Glue Data Catalog: Central metadata repository — databases, tables, schemas, partitions (Hive metastore compatible)
• Lake Formation Permissions: Fine-grained access control (row-level, column-level, cell-level filtering)
• Tag-Based Access Control (TBAC): Assign tags to tables/columns (e.g., “PII=true”), grant access based on tags — scales across thousands of tables
• Cross-account sharing: Share databases/tables with other AWS accounts via RAM or direct grant (no data copying)
• Governed Tables: ACID transactions on S3 data (via Apache Iceberg integration)

Processing & Transformation

Analytics & Visualization

Cost Optimization

• Columnar formats (Parquet/ORC): 3-5x compression vs CSV/JSON, Athena scans less data = lower cost
• Partitioning: Partition by date/region/product — queries scan only relevant partitions (e.g., WHERE year=2026 scans 1/N of data)
• Compaction: Merge small files into larger ones (1-128MB optimal) — reduces list/get operations and improves query speed
• S3 Intelligent-Tiering: Automatically moves infrequently accessed data to cheaper tiers (no retrieval fees)
• Athena workgroups: Set per-query and per-workgroup data scan limits to prevent runaway costs
• Redshift Serverless: Pay per RPU-second — no idle cost for intermittent warehouse workloads

Exam Tips by Certification

AWS Certification Exam Practice Questions

Question 1:

A company stores 50TB of data in S3 as CSV files. Data analysts run Athena queries daily but costs are high ($500/day) because queries scan the entire dataset. What TWO changes would MOST reduce Athena costs?

1. Convert CSV to Parquet and partition by date
2. Move data to Redshift and query there instead
3. Enable S3 Transfer Acceleration for faster queries
4. Add more Athena workgroups to distribute load

Question 2:

A financial company needs different teams to access different columns of the same table. The compliance team needs all columns, while the analytics team should NOT see PII columns (SSN, email). The table has 200+ columns. What is the MOST scalable approach?

1. Create separate copies of the table with PII columns removed for analytics
2. Lake Formation column-level permissions with tag-based access control
3. S3 bucket policies restricting access to specific prefixes per team
4. Athena views that exclude PII columns for the analytics team

Question 3:

A company needs to continuously replicate changes from their on-premises Oracle database to S3 for analytics. They need ongoing synchronization (not just a one-time migration) with minimal impact on the source database. Which service provides this?

1. AWS Glue with JDBC connection on a schedule
2. AWS DMS with CDC (Change Data Capture) to S3 target
3. Kinesis Data Firehose with custom producer
4. AWS DataSync on a recurring schedule

Question 4:

A data engineering team needs to process 5TB of raw JSON data daily, apply complex transformations (joins across 10 tables, deduplication, schema evolution), and write results to S3 in Parquet. They want serverless with no cluster management. Which service fits?

1. Athena CTAS (Create Table As Select)
2. AWS Glue ETL with auto-scaling
3. Amazon EMR Serverless
4. Lambda with S3 batch operations

Question 5:

A company wants to share specific tables from their data lake with a partner AWS account without copying data. The partner should only see curated tables with certain rows filtered (e.g., only their region’s data). What is the recommended approach?

1. Copy filtered data to an S3 bucket in the partner account nightly
2. Lake Formation cross-account sharing with row-level security filters
3. Redshift data sharing with the partner account
4. S3 bucket policy granting partner account read access to specific prefixes

Related Architecture Patterns

• Kinesis vs MSK
• MLOps Pipeline Architecture
• Database Migration (DMS)
• All Architecture Patterns – Hub

Related Posts

• AWS S3 Storage Classes
• Kinesis Data Streams vs Firehose vs Managed Flink
• Amazon Athena
• AWS S3 vs EBS vs EFS – Storage Comparison
• AWS MLOps Pipeline Architecture
• AWS Event-Driven Serverless Architecture
• EMR & Spark Architecture

References

• Building a Data Lake on AWS — AWS Whitepaper
• Design a Data Mesh Architecture Using Lake Formation and Glue — AWS Blog
• Build a Serverless Data Pipeline with Glue and Athena — AWS Blog
• Best Practices for Data Lake S3 Bucket Organization — AWS Prescriptive Guidance
• Reduce Cost and Improve Query Performance with Athena — AWS Blog

📖 Related: AWS EBS Volume Types – Performance & Cost Comparison

Frequently Asked Questions

Data lake vs data warehouse — when to use which?

A data lake (S3) stores raw data in any format at low cost — best for data exploration, ML, and diverse data types. A data warehouse (Redshift) stores structured, pre-modeled data optimized for fast, repeated analytical queries. Most organizations use both: lake for storage/exploration, warehouse for production dashboards and business intelligence.

AWS Glue vs EMR — when to use which?

Use Glue for standard ETL jobs where you want serverless, auto-scaling, and visual job authoring. Use EMR when you need custom Spark/Hadoop configurations, specific library versions, long-running clusters, or workloads exceeding Glue’s capabilities. Glue is simpler; EMR is more flexible and cost-effective at very large scale with spot instances.

What is Lake Formation vs Glue Data Catalog?

Glue Data Catalog is the metadata repository (table definitions, schemas, partitions). Lake Formation is the governance layer on top that adds fine-grained access control (row/column/cell-level), tag-based permissions, cross-account sharing, and data location registration. Lake Formation uses the Catalog underneath but adds security and governance.