AWS Data Lake & Analytics Architecture — Overview
A data lake centralizes structured, semi-structured, and unstructured data in S3, enabling analytics at scale without upfront schema design. Combined with AWS analytics services, it forms a complete pipeline from ingestion through transformation to visualization. This architecture is critical for SAP-C02 (architectural decisions) and MLA-C01 (data preparation for ML).
Glue (batch ETL)
DMS (databases)
AppFlow (SaaS)
Transfer Family
Raw zone
Curated zone
Enriched zone
Parquet / Iceberg
Lake Formation
Row/Column ACLs
Tag-based access
Macie (PII)
EMR (Hadoop/Spark)
Athena (SQL)
Redshift Spectrum
Lambda (light)
Redshift (warehouse)
QuickSight (BI)
OpenSearch (logs)
SageMaker (ML)
Lake Formation
Fine-grained permissions
Cross-account sharing
KMS encryption
VPC endpoints
CloudTrail audit
S3 Intelligent-Tiering
Parquet/ORC columnar
Partitioning + compaction
Data Lake Zones
A well-designed data lake organizes data into zones (S3 prefixes) representing data maturity:
| Zone | Content | Format | Access |
|---|---|---|---|
| Raw (Landing) | Exact copy of source data — immutable | Original (JSON, CSV, logs, DB exports) | Data engineers only |
| Curated (Cleansed) | Validated, deduplicated, schema-conformed | Columnar (Parquet/ORC), partitioned | Data analysts, ML engineers |
| Enriched (Business) | Aggregated, joined, business-ready datasets | Parquet, Iceberg tables | Business analysts, BI tools |
Key principle: Raw zone is immutable — never modify source data. All transformations produce new objects in curated/enriched zones. This enables reprocessing if transformation logic changes.
Ingestion Layer
| Service | Type | Use Case |
|---|---|---|
| Kinesis Data Streams | Real-time streaming | Clickstreams, IoT telemetry, application logs (sub-second) |
| Kinesis Data Firehose | Near real-time delivery | Stream directly to S3/Redshift/OpenSearch with auto-batching (60s-900s buffer) |
| AWS Glue (Crawlers + Jobs) | Batch ETL | Scheduled extraction from databases, APIs, file systems |
| AWS DMS | Database migration/replication | Ongoing CDC (Change Data Capture) from RDS/on-prem databases → S3 |
| Amazon AppFlow | SaaS integration | Pull data from Salesforce, SAP, Google Analytics, Slack → S3 |
| Transfer Family | SFTP/FTPS/FTP | Partner data feeds using legacy file transfer protocols → S3 |
Catalog & Governance — Lake Formation
AWS Lake Formation is the governance layer that sits on top of the Glue Data Catalog:
- Glue Data Catalog: Central metadata repository — databases, tables, schemas, partitions (Hive metastore compatible)
- Lake Formation Permissions: Fine-grained access control (row-level, column-level, cell-level filtering)
- Tag-Based Access Control (TBAC): Assign tags to tables/columns (e.g., “PII=true”), grant access based on tags — scales across thousands of tables
- Cross-account sharing: Share databases/tables with other AWS accounts via RAM or direct grant (no data copying)
- Governed Tables: ACID transactions on S3 data (via Apache Iceberg integration)
Processing & Transformation
| Service | Best For | Scale |
|---|---|---|
| Glue ETL | Serverless Spark — schema evolution, dedup, format conversion, joins | Auto-scaling DPUs, no cluster management |
| EMR | Heavy big data processing — Spark, Hive, Presto, Flink on managed clusters | PB-scale, custom libraries, spot instances for cost |
| Athena | Interactive SQL on S3 — no infrastructure, pay per query | Ad-hoc analysis, CTAS for transformations |
| Redshift Spectrum | Query S3 data from Redshift (extend warehouse to lake) | Join lake data with warehouse data in single query |
| Lambda | Lightweight transformations triggered by S3 events | Small files, simple transforms, real-time triggers |
Analytics & Visualization
| Service | Type | When to Use |
|---|---|---|
| Athena | Serverless SQL | Ad-hoc queries, exploration, infrequent access (pay per TB scanned) |
| Redshift | Data warehouse | Frequent complex queries, dashboards, predictable workloads, sub-second response |
| QuickSight | BI / Dashboards | Business dashboards, embedded analytics, ML insights (anomaly/forecasting) |
| OpenSearch | Log analytics / Search | Log analysis, full-text search, real-time dashboards (Kibana) |
| SageMaker | Machine Learning | Train models on lake data, feature engineering from curated zone |
Cost Optimization
- Columnar formats (Parquet/ORC): 3-5x compression vs CSV/JSON, Athena scans less data = lower cost
- Partitioning: Partition by date/region/product — queries scan only relevant partitions (e.g., WHERE year=2026 scans 1/N of data)
- Compaction: Merge small files into larger ones (1-128MB optimal) — reduces list/get operations and improves query speed
- S3 Intelligent-Tiering: Automatically moves infrequently accessed data to cheaper tiers (no retrieval fees)
- Athena workgroups: Set per-query and per-workgroup data scan limits to prevent runaway costs
- Redshift Serverless: Pay per RPU-second — no idle cost for intermittent warehouse workloads
Exam Tips by Certification
| Exam | Focus Areas |
|---|---|
| SAP-C02 | Athena vs Redshift decision, Lake Formation cross-account sharing, data zone design, Kinesis vs Glue for ingestion, cost optimization (partitioning, columnar, lifecycle), DMS for CDC |
| MLA-C01 | Feature Store integration with data lake, Glue ETL for feature engineering, data quality with Glue Data Quality, SageMaker access to lake via Athena/Spark, training data in Parquet on S3 |
AWS Certification Exam Practice Questions
Question 1:
A company stores 50TB of data in S3 as CSV files. Data analysts run Athena queries daily but costs are high ($500/day) because queries scan the entire dataset. What TWO changes would MOST reduce Athena costs?
- Convert CSV to Parquet and partition by date
- Move data to Redshift and query there instead
- Enable S3 Transfer Acceleration for faster queries
- Add more Athena workgroups to distribute load
Show Answer
Answer: A – Parquet provides 3-5x compression AND columnar storage (only requested columns are scanned). Partitioning by date means queries with WHERE date=… only scan relevant partitions. Combined, these typically reduce Athena costs by 90%+. Transfer Acceleration is for uploads, not queries. Workgroups limit costs but don’t reduce scan volume.
Question 2:
A financial company needs different teams to access different columns of the same table. The compliance team needs all columns, while the analytics team should NOT see PII columns (SSN, email). The table has 200+ columns. What is the MOST scalable approach?
- Create separate copies of the table with PII columns removed for analytics
- Lake Formation column-level permissions with tag-based access control
- S3 bucket policies restricting access to specific prefixes per team
- Athena views that exclude PII columns for the analytics team
Show Answer
Answer: B – Lake Formation column-level permissions restrict which columns each principal can see without duplicating data. Tag-based access control (TBAC) scales by tagging PII columns with “sensitivity=high” and granting analytics team access only to columns without that tag. Adding new PII columns just requires tagging — no permission changes needed. Views work but must be maintained manually.
Question 3:
A company needs to continuously replicate changes from their on-premises Oracle database to S3 for analytics. They need ongoing synchronization (not just a one-time migration) with minimal impact on the source database. Which service provides this?
- AWS Glue with JDBC connection on a schedule
- AWS DMS with CDC (Change Data Capture) to S3 target
- Kinesis Data Firehose with custom producer
- AWS DataSync on a recurring schedule
Show Answer
Answer: B – AWS DMS with CDC continuously captures changes (inserts, updates, deletes) from the source database using log-based replication (minimal source impact) and writes them to S3 in Parquet/CSV format. It handles both the initial full load and ongoing replication. Glue JDBC would perform full table scans on schedule (high source impact). DataSync is for file transfers, not database replication.
Question 4:
A data engineering team needs to process 5TB of raw JSON data daily, apply complex transformations (joins across 10 tables, deduplication, schema evolution), and write results to S3 in Parquet. They want serverless with no cluster management. Which service fits?
- Athena CTAS (Create Table As Select)
- AWS Glue ETL with auto-scaling
- Amazon EMR Serverless
- Lambda with S3 batch operations
Show Answer
Answer: B – Glue ETL provides serverless Apache Spark with auto-scaling DPUs. It handles complex multi-table joins, deduplication, schema evolution (schema-on-read + schema merging), and writes Parquet natively. No cluster management required. Athena CTAS works for simpler SQL transforms but isn’t ideal for complex multi-step ETL. EMR Serverless also works but Glue is simpler for standard ETL patterns.
Question 5:
A company wants to share specific tables from their data lake with a partner AWS account without copying data. The partner should only see curated tables with certain rows filtered (e.g., only their region’s data). What is the recommended approach?
- Copy filtered data to an S3 bucket in the partner account nightly
- Lake Formation cross-account sharing with row-level security filters
- Redshift data sharing with the partner account
- S3 bucket policy granting partner account read access to specific prefixes
Show Answer
Answer: B – Lake Formation cross-account sharing grants the partner account access to specific tables with row-level filters (e.g., WHERE region=’partner_region’). No data copying, no duplication costs, permissions are centrally managed, and the partner queries via their own Athena/Redshift Spectrum. Data stays in your account; they get a filtered view.
Related Posts
- AWS S3 Storage Classes
- Kinesis Data Streams vs Firehose vs Managed Flink
- Amazon Athena
- AWS S3 vs EBS vs EFS – Storage Comparison
- AWS MLOps Pipeline Architecture
- AWS Event-Driven Serverless Architecture
References
- Building a Data Lake on AWS — AWS Whitepaper
- Design a Data Mesh Architecture Using Lake Formation and Glue — AWS Blog
- Build a Serverless Data Pipeline with Glue and Athena — AWS Blog
- Best Practices for Data Lake S3 Bucket Organization — AWS Prescriptive Guidance
- Reduce Cost and Improve Query Performance with Athena — AWS Blog
📖 Related: AWS EBS Volume Types – Performance & Cost Comparison
Frequently Asked Questions
Data lake vs data warehouse — when to use which?
A data lake (S3) stores raw data in any format at low cost — best for data exploration, ML, and diverse data types. A data warehouse (Redshift) stores structured, pre-modeled data optimized for fast, repeated analytical queries. Most organizations use both: lake for storage/exploration, warehouse for production dashboards and business intelligence.
AWS Glue vs EMR — when to use which?
Use Glue for standard ETL jobs where you want serverless, auto-scaling, and visual job authoring. Use EMR when you need custom Spark/Hadoop configurations, specific library versions, long-running clusters, or workloads exceeding Glue’s capabilities. Glue is simpler; EMR is more flexible and cost-effective at very large scale with spot instances.
What is Lake Formation vs Glue Data Catalog?
Glue Data Catalog is the metadata repository (table definitions, schemas, partitions). Lake Formation is the governance layer on top that adds fine-grained access control (row/column/cell-level), tag-based permissions, cross-account sharing, and data location registration. Lake Formation uses the Catalog underneath but adds security and governance.