AWS Glue ETL & Data Pipeline Architecture — Overview
AWS Glue is the core ETL service tested on DEA-C01 (34 questions) and MLA-C01 (36 questions). The exam tests Glue jobs (Spark/Python), Glue Data Catalog (metadata), crawlers (schema discovery), data quality, and orchestration patterns for data engineering pipelines.
S3, RDS, DynamoDB
JDBC, Kinesis
On-premises DBs
Auto-discover schema
Populate Data Catalog
Classify formats
Spark/Python/Ray
Transform, clean
DQ checks
S3 (Parquet/Iceberg)
Redshift, RDS
OpenSearch
Athena queries
Redshift analytics
QuickSight dashboards
Glue Components
| Component | What | Key Details |
|---|---|---|
| Data Catalog | Central metadata repository (databases, tables, schemas) | Shared across Athena, Redshift Spectrum, EMR, Lake Formation. Hive-compatible metastore. |
| Crawlers | Auto-discover data schema and populate Catalog | Runs on schedule or on-demand. Classifiers detect format (Parquet, CSV, JSON, Avro). |
| ETL Jobs (Spark) | Distributed data transformation using Apache Spark | Glue 4.0 (Spark 3.3). DPU-based pricing. Auto-scaling. Bookmarks for incremental processing. |
| ETL Jobs (Python Shell) | Lightweight Python scripts (no Spark overhead) | Cheaper for simple transforms. Max 1 DPU. Good for API calls, small files. |
| Data Quality | Define and monitor data quality rules | DQDL rules (completeness, uniqueness, freshness). Alert or stop pipeline on failure. |
| Glue Studio | Visual ETL job designer (no-code/low-code) | Drag-and-drop transforms. Generates Spark code. Good for data engineers. |
| Glue DataBrew | Visual data preparation (profiling, cleaning) | 250+ transforms without code. Profile datasets for quality. Recipes for reuse. |
Job Bookmarks — Incremental Processing
- Problem: Daily ETL job should only process NEW data (not re-process everything)
- Solution: Job bookmarks track what was already processed (S3 paths, JDBC timestamps)
- How: Enable bookmark on job. Glue remembers last processed file/row. Next run starts from where it left off.
- Exam note: “Process only new files in S3” → Enable Glue job bookmarks
Glue Connections & VPC
- JDBC sources: Glue jobs connect to RDS/Redshift via JDBC connections (require VPC, subnet, SG configuration)
- ENI: Glue creates ENIs in your VPC subnet to access private data sources
- NAT Gateway: If Glue job needs internet access (e.g., API call) from private subnet, requires NAT Gateway
- S3 endpoint: Always add VPC Gateway Endpoint for S3 (Glue reads/writes S3 heavily — avoid NAT charges)
Orchestration Patterns
| Orchestrator | Best For | Key Feature |
|---|---|---|
| Glue Workflows | Simple Glue-only pipelines (crawler → job → crawler) | Built-in, triggers on schedule/event/condition |
| Step Functions | Complex multi-service pipelines with error handling | Visual workflow, retry/catch, parallel execution, human approval |
| MWAA (Airflow) | Teams already using Airflow, complex DAG dependencies | Managed Apache Airflow, Python DAGs, rich ecosystem |
| EventBridge Scheduler | Simple scheduled triggers (cron/rate) | Serverless, triggers Glue/Step Functions/Lambda on schedule |
Performance Optimization
- Partitioning: Partition output by date/region/category. Athena/Redshift Spectrum prunes partitions for faster queries.
- File format: Convert CSV/JSON → Parquet/ORC (columnar, compressed, 10-100x faster queries)
- File size: Avoid small files (coalesce to 128MB-1GB per file). Small files kill query performance.
- Auto-scaling: Glue 4.0 auto-scales DPUs based on workload (no over-provisioning)
- Pushdown predicates: Filter data at source (pushdown to JDBC, S3 Select) — less data to process
Exam Tips
| Exam | Key Points |
|---|---|
| DEA-C01 | “Discover schema” → Crawler. “Process only new files” → Job Bookmarks. “Convert CSV to Parquet” → Glue ETL job. “Central metadata” → Data Catalog. “Visual ETL” → Glue Studio. “Data quality rules” → Glue Data Quality (DQDL). “Orchestrate pipeline” → Step Functions (complex) or Glue Workflows (simple). “Access RDS from Glue” → JDBC connection in VPC. |
AWS Certification Exam Practice Questions
Question 1:
A company receives daily CSV files in S3. They need to transform them to Parquet format, partition by date, and make them queryable by Athena. The pipeline should only process new files each day. Which solution achieves this?
- Lambda triggered by S3 event → convert each file individually
- Glue ETL job with bookmarks enabled, scheduled daily. Reads new CSVs, transforms to Parquet, writes partitioned output. Crawler updates Catalog.
- Athena CTAS query to convert CSV to Parquet
- EMR Spark cluster running 24/7 processing files as they arrive
Show Answer
Answer: B — Glue ETL job handles the transformation (CSV → Parquet with partitioning). Job bookmarks ensure only new files are processed each day (no reprocessing). Crawler updates the Data Catalog so Athena sees new partitions. Serverless — no infrastructure to manage. Lambda (A) works for small files but doesn’t handle large-scale transformations well. EMR (D) is over-provisioned for a daily batch job.
Question 2:
A data team needs a central metadata repository that Athena, Redshift Spectrum, and EMR can all query against. They want schema discovered automatically from S3 data. Which combination provides this?
- Custom Hive metastore on EC2 with manual schema creation
- Glue Data Catalog populated by Glue Crawlers — automatically discovered schemas accessible by Athena, Redshift Spectrum, and EMR
- S3 bucket tags as metadata with Lambda reading them
- DynamoDB table storing schema information
Show Answer
Answer: B — Glue Data Catalog is the central metastore that integrates natively with Athena (uses it by default), Redshift Spectrum (external schema), and EMR (as Hive-compatible metastore). Crawlers auto-discover schemas from S3 data. No manual schema management needed. This is the AWS-native data catalog solution — replaces external Hive metastore.
Question 3:
A Glue ETL job needs to read from an RDS PostgreSQL database in a private subnet. The job also writes output to S3. The job fails with connection timeout errors to both RDS and S3. What is the most likely configuration issue?
- Glue job doesn’t have IAM permissions to access RDS
- Glue connection not configured with VPC, subnet, and security group. Also missing S3 VPC Gateway Endpoint.
- RDS instance is encrypted and Glue can’t decrypt
- Glue job DPU count is too low for the data volume
Show Answer
Answer: B — Glue jobs accessing private resources need: (1) Connection with VPC, subnet, and SG configured (creates ENI in your VPC). (2) SG must allow Glue ENI → RDS on port 5432. (3) S3 VPC Gateway Endpoint needed because Glue in a private subnet can’t reach S3 over the internet. This is the #1 Glue troubleshooting question on the exam. Without VPC config, Glue can’t reach private resources.
Question 4:
A data pipeline produces Parquet files in S3 that are later queried by Athena. Queries are slow because the pipeline creates thousands of small files (1-5 MB each). How should the pipeline be optimized?
- Switch from Parquet to CSV for faster reading
- Configure the Glue job to coalesce output files to 128MB-1GB using repartition/coalesce, and partition output by date for pruning
- Increase Athena query timeout to allow more time
- Add more Glue DPUs to write faster
Show Answer
Answer: B — Small files are the #1 performance killer for S3-based analytics. Each file requires a separate S3 GET request and has metadata overhead. Coalescing to 128MB-1GB optimal size + date partitioning allows Athena to: (1) read fewer, larger files (less I/O overhead), and (2) skip irrelevant partitions. CSV (A) is worse than Parquet. More DPUs (D) don’t fix the output file size issue.
Question 5:
A data team needs to validate that incoming data meets quality rules (no null values in required fields, dates in correct format, values within expected ranges) before loading into the data warehouse. If rules fail, the pipeline should stop and alert the team. Which Glue feature provides this?
- Glue Crawler with classification rules
- Glue Data Quality with DQDL rules configured to halt the job on failure and send SNS notification
- Custom Lambda function that samples and validates data before Glue runs
- Athena query that checks constraints after loading
Show Answer
Answer: B — Glue Data Quality uses DQDL (Data Quality Definition Language) to define rules: Completeness, Uniqueness, Freshness, ColumnValues ranges. Rules run within the ETL job. On failure: stop pipeline (prevent bad data loading) + trigger CloudWatch alarm → SNS notification. Built-in, no custom code needed. Athena (D) is post-load (too late). Lambda (C) requires custom implementation.
Related Posts
- Data Lake & Analytics Architecture
- Kinesis vs MSK Streaming
- RAG Advanced Patterns (Vector DB)
- Event-Driven Serverless Architecture
References
Frequently Asked Questions
Glue ETL vs EMR — when to use which?
Glue: Serverless, auto-scaling, pay per DPU-hour. Best for scheduled ETL jobs, data catalog integration, and teams wanting managed Spark without cluster management. EMR: Full Hadoop/Spark ecosystem, persistent or transient clusters, custom configurations (Hive, Presto, HBase). Best for complex big data workloads needing fine-tuned Spark configs, long-running clusters, or non-Spark frameworks.
What is the Glue Data Catalog used for?
Central metadata repository storing table definitions (schema, location, format, partitions). Shared across AWS analytics services: Athena queries tables from the Catalog, Redshift Spectrum references external tables, EMR uses it as Hive metastore, Lake Formation controls access to Catalog tables. It’s the “phone book” for your data lake — services look up where data lives and what format it’s in.