AWS EMR & Spark Architecture — Overview
EMR is the largest topic on DAS-C01 (274 questions) and appears on DEA-C01 and MLA-C01 for data processing. The exam tests cluster modes (transient vs persistent), Spark optimization, EMRFS (S3 integration), instance selection, and EMR Serverless vs managed clusters.
Master + Core + Task nodes
Spot for Task nodes
HDFS on Core nodes
Bootstrap actions
Custom AMIs
Shared EKS cluster
Multi-tenant
Fine-grained resource
Container-based
Consolidate infra
Auto-scales workers
Pay per vCPU-hour used
Pre-initialized capacity
Spark, Hive, Presto
Submit jobs only
EMR on EC2 — Cluster Architecture
| Node Type | Role | Scaling | Instance Strategy |
|---|---|---|---|
| Primary (Master) | YARN ResourceManager, NameNode, job tracking | Single node (HA: 3 for multi-master) | On-Demand only (critical) |
| Core | HDFS DataNode + task execution | Can scale out (not in easily) | On-Demand or mix (stores HDFS data) |
| Task | Task execution only (no HDFS) | Freely scale in/out | Spot instances (no data loss risk) |
Transient vs Persistent Clusters
| Pattern | How | Best For |
|---|---|---|
| Transient | Launch cluster → run job → terminate. Data on S3 (EMRFS). | Batch ETL, scheduled jobs, cost optimization (pay only during execution) |
| Persistent | Long-running cluster. Interactive queries, notebooks. | Ad-hoc analysis, Jupyter notebooks, interactive Spark/Presto/Hive |
Exam rule: “Minimize cost for batch ETL” → transient cluster + S3 (EMRFS). “Interactive analysis” → persistent cluster.
EMRFS — S3 as Primary Storage
- What: EMR’s implementation of HDFS file system interface on top of S3
- Why: Decouple storage from compute. Cluster terminates, data persists in S3. Enables transient clusters.
- Consistent view: EMRFS provides read-after-write consistency (S3 now natively consistent)
- S3 Committer: EMRFS S3-optimized committer avoids the rename problem (faster job completion)
- vs HDFS: Use HDFS only for intermediate shuffle data or when sub-second latency needed. S3 for input/output data.
Spark Optimization
- Partitioning: Match parallelism to data size. Too few partitions = underutilized cluster. Too many = scheduling overhead.
- Memory: spark.executor.memory + spark.executor.memoryOverhead. OOM = increase memory or reduce partition size.
- Shuffle: Expensive operation (data crosses network). Reduce with: broadcast joins, coalesce, partition-wise operations.
- Dynamic Allocation: Automatically scales executors based on workload. Enable for variable job sizes.
- Instance Fleet: Mix instance types in a fleet (Spot diversification). EMR picks cheapest available. Reduces Spot interruptions.
EMR Serverless vs Glue vs EMR on EC2
| Feature | EMR on EC2 | EMR Serverless | Glue |
|---|---|---|---|
| Management | You manage cluster | Serverless (no cluster) | Serverless (no cluster) |
| Frameworks | Spark, Hive, Presto, HBase, Flink, Pig | Spark, Hive, Presto | Spark (PySpark), Python Shell |
| Custom config | Full (bootstrap, custom AMI, libs) | Limited (custom JARs, spark-submit args) | Limited (Glue libraries) |
| Data Catalog | Can use Glue Data Catalog as metastore | Glue Data Catalog | Native integration |
| Best for | Complex multi-framework, custom dependencies | Spark/Hive without cluster ops | ETL-focused, visual designer, bookmarks |
Exam Tips
| Exam | Key Points |
|---|---|
| DAS-C01 | “Cost-effective batch” → transient cluster + EMRFS (S3). “Spot instances” → Task nodes only (no HDFS data). “Interactive queries” → persistent cluster + Presto/Hive. “Cluster terminates data lost” → was using HDFS, switch to EMRFS. “Scale processing not storage” → EMRFS. “Spark OOM” → increase executor memory or reduce partition size. “Cheapest Spark” → EMR Serverless for variable loads. |
AWS Certification Exam Practice Questions
Question 1:
A company runs nightly Spark ETL jobs that process 2TB of data. The cluster is provisioned 24/7 but only runs jobs for 3 hours each night. How can they reduce costs by 80%+?
- Use Reserved Instances for the cluster
- Switch to transient cluster pattern — launch cluster before job, use EMRFS (S3) for data, terminate after job completes
- Reduce cluster size and run jobs longer
- Switch to EMR on EKS
Show Answer
Answer: B — Transient clusters run only during job execution (3 hours/day instead of 24). Store data in S3 (EMRFS), not HDFS — data persists after cluster terminates. Pay for 3 hours instead of 24 = 87.5% cost reduction. Add Spot instances for Task nodes for additional savings. This is the standard cost optimization pattern for batch ETL.
Question 2:
A Spark job on EMR occasionally fails because Spot instances in the Task node fleet are reclaimed. The job must complete reliably while still using Spot for cost savings. How should this be configured?
- Use On-Demand instances only (no Spot)
- Use Instance Fleets with multiple instance types for Task nodes (Spot diversification), keep Core nodes On-Demand, enable Spark checkpointing
- Increase the number of Task nodes to compensate for lost instances
- Use EMR Serverless instead
Show Answer
Answer: B — Instance Fleets with diverse instance types reduce Spot interruption probability (EMR picks from available pools). Core nodes On-Demand ensures HDFS/shuffle data isn’t lost. Spark checkpointing allows recovery from interrupted tasks without restarting entire job. This is the best-practice pattern for reliable Spot usage on EMR.
Question 3:
An analytics team needs to run interactive Presto queries on data in S3 alongside scheduled Spark ETL jobs. They want a single cluster for both workloads. Which EMR configuration supports this?
- Two separate transient clusters — one for Presto, one for Spark
- Persistent cluster with YARN capacity scheduler — allocate queues for Presto (interactive) and Spark (batch) with resource limits
- EMR Serverless for both workloads
- Glue for Spark + Athena for queries (no EMR needed)
Show Answer
Answer: B — Persistent cluster with YARN capacity scheduler allows resource sharing between workloads. Presto gets a dedicated queue with guaranteed resources (low-latency interactive). Spark batch jobs use remaining capacity. Both share the same cluster infrastructure and EMRFS data. Option D is valid but doesn’t use “a single cluster” as specified.
Related Posts
- Glue ETL & Data Pipeline
- Data Lake & Analytics Architecture
- Kinesis vs MSK Streaming
- Cost Optimization Architecture
References
Frequently Asked Questions
EMR vs Glue — when to use which?
Glue: Simpler, fully serverless, Data Catalog native, visual designer, job bookmarks. Best for ETL pipelines. EMR: Full Hadoop ecosystem (Spark, Hive, Presto, HBase, Flink), custom configurations, better for complex multi-framework workloads, interactive analysis. Use Glue for standard ETL. Use EMR when you need frameworks beyond Spark or custom cluster configuration.
When should I use EMRFS vs HDFS?
EMRFS (S3): Primary storage for input/output data. Survives cluster termination. Enables transient clusters. Use for most data. HDFS: Only for intermediate data needing lowest latency (shuffle data, temporary computation). HDFS is lost when cluster terminates. Modern best practice: EMRFS for everything, HDFS only as automatic intermediate storage.