AWS Network Performance Architecture — Overview
Network performance is the #1 topic on ANS-C01/C02 (160+ questions) and appears on SAP-C02 for HPC and high-throughput scenarios. The exam tests Enhanced Networking (ENA/EFA), placement groups, jumbo frames, bandwidth allocation, and network optimization patterns.
EFA: HPC/ML inter-node
SR-IOV: Bypass hypervisor
Intel 82599 VF (legacy)
Spread: HA (distinct hardware)
Partition: Large distributed (HDFS, Kafka)
Jumbo: 9001 MTU
Within VPC/peering/DX
NOT over internet/VPN
Baseline vs burst
Multi-flow: aggregate BW
Single-flow: 5-10 Gbps cap
Enhanced Networking — ENA vs EFA
| Feature | ENA (Elastic Network Adapter) | EFA (Elastic Fabric Adapter) |
|---|---|---|
| Purpose | High-performance general networking | HPC inter-node communication (MPI, NCCL) |
| Speed | Up to 200 Gbps | Up to 200 Gbps + OS-bypass |
| Latency | Low (microseconds) | Ultra-low (bypasses OS kernel for node-to-node) |
| Protocol | TCP/UDP | Libfabric (MPI, NCCL) + TCP/UDP |
| Use case | All workloads (enabled by default on Nitro) | Tightly-coupled HPC (weather modeling, CFD), distributed ML training |
| Placement | Any | Cluster placement group required for best performance |
Placement Groups
| Type | How | Use Case | Limitation |
|---|---|---|---|
| Cluster | Instances on same rack, single AZ | HPC, low-latency node-to-node (10 Gbps between instances) | Single AZ = single point of failure. Capacity errors if rack full. |
| Spread | Each instance on distinct hardware (max 7 per AZ) | Critical instances that must not fail together (HA for small groups) | Max 7 instances per AZ per group. |
| Partition | Instances grouped into partitions on separate racks (up to 7 partitions/AZ) | Large distributed systems (HDFS, HBase, Kafka, Cassandra) | Topology-aware apps needed to leverage partition isolation. |
Jumbo Frames (MTU 9001)
- What: Larger Ethernet frames (9001 bytes vs standard 1500 bytes). Less overhead per byte transferred → higher throughput.
- Supported: Within VPC, across VPC peering (same region), over Direct Connect, between instances in same placement group
- NOT supported: Over the internet, over VPN, over TGW (TGW reduces to 8500 MTU), across inter-region peering
- Path MTU Discovery: ICMP “Fragmentation Needed” messages. If blocked by NACL/SG, causes packet drops (exam trap!)
- Configuration: Set on OS level (Linux:
ip link set dev eth0 mtu 9001). Both sender AND receiver must support.
Instance Bandwidth
- Determined by instance type: t3.micro = 5 Gbps burst, c5.18xlarge = 25 Gbps, c5n.18xlarge = 100 Gbps
- Single-flow limit: Single TCP connection maxes at ~5-10 Gbps (use multi-flow/parallel connections for full bandwidth)
- Baseline vs burst: Smaller instances have baseline + burst (like EBS bandwidth). Larger instances have consistent bandwidth.
- “n” instances: c5n, m5n, r5n = network-optimized variants with higher bandwidth (100 Gbps)
- Cross-AZ: ~$0.01/GB data transfer charge. Same performance as intra-AZ but costs money.
Network Optimization Patterns
| Problem | Solution |
|---|---|
| HPC needs ultra-low latency between nodes | EFA + Cluster placement group + jumbo frames |
| Single TCP flow not reaching instance bandwidth | Use multiple parallel TCP connections (multi-flow aggregates) |
| Packet drops on large transfers | Check MTU mismatch (Path MTU Discovery), enable jumbo frames end-to-end |
| Distributed DB needs fault isolation | Partition placement group (HDFS, Kafka rack-aware) |
| Critical services can’t share hardware failure | Spread placement group (max 7/AZ) |
Exam Tips
| Exam | Key Points |
|---|---|
| ANS-C01 | “Lowest latency between instances” → Cluster placement group + ENA. “HPC MPI workload” → EFA + Cluster placement. “Jumbo frames not working” → check TGW (8500), VPN (1500), internet (1500). “HDFS rack awareness” → Partition placement. “Max network bandwidth” → check instance type (“n” variant). “Single flow limited” → use multiple parallel connections. “PMTUD blocked” → check NACL allows ICMP type 3 code 4. |
AWS Certification Exam Practice Questions
Question 1:
A company runs an HPC simulation that requires ultra-low latency communication between 100 compute instances using MPI. The instances need to communicate at the highest possible bandwidth with OS-bypass for inter-node traffic. Which configuration provides this?
- ENA-enabled instances in a spread placement group with jumbo frames
- EFA-enabled instances in a cluster placement group with jumbo frames (MTU 9001)
- ENA-enabled instances with enhanced networking across 3 AZs
- Standard networking with NLB distributing traffic between instances
Show Answer
Answer: B — EFA provides OS-bypass for MPI/NCCL (libfabric), eliminating kernel overhead for inter-node communication. Cluster placement group places all instances on the same rack for lowest latency and highest bandwidth (up to 200 Gbps between instances). Jumbo frames reduce per-packet overhead. ENA (A) doesn’t provide OS-bypass for MPI. Spread (A) separates instances — opposite of what HPC needs.
Question 2:
Instances in a VPC communicate with jumbo frames (MTU 9001). After adding a Transit Gateway to route traffic between VPCs, large packet transfers start failing with fragmentation. What is the cause and fix?
- TGW doesn’t support jumbo frames — reduce MTU to 1500 on all instances
- TGW supports 8500 byte MTU (not 9001). Enable Path MTU Discovery and ensure NACLs allow ICMP type 3 code 4
- Enable jumbo frame support on the TGW attachment
- Increase the TGW bandwidth allocation
Show Answer
Answer: B — Transit Gateway supports a maximum MTU of 8500 bytes (not 9001). When instances send 9001-byte frames through TGW, packets need fragmentation. Path MTU Discovery (PMTUD) sends ICMP “Fragmentation Needed” messages back to the sender to reduce MTU. If NACLs block ICMP type 3 code 4, the sender never learns to reduce size → packets silently drop. Fix: allow ICMP in NACLs, or set instance MTU to 8500 for TGW paths.
Question 3:
A company’s c5.xlarge instances are achieving only 5 Gbps throughput when transferring large files between instances, despite the instance type supporting 10 Gbps. The transfer uses a single TCP connection. How can they achieve full 10 Gbps?
- Enable enhanced networking (ENA) on the instances
- Use multiple parallel TCP connections (multi-flow) to aggregate bandwidth beyond the single-flow limit
- Move instances to a cluster placement group
- Increase instance type to c5.2xlarge
Show Answer
Answer: B — Single TCP flow (5-tuple: src IP, dst IP, src port, dst port, protocol) is limited to ~5 Gbps on AWS. To reach the instance’s full bandwidth (10 Gbps), use multiple parallel connections (different source ports = different flows). Tools like iperf3 with -P flag or multi-threaded file transfer. ENA (A) is already enabled on c5 by default. Placement group (C) helps latency, not single-flow bandwidth.
Question 4:
A distributed Kafka cluster on EC2 needs rack-awareness for partition replicas (replicas should be on different physical racks for fault tolerance). The cluster has 30 brokers across 3 AZs. Which placement strategy provides rack isolation information to Kafka?
- Cluster placement group (all on same rack)
- Spread placement group (max 7 per AZ — insufficient for 10 per AZ)
- Partition placement group — Kafka maps partitions to placement partitions for rack-awareness
- No placement group — use AZ as rack identifier
Show Answer
Answer: C — Partition placement groups provide up to 7 partitions per AZ. Each partition runs on a separate rack. Kafka (or HDFS) uses the partition number as the “rack” identifier for replica placement. This ensures replicas are on physically separate racks. Spread (B) maxes at 7 instances per AZ total — not enough for 10 brokers/AZ. Cluster (A) puts everything on ONE rack (worst for HA).
Question 5:
A company’s application transfers data between instances in the same VPC and also to instances in a peered VPC in another region. They want to use jumbo frames (MTU 9001) for maximum throughput. Where will jumbo frames work?
- Both intra-VPC and inter-region peering support MTU 9001
- Only intra-VPC supports MTU 9001. Inter-region peering is limited to MTU 1500.
- Neither supports jumbo frames without special configuration
- Both work but require enabling jumbo frames on the peering connection
Show Answer
Answer: B — Jumbo frames (9001 MTU) work within a VPC and across same-region VPC peering. Inter-region VPC peering reduces MTU to 1500 bytes. Traffic over the internet, VPN, or inter-region peering does NOT support jumbo frames. If you set 9001 MTU and send cross-region, packets will be fragmented (or dropped if DF bit set without PMTUD). Always verify MTU support per path.
Related Posts
- Hybrid Cloud Networking
- VPC Connectivity Decision Guide
- Auto Scaling & Performance
- Container Platform Architecture
References
- Enhanced Networking on EC2 — AWS Docs
- Elastic Fabric Adapter (EFA) — AWS Docs
- Placement Groups — AWS Docs
- Network MTU — AWS Docs
Frequently Asked Questions
ENA vs EFA — when do I need EFA?
ENA is sufficient for 99% of workloads (web, databases, containers, general compute). It provides up to 200 Gbps with low latency. EFA adds OS-bypass (kernel bypass) specifically for tightly-coupled parallel computing: MPI-based HPC simulations, distributed ML training (NCCL). If your workload doesn’t use MPI or NCCL, you don’t need EFA. EFA instances also have ENA for standard traffic.
Why does jumbo frame performance matter?
Standard frames carry 1500 bytes per packet with ~40 bytes header overhead (2.6%). Jumbo frames carry 9001 bytes with the same ~40 bytes overhead (0.4%). For large data transfers, jumbo frames: reduce CPU interrupt rate (fewer packets per GB), reduce header overhead (more payload per packet), and increase throughput. Most impactful for bulk data transfer, database replication, and storage traffic.