HVM

AWS EC2 Network – Enhanced Networking

~ Last updated on : ~ jayendrapatil ~ 9 Comments

EC2 Enhanced Networking

  • Enhanced networking results in higher bandwidth, higher packet per second (PPS) performance, lower latency, consistency, scalability and lower jitter
  • EC2 provides enhanced networking capabilities using single root I/O virtualization (SR-IOV) only on supported instance types
    • SR-IOV is a method of device virtualization that provides higher I/O performance and lower CPU utilization
  • There is no additional charge for using enhanced networking.
  • Enhanced networking is supported only in a VPC.
  • All current-generation instances built on the AWS Nitro System use ENA for enhanced networking by default.
  • Amazon Linux AMIs, Ubuntu HVM AMIs, and Windows Server AMIs already have the ENA module installed with the attributes set and do not require any additional configurations.
  • It can be enabled for other OS distributions by installing the module with the correct attributes configured
  • Enhanced Networking is supported using
    • Elastic Network Adapter (ENA)
      • The Elastic Network Adapter (ENA) supports network speeds of up to 200 Gbps for supported instance types (e.g., C6in, R6in, M6in instances). Some accelerated instances like P4d support up to 400 Gbps.
      • All Nitro-based instances use ENA for enhanced networking.
      • The following Xen-based instances also use ENA: H1, I3, G3, m4.16xlarge, P3, P3dn, and R4.
      • ENA is the recommended and standard adapter for all current-generation workloads.
    • Intel 82599 Virtual Function (VF) interface
      • The Intel 82599 Virtual Function interface supports network speeds of up to 10 Gbps for supported instance types.
      • Supported only on previous-generation instance types: C3, C4, D2, I2, M4 (excl. m4.16xlarge), and R3.
      • These are all previous-generation instances. AWS recommends migrating to current-generation Nitro-based instances with ENA for better performance.

ENA Express

  • ENA Express is powered by AWS Scalable Reliable Datagram (SRD) technology, a high-performance network transport protocol.
  • ENA Express increases the maximum single flow bandwidth from 5 Gbps up to 25 Gbps within the same Region, up to the aggregate instance limit.
  • Reduces tail latency: up to 50% reduction in P99 latency and up to 85% reduction in P99.9 latency compared to TCP.
  • Works transparently with existing TCP and UDP applications — no code changes required.
  • SRD distributes packets across different network paths and dynamically adjusts when congestion is detected.
  • Handles packet reordering on the receiving end and most retransmits in the network layer.
  • Cross-AZ support (May 2026): ENA Express now supports traffic between instances in different Availability Zones within the same Region, delivering up to 25 Gbps single-flow bandwidth.
  • Requirements:
    • Both sending and receiving instances must be supported instance types.
    • Both instances must have ENA Express enabled on their network interface attachment.
    • The network path must not include middleware boxes.
    • Linux instances require ENA driver version 2.2.9 or higher for full bandwidth; version 2.8+ for metrics.
  • ENA Express is available on supported 6th generation and later instance types (e.g., m6i, m6a, c6i, r6i, and newer).
  • If ENA Express is not supported on both ends, communication falls back to standard ENA transmission.
  • Note: For workloads requiring high packets-per-second with lowest latency during uncongested periods, standard enhanced networking (without ENA Express) may be more appropriate.

Elastic Fabric Adapter (EFA)

  • An Elastic Fabric Adapter (EFA) is a network device for Amazon EC2 instances to accelerate AI/ML, and High Performance Computing (HPC) applications.
  • EFA provides lower and more consistent latency and higher throughput than TCP transport for inter-instance communication.
  • Supports Message Passing Interface (MPI) for HPC and NVIDIA Collective Communications Library (NCCL) for ML workloads, scaling to thousands of cores or GPUs.
  • Available as an optional EC2 networking feature at no additional cost on supported instance types.
  • EFA uses OS-bypass capabilities to provide low-latency, high-bandwidth RDMA-like networking.
  • EFA decoupled from ENA (October 2024): AWS introduced a new interface type that decouples EFA from ENA, enabling dedicated high-bandwidth, low-latency networking crucial for scaling AI/ML workloads.
  • EFA-only interfaces (June 2026): Amazon SageMaker HyperPod supports EFA-only network interfaces without ENA for IP networking, enabling dedicated accelerator networking.
  • Supported on instances like P4d (400 Gbps), P5, Trn1, Trn2, Hpc6a, Hpc7a, Hpc7g (200 Gbps), and others.
  • EFA is ideal for tightly coupled workloads requiring high internode communication bandwidth.

ENA Enhanced Networking Requirements

  • Instance must be in a VPC (EC2-Classic was fully retired in August 2023)
  • An HVM virtualization type AMI
  • Instance must be based on the Nitro System (for current-generation instances)
  • For Xen-based instances (H1, I3, G3, m4.16xlarge, P3, R4): must have ENA module installed and enaSupport attribute enabled
  • Supported instance types: All Nitro-based instances (5th generation and later: C5, M5, R5, C6i, M6i, R6i, C7g, M7g, R7g, C8g, M8g, etc.)
  • Enhanced networking cannot be managed from the Amazon EC2 console — use AWS CLI or CloudShell

Intel 82599 VF Enhanced Networking Requirements (Previous Generation)

  • VPC (EC2-Classic was fully retired in August 2023)
  • An HVM virtualization type AMI
  • Instance kernel version
    • Linux kernel version of 2.6.32+
    • Windows: Server 2008 R2+
  • Appropriate Virtual Function (VF) driver
    • Linux – should have the ixgbevf module installed and that sriovNetSupport attribute set for the instance
    • Windows – Intel 82599 Virtual Function driver
  • Supported instance types (previous generation only): C3, C4, D2, I2, M4 (excl. m4.16xlarge), and R3.
  • Note: AWS recommends migrating to current-generation Nitro-based instances with ENA for significantly better networking performance (up to 200 Gbps vs. 10 Gbps).

Enhanced Networking vs. ENA Express vs. EFA

  • Enhanced Networking (ENA/VF): Higher PPS, lower latency, lower jitter using SR-IOV. Available on all Nitro instances. Best for general workloads requiring consistent network performance.
  • ENA Express: Uses SRD protocol on top of ENA. Increases single-flow bandwidth to 25 Gbps and significantly reduces tail latency. Best for workloads with large data transfers or latency-sensitive applications. Available on 6th gen+ instances.
  • Elastic Fabric Adapter (EFA): Network device providing OS-bypass RDMA-like capabilities. Best for HPC (MPI) and AI/ML (NCCL) workloads requiring ultra-low latency inter-node communication. Available on specific compute/GPU instances.

AWS Certification Exam Practice Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  1. You have multiple Amazon EC2 instances running in a cluster across multiple Availability Zones within the same region. What combination of the following should be used to ensure the highest network performance (packets per second), lowest latency, and lowest jitter? Choose 3 answers
    1. Amazon EC2 placement groups (Cluster placement groups are within a single AZ, would not work for multiple AZs)
    2. Enhanced networking (provides network performance, lowest latency)
    3. Amazon PV AMI (Requires HVM)
    4. Amazon HVM AMI (Requires HVM)
    5. Amazon Linux (Can be on others as well)
    6. Amazon VPC (works only in VPC; EC2-Classic was retired August 2023)
  2. A group of researchers is studying the migration pattern of a beetle that eats and destroys grain. The researchers must process massive amounts of data and run statistics. Which one of the following options provides the high performance computing for this purpose.
    1. Configure an Autoscaling Scaling group to launch dozens of spot instances to run the statistical analysis simultaneously
    2. Launch AMI instances that support SR-IOV in a single Availability Zone
    3. Launch compute optimized (C4) instances in at least two Availability Zones
    4. Launch enhanced network type instances in a placement group
  3. A company is running a latency-sensitive financial trading application on EC2 instances. They need to maximize single-flow bandwidth between two instances in the same Availability Zone. Which feature should they enable?
    1. Enhanced networking with Intel 82599 VF
    2. Elastic Fabric Adapter (EFA)
    3. ENA Express (ENA Express uses SRD to increase single-flow bandwidth from 5 Gbps to 25 Gbps and reduces tail latency)
    4. Placement group with standard ENA
  4. A machine learning team needs to scale their distributed training workload across hundreds of GPU instances with the lowest possible inter-node latency. Which networking feature is most appropriate?
    1. ENA Express with SRD protocol
    2. Enhanced networking with cluster placement groups
    3. Elastic Fabric Adapter (EFA) (EFA provides OS-bypass, RDMA-like capabilities optimized for MPI and NCCL workloads at scale)
    4. Multiple Elastic Network Interfaces
  5. Which of the following statements about ENA Express are correct? (Choose 2)
    1. ENA Express uses AWS Scalable Reliable Datagram (SRD) protocol to improve network performance (Correct – SRD is the underlying protocol)
    2. ENA Express requires application code changes to work
    3. ENA Express only works with TCP traffic
    4. ENA Express can increase single-flow bandwidth from 5 Gbps up to 25 Gbps (Correct – major benefit of ENA Express)
  6. A company wants to migrate from C3 instances to improve network performance. Which statement is correct regarding the migration?
    1. C3 instances support ENA with speeds up to 100 Gbps
    2. C3 instances use Intel 82599 VF (up to 10 Gbps) and should be migrated to current-generation Nitro instances with ENA for up to 200 Gbps (C3 is previous gen with VF; current gen instances offer significantly better networking)
    3. C3 instances cannot use enhanced networking
    4. C3 instances already support ENA Express

References