Google Kubernetes Engine Networking

Google Kubernetes Engine – Networking

IP allocation

Kubernetes uses various IP ranges to assign IP addresses to Nodes, Pods, and Services.

  • Node IP
    • Each node has an IP address assigned from the cluster’s VPC network.
    • Node IP provides connectivity from control components like kube-proxy and kubelet to the Kubernetes API server.
    • Node IP is the node’s connection to the rest of the cluster.
  • Pod CIDR or Address Range
    • Each node has a pool of IP addresses that GKE assigns the Pods running on that node (a /24 CIDR block by default).
  • Pod Address
    • Each Pod has a single IP address assigned from the Pod CIDR range of its node.
    • Pod IP address is shared by all containers running within the Pod and connects them to other Pods running in the cluster.
  • Service Address Range
    • Each Service has an IP address, called the ClusterIP, assigned from the cluster’s VPC network.
  • For Standard clusters
    • a maximum of 110 Pods can run on a node with a /24 range, not 256 as you might expect. This provides a buffer so that Pods don’t become unschedulable due to a transient lack of IP addresses in the Pod IP range for a given node.
    • For ranges smaller than /24, roughly half as many Pods can be scheduled as IP addresses in the range.
  • Autopilot clusters can run a maximum of 32 Pods per node.

GKE Cluster Networking Types

  • GKE, clusters can be distinguished according to the way they route traffic from one Pod to another Pod.
    • VPC-native cluster: A cluster that uses alias IP address ranges
    • Routes-based cluster: A cluster that uses custom static routes in a VPC network

VPC-Native Clusters

  • VPC-native cluster uses alias IP address ranges
  • VPC-native is the recommended network mode for new clusters
  • VPC-native clusters have several benefits:
    • Pod IP addresses are natively routable within the cluster’s VPC network and other VPC networks connected to it by VPC Network Peering.
    • Pod IP address ranges, and subnet secondary IP address ranges in general, are accessible from on-premises networks connected with Cloud VPN or Cloud Interconnect using Cloud Routers.
    • Pod IP addresses are reserved in the VPC network before the Pods are created in the cluster. This prevents conflict with other resources in the VPC network and allows you to better plan IP address allocations.
    • Pod IP address ranges do not depend on custom static routes and do not consume the system-generated and custom static routes quota. Instead, automatically generated subnet routes handle routing for VPC-native clusters.
    • Firewall rules can be created that apply to just Pod IP address ranges instead of any IP address on the cluster’s nodes.

VPC-Native Clusters IP Allocation

Google Kubernetes Engine Networking VPC-Native Cluster IP Management

  • VPC-native cluster uses three unique subnet IP address ranges
    • Subnet’s primary IP address range for all node IP addresses.
      • Node IP addresses are assigned from the primary IP address range of the subnet associated with the cluster.
      • Both node IP addresses and the size of the subnet’s secondary IP address range for Pods limit the number of nodes that a cluster can support
    • One secondary IP address range for all Pod IP addresses.
      • Pod IP addresses are taken from the cluster subnet’s secondary IP address range for Pods.
      • By default, GKE allocates a /24 alias IP range (256 addresses) to each node for the Pods running on it.
      • On each node, those 256 alias IP addresses support up to 110 Pods.
      • Pod Address Range cannot be changed. If exhausted,
        • a new cluster with a larger Pod address range must be created or
        • node pools should be recreated after decreasing the --max-pods-per-node for the node pools.
    • Another secondary IP address range for all Service (cluster IP) addresses.
      • Service (cluster IP) addresses are taken from the cluster’s subnet’s secondary IP address range for Services.
      • Service address range should be large enough to provide addresses for all the Kubernetes Services hosted in the cluster.
  • Node, Pod, and Services IP address ranges must all be unique and subnets with overlapping primary and secondary IP addresses cannot be created.

Routes-based Cluster

  • Routes-based cluster that uses custom static routes in a VPC network i.e. it uses Google Cloud Routes to route traffic between nodes
  • In a routes-based cluster,
    • each node is allocated a /24 range of IP addresses for Pods.
    • With a /24 range, there are 256 addresses, but the maximum number of Pods per node is 110.
    • With approximately twice as many available IP addresses as possible Pods, Kubernetes is able to mitigate IP address reuse as Pods are added to and removed from a node.
  • Routes-based cluster uses two unique subnet IP address ranges
    • Subnet’s primary IP address range for all node IP addresses.
      • Node IP addresses are taken from the primary range of the cluster subnet
      • Cluster subnet must be large enough to hold the total number of nodes in your cluster.
    • Pod address range
      • A routes-based cluster has a range of IP addresses that are used for Pods and Services
      • Last /20 (4096 addresses) of the Pod address range is used for Services and the rest of the range is used for Pods
      • Pod address range size cannot be changed after cluster creation. So ensure that a large enough Pod address range is chosen to accommodate the cluster’s anticipated growth during cluster creation
  • Maximum number of nodes, Pods, and Services for a given GKE cluster is determined by the size of the cluster subnet and the size of the Pod address range.

Related Reads

GCP Certification Exam Practice Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • GCP services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • GCP exam questions are not updated to keep up the pace with GCP updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.


AWS Certified Advanced Networking – Speciality (ANS-C00) Exam Learning Path

AWS Certified Advanced Networking – Speciality (ANS-C00) Exam Learning Path

I recently cleared the AWS Certified Advanced Networking – Speciality (ANS-C00), which was my first, en route my path to the AWS Speciality certifications. Frankly, I feel the time I gave for preparation was still not enough, but I just about managed to get through. So a word of caution, this exam is inline or tougher than the professional exam especially for the reason that the Networking concepts it covers are not something you can get your hands dirty with easily.

AWS Certified Advanced Networking – Speciality (ANS-C00) exam is the focusing on the AWS Networking concepts. It basically validates

  • Design, develop, and deploy cloud-based solutions using AWS
    Implement core AWS services according to basic architecture best practices
  • Design and maintain network architecture for all AWS services
  • Leverage tools to automate AWS networking tasks

Refer to AWS Certified Advanced Networking – Speciality Exam Guide

AWS Certified Advanced Networking – Speciality (ANS-C00) Exam Resources

AWS Certified Advanced Networking – Speciality (ANS-C00) Exam Summary

  • AWS Certified Advanced Networking – Speciality exam covers a lot of Networking concepts like VPC, VPN, Direct Connect, Route 53, ALB, NLB.
  • One of the key tactic I followed when solving the DevOps Engineer questions was to read the question and use paper and pencil to draw a rough architecture and focus on the areas that you need to improve. Trust me, you will be able eliminate 2 answers for sure and then need to focus on only the other two. Read the other 2 answers to check the difference area and that would help you reach to the right answer or atleast have a 50% chance of getting it right.
  • Be sure to cover the following topics
    • Networking & Content Delivery
      • You should know everything in Networking.
      • Understand VPC in depth
      • Virtual Private Network to establish connectivity between on-premises data center and AWS VPC
      • Direct Connect to establish connectivity between on-premises data center and AWS VPC and Public Services
        • Make sure you understand Direct Connect in detail, without this you cannot clear the exam
        • Understand Direct Connect connections – Dedicated and Hosted connections
        • Understand how to create a Direct Connect connection (hint: LOA-CFA provides the details for partner to connect to AWS Direct Connect location)
        • Understand virtual interfaces options – Private Virtual Interface for VPC resources and Public Virtual Interface for Public resources
        • Understand setup Private and Public VIF
        • Understand Route Propagation, propagation priority, BGP connectivity
        • Understand High Availability options based on cost and time i.e. Second Direct Connect connection OR VPN connection
        • Understand Direct Connect Gateway – it provides a way to connect to multiple VPCs from on-premises data center using the same Direct Connect connection
      • Route 53
        • Understand Route 53 and Routing Policies and their use cases Focus on Weighted, Latency routing policies
        • Understand Route 53 Split View DNS to have the same DNS to access a site externally and internally
      • Understand CloudFront and use cases
      • Load Balancer
        • Understand ELB, ALB and NLB 
        • Understand the difference ELB, ALB and NLB esp. ALB provides Content, Host and Path based Routing while NLB provides the ability to have static IP address
        • Know how to design VPC CIDR block with NLB (Hint – minimum number of IPs required are 8)
        • Know how to pass original Client IP to the backend instances (Hint – X-Forwarded-for and Proxy Protocol)
      • Know WorkSpaces requirements and setup
    • Security
      • Know AWS GuardDuty as managed threat detection service
      • Know AWS Shield esp. the Shield Advanced option and the features it provides
      • Know WAF as Web Traffic Firewall – (Hint – WAF can be attached to your CloudFront, Application Load Balancer, API Gateway to dynamically detect and prevent attacks)

AWS Network Connectivity Options


More details @ Virtual Private Cloud

Internet Gateway

  • An Internet Gateway provides Internet connectivity to VPC
  • Internet gateway is a horizontally scaled, redundant, and highly available VPC component that allows communication between instances in your VPC and the internet.
  • Internet Gateway imposes no availability risks or bandwidth constraints on your network traffic.
  • An Internet gateway serves two purposes: to provide a target in the VPC route tables for internet-routable traffic, and to perform network address translation (NAT) for instances that have not been assigned public IPv4 addresses.
  • An internet gateway supports IPv4 and IPv6 traffic.

NAT Gateway

  • NAT Gateway enables instances in a private subnet to connect to the internet (for example, for software updates) or other AWS services, but prevent the internet from initiating connections with the instances.
  • A NAT gateway forwards traffic from the instances in the private subnet to the internet or other AWS services, and then sends the response back to the instances.
  • When traffic goes to the internet, the source IPv4 address is replaced with the NAT device’s address and similarly, when the response traffic goes to those instances, the NAT device translates the address back to those instances’ private IPv4 addresses.

Egress Only Internet Gateway

  • NAT devices are not supported for IPv6 traffic, use an Egress-only Internet gateway instead
  • Egress-only Internet gateway is a horizontally scaled, redundant, and highly available VPC component that allows outbound communication over IPv6 from instances in the VPC to the Internet, and prevents the Internet from initiating an IPv6 connection with your instances.

VPC Endpoints

  • VPC endpoint provides a private connection from VPC to supported AWS services and VPC endpoint services powered by PrivateLink without requiring an internet gateway, NAT device, VPN connection, or AWS Direct Connect connection.
  • Instances in the VPC do not require public IP addresses to communicate with resources in the service. Traffic between the VPC and the other service does not leave the Amazon network.
  • VPC Endpoints are virtual devices and are horizontally scaled, redundant, and highly available VPC components that allow communication between instances in the VPC and services without imposing availability risks or bandwidth constraints on the network traffic.
  • VPC Endpoints are of two types
    • Interface Endpoints – is an elastic network interface with a private IP address that serves as an entry point for traffic destined to supported services.
    • Gateway Endpoints – is a gateway that is a target for a specified route in your route table, used for traffic destined to a supported AWS service. Currently only Amazon S3 and DynamoDB.

More details @ VPC Endpoints

VPC Peering

  • VPC peering connection enables networking connection between two VPCs to route traffic between them using private IPv4 addresses or IPv6 addresses
  • VPC peering connections can be created between your own VPCs, or with a VPC in another AWS account
  • VPC peering connections can be created across regions, referred to as inter-region VPC peering connection
  • VPC peering uses existing underlying AWS infrastructure; it is neither a gateway nor a VPN connection, and does not rely on a separate piece of physical hardware.
  • VPC Peering does not have a single point of failure for communication or a bandwidth bottleneck.
  • VPC Peering connections have limitations
    • Cannot be used with Overlapping CIDR blocks
    • Does not provide Transitive peering
    • Doe not support Edge to Edge routing through Gateway or private connection

More details @ VPC Peering

VPN CloudHub

VPC CloudHub
  • AWS VPN CloudHub allows you to securely communicate from one site to another using AWS Managed VPN or Direct Connect
  • AWS VPN CloudHub operates on a simple hub-and-spoke model that can be used with or without a VPC
  • AWS VPN CloudHub can be used if you have multiple branch offices and existing internet connections and would like to implement a convenient, potentially low cost hub-and-spoke model for primary or backup connectivity between these remote offices.
  • AWS VPN CloudHub leverages VPC virtual private gateway with multiple gateways, each using unique BGP autonomous system numbers (ASNs).

Transit VPC

  • A transit VPC is a common strategy for connecting multiple, geographically disperse VPCs and remote networks in order to create a global network transit center.
  • A transit VPC simplifies network management and minimizes the number of connections required to connect multiple VPCs and remote networks
  • Transit VPC can be used to support important use cases
    • Private Networking – You can build a private network that spans two or more AWS Regions.
    • Shared Connectivity – Multiple VPCs can share connections to data centers, partner networks, and other clouds.
    • Cross-Account AWS Usage – The VPCs and the AWS resources within them can reside in multiple AWS accounts.
  • Transit VPC design helps implement more complex routing rules, such as network address translation between overlapping network ranges, or to add additional network-level packet filtering or inspection.
  • Transit VPC
    • supports Transitive routing using the overlay VPN network — allowing for a simpler hub and spoke design. Can be used to provide shared services for VPC Endpoints, Direct Connect connection, etc.
    • supports network address translation between overlapping network ranges.
    • supports vendor functionality around advanced security (layer 7 firewall/Intrusion Prevention System (IPS)/Intrusion Detection System (IDS) ) using third-party software on EC2
    • leverages instance-based routing that increases costs while lowering availability and limiting the bandwidth.
    • Customers are responsible for managing the HA and redundancy of EC2 instances running the third-party vendor virtual appliances

Transit Gateway (Virtual Routing and Forwarding)

  • Transit gateway enables you to attach VPCs (across accounts) and VPN connections in the same Region and route traffic between them
  • Transit gateways support dynamic and static routing between attached VPCs and VPN connections
  • Transit gateway removes the need for using full mesh VPC Peering and Transit VPC

Virtual Private Network (VPN)

VPC Managed VPN Connection
  • VPC provides the option of creating an IPsec VPN connection between remote customer networks and their VPC over the internet
  • AWS managed VPN endpoint includes automated multi–data center redundancy & failover built into the AWS side of the VPN connection
  • AWS managed VPN consists of two parts
    • Virtual Private Gateway (VPG) on AWS side
    • Customer Gateway (CGW) on the on-premises data center
  • AWS Managed VPN only provides Site-to-Site VPN connectivity. It does not provide Point-to-Site VPC connectivity for e.g. from Mobile
  • Virtual Private Gateway are Highly Available as it represents two distinct VPN endpoints, physically located in separate data centers to increase the availability of the VPN connection.
  • High Availability on the on-premises data center must be handled by creating additional Customer Gateway.
  • AWS Managed VPN connections are low cost, quick to setup and start with compared to Direct Connect. However, they are not reliable as they traverse through Internet.

More details @ Virtual Private Network

Software VPN

  • VPC offers the flexibility to fully manage both sides of the VPC connectivity by creating a VPN connection between your remote network and a software VPN appliance running in your VPC network.
  • Software VPNs help manage both ends of the VPN connection either for compliance purposes or for leveraging gateway devices that are not currently supported by Amazon VPC’s VPN solution.
  • Software VPNs allows you to handle Point-to-Site connectivity
  • Software VPNs, with the above design, introduces a single point of failure and needs to be handled.

Direct Connect – DX

  • AWS Direct Connect helps establish a dedicated private connection between an on-premises network and AWS.
  • Direct Connect can reduce network costs, increase bandwidth throughput, and provide a more consistent network experience than internet-based or VPN connections
  • Direct Connect uses industry-standard VLANs to access EC2 instances running within a VPC using private IP addresses
  • Direct Connect lets you establish
    • Dedicated Connection: A 1G, 10G, or 100G physical Ethernet connection associated with a single customer through AWS.
    • Hosted Connection: A 1G or 10G physical Ethernet connection that an AWS Direct Connect Partner provisions on behalf of a customer.
  • Direct Connect provides the following Virtual Interfaces
    • Private virtual interface – to access a VPC using private IP addresses.
    • Public virtual interface – to access all AWS public services using public IP addresses.
    • Transit virtual interface – to access one or more transit gateways associated with Direct Connect gateways.
  • Direct Connect connections are not redundant as each connection consists of a single dedicated connection between ports on your router and an Amazon router
  • Direct Connect High Availability can be configured using
    • Multiple Direct Connect connections
    • Back-up IPSec VPN connection


  • Direct Connect link aggregation group (LAG) is a logical interface that uses the Link Aggregation Control Protocol (LACP) to aggregate multiple connections at a single AWS Direct Connect endpoint, allowing you to treat them as a single, managed connection.
  • LAGs need the following
    • All connections in the LAG must use the same bandwidth.
    • A maximum of four connections in a LAG. Each connection in the LAG counts toward the overall connection limit for the Region.
    • All connections in the LAG must terminate at the same AWS Direct Connect endpoint.

Direct Connect Gateway

  • Direct Connect Gateway allows you to connect an AWS Direct Connect connection to one or more VPCs in your account that are located in the same or different regions
  • Direct Connect gateway can be created in any public region and accessed from all other public regions
  • Direct Connect gateway CAN NOT be used to connect to a VPC in another account.
  • Direct Connect locations can also access the public resources in any AWS Region using a public virtual interface.

Direct Connect with VPN

  • AWS Direct Connect plus VPN provides an IPsec-encrypted private connection that also reduces network costs, increases bandwidth throughput, and provides a more consistent network experience than internet-based VPN connections.