Google Kubernetes Engine – Networking
IP allocation
Kubernetes uses various IP ranges to assign IP addresses to Nodes, Pods, and Services.
- Node IP
- Each node has an IP address assigned from the cluster’s VPC network.
- Node IP provides connectivity from control components like
kube-proxyandkubeletto the Kubernetes API server. - Node IP is the node’s connection to the rest of the cluster.
- Pod CIDR or Address Range
- Each node has a pool of IP addresses that GKE assigns the Pods running on that node (a /24 CIDR block by default).
- Pod Address
- Each Pod has a single IP address assigned from the Pod CIDR range of its node.
- Pod IP address is shared by all containers running within the Pod and connects them to other Pods running in the cluster.
- Service Address Range
- Each Service has an IP address, called the ClusterIP, assigned from the cluster’s VPC network.
- For Standard clusters
- a maximum of 110 Pods can run on a node with a /24 range, not 256 as you might expect. This provides a buffer so that Pods don’t become unschedulable due to a transient lack of IP addresses in the Pod IP range for a given node.
- For ranges smaller than /24, roughly half as many Pods can be scheduled as IP addresses in the range.
- Autopilot clusters can run a maximum of 32 Pods per node.
GKE Cluster Networking Types
- GKE, clusters can be distinguished according to the way they route traffic from one Pod to another Pod.
- VPC-native cluster: A cluster that uses alias IP address ranges
- Routes-based cluster: A cluster that uses custom static routes in a VPC network
VPC-Native Clusters
- VPC-native cluster uses alias IP address ranges
- VPC-native is the recommended network mode for new clusters
- VPC-native clusters have several benefits:
- Pod IP addresses are natively routable within the cluster’s VPC network and other VPC networks connected to it by VPC Network Peering.
- Pod IP address ranges, and subnet secondary IP address ranges in general, are accessible from on-premises networks connected with Cloud VPN or Cloud Interconnect using Cloud Routers.
- Pod IP addresses are reserved in the VPC network before the Pods are created in the cluster. This prevents conflict with other resources in the VPC network and allows you to better plan IP address allocations.
- Pod IP address ranges do not depend on custom static routes and do not consume the system-generated and custom static routes quota. Instead, automatically generated subnet routes handle routing for VPC-native clusters.
- Firewall rules can be created that apply to just Pod IP address ranges instead of any IP address on the cluster’s nodes.
VPC-Native Clusters IP Allocation
- VPC-native cluster uses three unique subnet IP address ranges
- Subnet’s primary IP address range for all node IP addresses.
- Node IP addresses are assigned from the primary IP address range of the subnet associated with the cluster.
- Both node IP addresses and the size of the subnet’s secondary IP address range for Pods limit the number of nodes that a cluster can support
- One secondary IP address range for all Pod IP addresses.
- Pod IP addresses are taken from the cluster subnet’s secondary IP address range for Pods.
- By default, GKE allocates a
/24alias IP range (256 addresses) to each node for the Pods running on it. - On each node, those 256 alias IP addresses support up to 110 Pods.
- Pod Address Range cannot be changed. If exhausted,
- a new cluster with a larger Pod address range must be created or
- node pools should be recreated after decreasing the
--max-pods-per-nodefor the node pools.
- Another secondary IP address range for all Service (cluster IP) addresses.
- Service (cluster IP) addresses are taken from the cluster’s subnet’s secondary IP address range for Services.
- Service address range should be large enough to provide addresses for all the Kubernetes Services hosted in the cluster.
- Subnet’s primary IP address range for all node IP addresses.
- Node, Pod, and Services IP address ranges must all be unique and subnets with overlapping primary and secondary IP addresses cannot be created.
Routes-based Cluster
- Routes-based cluster that uses custom static routes in a VPC network i.e. it uses Google Cloud Routes to route traffic between nodes
- In a routes-based cluster,
- each node is allocated a /24 range of IP addresses for Pods.
- With a /24 range, there are 256 addresses, but the maximum number of Pods per node is 110.
- With approximately twice as many available IP addresses as possible Pods, Kubernetes is able to mitigate IP address reuse as Pods are added to and removed from a node.
- Routes-based cluster uses two unique subnet IP address ranges
- Subnet’s primary IP address range for all node IP addresses.
- Node IP addresses are taken from the primary range of the cluster subnet
- Cluster subnet must be large enough to hold the total number of nodes in your cluster.
- Pod address range
- A routes-based cluster has a range of IP addresses that are used for Pods and Services
- Last /20 (4096 addresses) of the Pod address range is used for Services and the rest of the range is used for Pods
- Pod address range size cannot be changed after cluster creation. So ensure that a large enough Pod address range is chosen to accommodate the cluster’s anticipated growth during cluster creation
- Subnet’s primary IP address range for all node IP addresses.
- Maximum number of nodes, Pods, and Services for a given GKE cluster is determined by the size of the cluster subnet and the size of the Pod address range.
Related Reads
GCP Certification Exam Practice Questions
- Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
- GCP services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
- GCP exam questions are not updated to keep up the pace with GCP updates, so even if the underlying feature has changed the question might not be updated
- Open to further feedback, discussion and correction.