A VPC native cluster uses three unique subnet ranges to allocate IPs to Nodes, Pods and Services.
Primary subnet IP address is used for Nodes. Node IP provides connectivity from control components like kube-proxy and kubelet to the Kubernetes API server. Node IP is the node’s connection to the rest of the cluster.
Secondary subnet IP address is used for Pods. Pod IP addresses are natively routable within the cluster’s VPC network and other VPC networks connected to it by VPC Network Peering. By default GKE allocates /24 alias ie., 256 alias IP addresses for 110 pods for each of the nodes.
Another Secondary subnet IP address is used for services. Each Service has an IP address, called the ClusterIP, assigned from the cluster’s VPC network.
In a VPC native cluster, these addresses are reserved before the creation of cluster to eliminate conflict and overlapping of IPs.
What happens when the secondary IP range exhausts?
Once the secondary IP address exhausts no Pods can be scheduled. The secondary Pod IP range cannot be changed once created. There is a provision to allocate a separate IP range to create separate node pools in the cluster. These two IP ranges are going to be discontigous and there are some caveats which needs to be taken care of.
If you use ip-masq-agent configured with the nonMasqueradeCIDRs parameter, you must update the nonMasqueradeCIDRs to include all Pod CIDR ranges.
If you use NetworkPolicy configured with ipBlock to specify traffic, you must update the cidr value to include all Pod CIDR ranges.
The other approach is to create a bigger CIDR range node pool and move workloads from the existing node pool to the newly created one. Steps for the same:
-
Mark the nodes in the existing node pool to be non schedulable by using the kubernetes cordon command
kubectl cordon <node-name> -
Redeploy the application and verify if the nodes are scheduling in the nodes of new node pool
-
Once verified, redeploy and move all the workloads in the new node pool.
-
Clean up and delete the old node pool.