Unable to get the loadbalancer IP's in a CKS Cluster deployed in a VPC

[kiranchavala]

Steps to reproduce the issue

Env details

Cloudstack version 4.20.0
KVM

1. Create a vpc network
2. Create a tier in the vpc network (step1)
3. Deploy a cks cluster in the tier (step 2)
4. Deploy a nginx application
5. Expose the application via loadbalancer IP

k expose deploy/nginx-deployment --port=80 --type=LoadBalancer

k get svc
NAME               TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
kubernetes         ClusterIP      10.96.0.1      <none>        443/TCP        10m
nginx-deployment   LoadBalancer   10.99.66.199   <pending>     80:31935/TCP   8m40s

6. There is no loadbalancer IP associated

Exception in the logs of

I0428 10:38:28.410408       1 serving.go:348] Generated self-signed cert in-memory
W0428 10:38:41.113777       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0428 10:38:41.308527       1 controllermanager.go:143] Version: v0.0.0-master+$Format:%H$
I0428 10:38:41.318197       1 secure_serving.go:210] Serving securely on [::]:10258
I0428 10:38:41.408664       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0428 10:38:41.409763       1 leaderelection.go:248] attempting to acquire leader lease kube-system/cloud-controller-manager...
I0428 10:38:41.410704       1 shared_informer.go:252] Waiting for caches to sync for RequestHeaderAuthRequestController
I0428 10:38:41.408700       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0428 10:38:41.411011       1 shared_informer.go:252] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0428 10:38:41.408713       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0428 10:38:41.411049       1 shared_informer.go:252] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0428 10:38:41.408715       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0428 10:38:41.811240       1 shared_informer.go:259] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0428 10:38:41.911994       1 shared_informer.go:259] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0428 10:38:41.912114       1 shared_informer.go:259] Caches are synced for RequestHeaderAuthRequestController
I0428 10:38:42.008685       1 leaderelection.go:258] successfully acquired lease kube-system/cloud-controller-manager
I0428 10:38:42.111469       1 event.go:294] "Event occurred" object="kube-system/cloud-controller-manager" fieldPath="" kind="Lease" apiVersion="coordination.k8s.io/v1" type="Normal" reason="LeaderElection" message="cloud-controller-manager-79d84c86b7-cc7td_886c7ad3-ef3e-4373-8a10-2dbccc00b094 became leader"
I0428 10:38:42.912253       1 node_controller.go:118] Sending events to api server.
I0428 10:38:42.913064       1 controllermanager.go:291] Started "cloud-node"
I0428 10:38:42.913243       1 node_controller.go:157] Waiting for informer caches to sync
I0428 10:38:42.913822       1 node_lifecycle_controller.go:77] Sending events to api server
I0428 10:38:42.914003       1 controllermanager.go:291] Started "cloud-node-lifecycle"
I0428 10:38:42.914837       1 controllermanager.go:291] Started "service"
W0428 10:38:42.914960       1 cloudstack.go:156] This cloud provider doesn't support routes
W0428 10:38:42.915053       1 core.go:110] --configure-cloud-routes is set, but cloud provider does not support routes. Will not configure cloud provider routes.
W0428 10:38:42.915192       1 controllermanager.go:279] Skipping "route"
I0428 10:38:42.914962       1 controller.go:233] Starting service controller
I0428 10:38:42.915444       1 shared_informer.go:252] Waiting for caches to sync for service
I0428 10:38:43.308269       1 shared_informer.go:259] Caches are synced for service
E0428 10:39:13.414707       1 node_controller.go:249] Error getting instance metadata for node addresses: error retrieving instance type: Get "http://10.0.34.111:8080/client/api?apiKey=VUhQVl5vc4ouQq0Ht2CebJfO9jcaLZsL79MTvg7NO9viUYuNA2nSvyE2FEls8WQvzT-wtlz4X56ulbJgIADDvA&command=listVirtualMachines&name=test-control-1967bf69986&response=json&signature=nksb%2FWrsHGLrRO%2FvabnzV2igIPE%3D": dial tcp 10.0.34.111:8080: i/o timeout
I0428 10:39:22.010102       1 event.go:294] "Event occurred" object="default/nginx-deployment" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
E0428 10:39:43.416042       1 node_controller.go:249] Error getting instance metadata for node addresses: error retrieving instance type: Get "http://10.0.34.111:8080/client/api?apiKey=VUhQVl5vc4ouQq0Ht2CebJfO9jcaLZsL79MTvg7NO9viUYuNA2nSvyE2FEls8WQvzT-wtlz4X56ulbJgIADDvA&command=listVirtualMachines&name=test-node-1967bf71379&response=json&signature=0ibB7IGf9mNP%2BjjaoKvaf4Qv%2B7I%3D": dial tcp 10.0.34.111:8080: i/o timeout
E0428 10:39:52.111827       1 controller.go:310] error processing service default/nginx-deployment (will retry): failed to ensure load balancer: error retrieving load balancer rules: Get "http://10.0.34.111:8080/client/api?apiKey=VUhQVl5vc4ouQq0Ht2CebJfO9jcaLZsL79MTvg7NO9viUYuNA2nSvyE2FEls8WQvzT-wtlz4X56ulbJgIADDvA&command=listLoadBalancerRules&keyword=a3c7b61141ae64351999e0457d01a785&listall=true&response=json&signature=D0YI7HbtJsgIlVLsjFYJQAV0BME%3D": dial tcp 10.0.34.111:8080: i/o timeout
I0428 10:39:52.111926       1 event.go:294] "Event occurred" object="default/nginx-deployment" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to ensure load balancer: error retrieving load balancer rules: Get \"http://10.0.34.111:8080/client/api?apiKey=VUhQVl5vc4ouQq0Ht2CebJfO9jcaLZsL79MTvg7NO9viUYuNA2nSvyE2FEls8WQvzT-wtlz4X56ulbJgIADDvA&command=listLoadBalancerRules&keyword=a3c7b61141ae64351999e0457d01a785&listall=true&response=json&signature=D0YI7HbtJsgIlVLsjFYJQAV0BME%3D\": dial tcp 10.0.34.111:8080: i/o timeout"
I0428 10:39:57.113030       1 event.go:294] "Event occurred" object="default/nginx-deployment" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
E0428 10:40:27.114561       1 controller.go:310] error processing service default/nginx-deployment (will retry): failed to ensure load balancer: error retrieving load balancer rules: Get "http://10.0.34.111:8080/client/api?apiKey=VUhQVl5vc4ouQq0Ht2CebJfO9jcaLZsL79MTvg7NO9viUYuNA2nSvyE2FEls8WQvzT-wtlz4X56ulbJgIADDvA&command=listLoadBalancerRules&keyword=a3c7b61141ae64351999e0457d01a785&listall=true&response=json&signature=D0YI7HbtJsgIlVLsjFYJQAV0BME%3D": dial tcp 10.0.34.111:8080: i/o timeout
I0428 10:40:27.114778       1 event.go:294] "Event occurred" object="default/nginx-deployment" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to ensure load balancer: error retrieving load balancer rules: Get \"http://10.0.34.111:8080/client/api?apiKey=VUhQVl5vc4ouQq0Ht2CebJfO9jcaLZsL79MTvg7NO9viUYuNA2nSvyE2FEls8WQvzT-wtlz4X56ulbJgIADDvA&command=listLoadBalancerRules&keyword=a3c7b61141ae64351999e0457d01a785&listall=true&response=json&signature=D0YI7HbtJsgIlVLsjFYJQAV0BME%3D\": dial tcp 10.0.34.111:8080: i/o timeout"
I0428 10:40:37.116776       1 event.go:294] "Event occurred" object="default/nginx-deployment" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
E0428 10:41:07.117019       1 controller.go:310] error processing service default/nginx-deployment (will retry): failed to ensure load balancer: error retrieving load balancer rules: Get "http://10.0.34.111:8080/client/api?apiKey=VUhQVl5vc4ouQq0Ht2CebJfO9jcaLZsL79MTvg7NO9viUYuNA2nSvyE2FEls8WQvzT-wtlz4X56ulbJgIADDvA&command=listLoadBalancerRules&keyword=a3c7b61141ae64351999e0457d01a785&listall=true&response=json&signature=D0YI7HbtJsgIlVLsjFYJQAV0BME%3D": dial tcp 10.0.34.111:8080: i/o timeout
I0428 10:41:07.117182       1 event.go:294] "Event occurred" object="default/nginx-deployment" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to ensure load balancer: error retrieving load balancer rules: Get \"http://10.0.34.111:8080/client/api?apiKey=VUhQVl5vc4ouQq0Ht2CebJfO9jcaLZsL79MTvg7NO9viUYuNA2nSvyE2FEls8WQvzT-wtlz4X56ulbJgIADDvA&command=listLoadBalancerRules&keyword=a3c7b61141ae64351999e0457d01a785&listall=true&response=json&signature=D0YI7HbtJsgIlVLsjFYJQAV0BME%3D\": dial tcp 10.0.34.111:8080: i/o timeout"
I0428 10:41:27.117754       1 event.go:294] "Event occurred" object="default/nginx-deployment" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
E0428 10:41:57.118414       1 controller.go:310] error processing service default/nginx-deployment (will retry): failed to ensure load balancer: error retrieving load balancer rules: Get "http://10.0.34.111:8080/client/api?apiKey=VUhQVl5vc4ouQq0Ht2CebJfO9jcaLZsL79MTvg7NO9viUYuNA2nSvyE2FEls8WQvzT-wtlz4X56ulbJgIADDvA&command=listLoadBalancerRules&keyword=a3c7b61141ae64351999e0457d01a785&listall=true&response=json&signature=D0YI7HbtJsgIlVLsjFYJQAV0BME%3D": dial tcp 10.0.34.111:8080: i/o timeout
I0428 10:41:57.119581       1 event.go:294] "Event occurred" object="default/nginx-deployment" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to ensure load balancer: error retrieving load balancer rules: Get \"http://10.0.34.111:8080/client/api?apiKey=VUhQVl5vc4ouQq0Ht2CebJfO9jcaLZsL79MTvg7NO9viUYuNA2nSvyE2FEls8WQvzT-wtlz4X56ulbJgIADDvA&command=listLoadBalancerRules&keyword=a3c7b61141ae64351999e0457d01a785&listall=true&response=json&signature=D0YI7HbtJsgIlVLsjFYJQAV0BME%3D\": dial tcp 10.0.34.111:8080: i/o timeout"
I0428 10:42:37.119970       1 event.go:294] "Event occurred" object="default/nginx-deployment" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
E0428 10:43:07.122115       1 controller.go:310] error processing service default/nginx-deployment (will retry): failed to ensure load balancer: error retrieving load balancer rules: Get "http://10.0.34.111:8080/client/api?apiKey=VUhQVl5vc4ouQq0Ht2CebJfO9jcaLZsL79MTvg7NO9viUYuNA2nSvyE2FEls8WQvzT-wtlz4X56ulbJgIADDvA&command=listLoadBalancerRules&keyword=a3c7b61141ae64351999e0457d01a785&listall=true&response=json&signature=D0YI7HbtJsgIlVLsjFYJQAV0BME%3D": dial tcp 10.0.34.111:8080: i/o timeout
I0428 10:43:07.122995       1 event.go:294] "Event occurred" object="default/nginx-deployment" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to ensure load balancer: error retrieving load balancer rules: Get \"http://10.0.34.111:8080/client/api?apiKey=VUhQVl5vc4ouQq0Ht2CebJfO9jcaLZsL79MTvg7NO9viUYuNA2nSvyE2FEls8WQvzT-wtlz4X56ulbJgIADDvA&command=listLoadBalancerRules&keyword=a3c7b61141ae64351999e0457d01a785&listall=true&response=json&signature=D0YI7HbtJsgIlVLsjFYJQAV0BME%3D\": dial tcp 10.0.34.111:8080: i/o timeout"
I0428 10:44:27.126466       1 event.go:294] "Event occurred" object="default/nginx-deployment" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
E0428 10:44:57.126482       1 controller.go:310] error processing service default/nginx-deployment (will retry): failed to ensure load balancer: error retrieving load balancer rules: Get "http://10.0.34.111:8080/client/api?apiKey=VUhQVl5vc4ouQq0Ht2CebJfO9jcaLZsL79MTvg7NO9viUYuNA2nSvyE2FEls8WQvzT-wtlz4X56ulbJgIADDvA&command=listLoadBalancerRules&keyword=a3c7b61141ae64351999e0457d01a785&listall=true&response=json&signature=D0YI7HbtJsgIlVLsjFYJQAV0BME%3D": dial tcp 10.0.34.111:8080: i/o timeout
I0428 10:44:57.127453       1 event.go:294] "Event occurred" object="default/nginx-deployment" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to ensure load balancer: error retrieving load balancer rules: Get \"http://10.0.34.111:8080/client/api?apiKey=VUhQVl5vc4ouQq0Ht2CebJfO9jcaLZsL79MTvg7NO9viUYuNA2nSvyE2FEls8WQvzT-wtlz4X56ulbJgIADDvA&command=listLoadBalancerRules&keyword=a3c7b61141ae64351999e0457d01a785&listall=true&response=json&signature=D0YI7HbtJsgIlVLsjFYJQAV0BME%3D\": dial tcp 10.0.34.111:8080: i/o timeout"
E0428 10:45:13.427832       1 node_controller.go:249] Error getting instance metadata for node addresses: error retrieving instance type: Get "http://10.0.34.111:8080/client/api?apiKey=VUhQVl5vc4ouQq0Ht2CebJfO9jcaLZsL79MTvg7NO9viUYuNA2nSvyE2FEls8WQvzT-wtlz4X56ulbJgIADDvA&command=listVirtualMachines&name=test-control-1967bf69986&response=json&signature=nksb%2FWrsHGLrRO%2FvabnzV2igIPE%3D": dial tcp 10.0.34.111:8080: i/o timeout
E0428 10:45:43.428345       1 node_controller.go:249] Error getting instance metadata for node addresses: error retrieving instance type: Get "http://10.0.34.111:8080/client/api?apiKey=VUhQVl5vc4ouQq0Ht2CebJfO9jcaLZsL79MTvg7NO9viUYuNA2nSvyE2FEls8WQvzT-wtlz4X56ulbJgIADDvA&command=listVirtualMachines&name=test-node-1967bf71379&response=json&signature=0ibB7IGf9mNP%2BjjaoKvaf4Qv%2B7I%3D": dial tcp 10.0.34.111:8080: i/o timeout

[kiranchavala]

The issue is not observed in a isolated network

[vishesh92]

I tried to reproduce the error, but I couldn't. From the logs, it looks like a networking issue in the network where an Instance in the network can't reach out to the management server.
dial tcp 10.0.34.111:8080: i/o timeout

[kiranchavala]

Thanks @vishesh92 closing the issue after performing the workaround

There could be Calico routing edge case encountered in some environments. By default, Calico uses the 192.168.0.0/16 network for its pod IP pool when you install it with the standard manifests. To avoid potential routing conflicts with existing networks in your infrastructure, it’s advisable to customize the Calico IP pool to use a different subnet that doesn’t overlap with your current network setup.

kubectl get ippool.crd.projectcalico.org -o yaml

You can edit the IP pool to change the CIDR to a different subnet that fits your network architecture better. For example, you might choose to use 10.0.0.0/16.

kubectl edit ippool default-ipv4-ippool

and redeploy the pods

kubectl delete pod –all -A