Internal Loadbalancers don't work if a cluster is using custom mode subnets

[upodroid]

https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kops/15972/pull-kops-kubernetes-e2e-cos-gce-slow/1707525650782359552

 mahamed  REDACTED  ~  $  curl -s https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/kops/15972/pull-kops-kubernetes-e2e-cos-gce-slow/1707525650782359552/artifacts/cluster-info/kube-system/cloud-controller-manager-nbgbm/logs.txt | grep lb-internal | head -10
I0928 23:17:57.922612       1 controller.go:397] Ensuring load balancer for service loadbalancers-2999/lb-internal
I0928 23:17:57.922679       1 controller.go:906] Adding finalizer to service loadbalancers-2999/lb-internal
I0928 23:17:57.923315       1 event.go:307] "Event occurred" object="loadbalancers-2999/lb-internal" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
E0928 23:18:27.799181       1 gce_loadbalancer.go:206] Failed to EnsureLoadBalancer(e2e-pr15972-pull-kops-kubernetes-e2e-cos-gce-slow-k8s-local, loadbalancers-2999, lb-internal, a95259e9dc5fe403282161291b197790, us-east4), err: googleapi: Error 400: Invalid value for field 'resource.subnetwork': ''. Network interface must specify a subnet if the network resource is in custom subnet mode., invalid
E0928 23:18:27.799521       1 controller.go:298] error processing service loadbalancers-2999/lb-internal (retrying with exponential backoff): failed to ensure load balancer: googleapi: Error 400: Invalid value for field 'resource.subnetwork': ''. Network interface must specify a subnet if the network resource is in custom subnet mode., invalid
I0928 23:18:27.799882       1 event.go:307] "Event occurred" object="loadbalancers-2999/lb-internal" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to ensure load balancer: googleapi: Error 400: Invalid value for field 'resource.subnetwork': ''. Network interface must specify a subnet if the network resource is in custom subnet mode., invalid"
I0928 23:20:53.782977       1 controller.go:397] Ensuring load balancer for service loadbalancers-2999/lb-internal
I0928 23:20:53.784454       1 event.go:307] "Event occurred" object="loadbalancers-2999/lb-internal" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
E0928 23:20:57.892662       1 gce_loadbalancer.go:206] Failed to EnsureLoadBalancer(e2e-pr15972-pull-kops-kubernetes-e2e-cos-gce-slow-k8s-local, loadbalancers-2999, lb-internal, a95259e9dc5fe403282161291b197790, us-east4), err: googleapi: Error 400: Invalid value for field 'resource.subnetwork': ''. Network interface must specify a subnet if the network resource is in custom subnet mode., invalid
E0928 23:20:57.893048       1 controller.go:298] error processing service loadbalancers-2999/lb-internal (retrying with exponential backoff): failed to ensure load balancer: googleapi: Error 400: Invalid value for field 'resource.subnetwork': ''. Network interface must specify a subnet if the network resource is in custom subnet mode., invalid

kube-up creates subnets in auto-mode subnets, kops creates subnets in custom mode

Worth mentioning that no really uses auto mode subnets in prod(you can't control the IPs of the subnets)

/kind bug

[k8s-ci-robot]

This issue is currently awaiting triage.

If the repository mantainers determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[aojea]

/assign @cezarygerard @code-elinka

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[upodroid]

/remove-lifecycle stale
/priority important-soon

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten