GCP: experiment nodepools per SIG

[ameukam]

Experiment if split prowjobs on different GKE nodepools allocated per SIG is viable.

SIG CI Node gave a +1 for participation to the experiment.

Experiment should be done after 1.33 is out and before 1.34 code freeze.

TODO:

• create GKE nodepool with taints using GCE N4 instances
• add tolerations to some prowjobs: https://github.com/kubernetes/test-infra/tree/master/config/jobs/kubernetes/sig-node
• check performance and CPU/memory consumption.

cc @SergeyKanzhelev

/assign
/sig k8s-infra
/sig node
/priority backlog
/milestone v1.34

[BenTheElder]

Wait, why would we do nodepools per sig? What's the purpose?

Won't this give us worse bin-packing than sharing one node pool ...?

[BenTheElder]

cc @kubernetes/sig-k8s-infra-leads

[ameukam]

Wait, why would we do nodepools per sig? What's the purpose?

I think breaking prowjobs could help simplify maintenance and potentially improve performance. A split per SIG looks like the simple approach to me for this expermient. Also we don't have to do it for all the SIGs. We can target only 2-3 SIGs to do this. with more nodepools we can have different instance types, disk types, etc.

Won't this give us worse bin-packing than sharing one node pool ...?

I think not really for the case of prowjobs by owned SIG node.

[BenTheElder]

Doing a few sigs in order to experiment with different node config and then consolidate on one afterwards makes sense to me.

Permanently splitting wouldn't, the issue description could do with more detail. As written it sounds like an experiment towards permanently splitting.

[BenTheElder]

I still wouldn't frame them as sig node pools. Or even split them, we should just try desired node pool configs on specific jobs which will span sigs

[ameukam]

I still wouldn't frame them as sig node pools. Or even split them, we should just try desired node pool configs on specific jobs which will span sigs

🤔 what are the cons of doing a permanent split ?

[BenTheElder]

🤔 what are the cons of doing a permanent split ?

worse bin packing / autoscaling

more complex and confusing prowjob config

poorer portability between clusters

false sense of per-sig utilization (in reality a lot of jobs are for many sigs, maybe "sig testing" owns them or something but they're producing test results for many sigs)