Set nodeSelector per subnet

[and-1]

What would you like to be added?

I want to suggest to set nodeSelector per subnet, not per all pools

What is the use case for this feature / enhancement?

I use sriov cni and have one subnet per rack. I need use this subnet for all pods hosted in rack. So if set nodeSelector per subnet, i can assign right subnet to pool of nodes and wherever pod scheduled (this rack or another) it get correct ip address

[and-1]

Another way to do this - change config scheme to list, like that:
[ { "pools": { "pool1": {"subnet": "192.168.0.0/24", "perNodeBlockSize": 29, "gateway": "192.168.0.1"}, "pool2": {"subnet": "192.168.1.0/24", "perNodeBlockSize": 29, "gateway": "192.168.1.1"} }, "nodeSelector": { "topology.kubernetes.io/rack": "100" } }, { "pools": { "pool1": {"subnet": "192.168.10.0/24", "perNodeBlockSize": 29, "gateway": "192.168.10.1"}, "pool2": {"subnet": "192.168.11.0/24", "perNodeBlockSize": 29, "gateway": "192.168.11.1"} }, "nodeSelector": { "topology.kubernetes.io/rack": "200" } } ]

[adrianchiris]

Hi @and-1

there is currently work being done to move the defintion of IP Pools to CustomResource. there each pool will have a node selector.

[and-1]

Great! What ETA do you forecast?

[adrianchiris]

initial PRs should land during Aug.

[and-1]

Name of custom resources will be name of pool or i can use same pool name in few custom resources?

[adrianchiris]

pool == CR instance.

star the project so u get notified when PR lands. first one will be the API so you can review and comment.

[rollandf]

See draft PR #24

[and-1]

Unfortunately such scheme don't resolve my case. I need many subnets per pool and each subnet need to have own nodeSelector. i can set only one pool name in cni config (ipam section) in NetworkAttachmentDefinition. This network name i need to set in annotation of PytorchJob master/worker template. Pods of this job resides on hosts from different rack and ipam has to assign IP from correct subnet (subnet1 if pod in rack1, and subnet2 if pod in rack2)

[adrianchiris]

currently nvidia-k8s-ipam is not intended to support such scheme.

to support your use-case it would be we would need to support:

1. multiple ranges per pool
2. a node selector per range in the pool

can you provide more info on the networking scheme you have ? one assumption we have here is that a subnet is reachable for all nodes (or at least all nodes that satisfy a selector)

are you working at NVIDIA ?