NV-IPAM ignores exclusion ranges in IPPool allocations

[aroynutanix]

What happened?

When defining an IPPool with exclusions to restrict IP allocation to a narrow range, NV-IPAM still allocates addresses from the excluded space. The status.allocations field shows IPs handed out from inside the exclusion ranges, suggesting the controller is not honoring exclusions.

What did you expect to happen?

Created this IpPool CR. This should leave only 10.117.127.32 – 10.117.127.127 (96 usable addresses) available for allocation.

cat <<EOF | kubectl apply -f -
apiVersion: nv-ipam.nvidia.com/v1alpha1
kind: IPPool
metadata:
  name: hostdev-pool-a-su-1
  namespace: nvidia-network-operator
spec:
  subnet: 10.117.112.0/20
  perNodeBlockSize: 8
  gateway: 10.117.112.1
  exclusions:
  # use 96 IPs between the range of 10.117.127.32 to 10.117.127.127
  # exclude everything before 10.117.127.31 
  - startIP: 10.117.112.1
    endIP: 10.117.127.31
  # exclude everything after 10.117.127.128
  - startIP: 10.117.127.128
    endIP: 10.117.127.255
  nodeSelector:
    nodeSelectorTerms:
    - matchExpressions:
      - key: node-role.kubernetes.io/worker
        operator: Exists

Expected Behavior:
Allocations should come only from the allowed range (10.117.127.32 – 10.117.127.127).
No IPs inside the excluded ranges should ever appear in .status.allocations.
Actual Behavior:
status.allocations contains blocks from the excluded space:

kubectl get ippool.nv-ipam.nvidia.com hostdev-pool-a-su-1 -o yaml -n nvidia-network-operator
apiVersion: nv-ipam.nvidia.com/v1alpha1
kind: IPPool
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"nv-ipam.nvidia.com/v1alpha1","kind":"IPPool","metadata":{"annotations":{},"name":"hostdev-pool-a-su-1","namespace":"nvidia-network-operator"},"spec":{"exclusions":[{"endIP":"10.117.127.31","startIP":"10.117.112.1"},{"endIP":"10.117.127.255","startIP":"10.117.127.128"}],"gateway":"10.117.112.1","nodeSelector":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"node-role.kubernetes.io/worker","operator":"Exists"}]}]},"perNodeBlockSize":8,"subnet":"10.117.112.0/20"}}
  creationTimestamp: "2025-09-18T17:39:30Z"
  generation: 2
  name: hostdev-pool-a-su-1
  namespace: nvidia-network-operator
  resourceVersion: "4911622"
  uid: 84ff5e43-4b4e-4432-b4af-f4c406314721
spec:
  exclusions:
  - endIP: 10.117.127.31
    startIP: 10.117.112.1
  - endIP: 10.117.127.255
    startIP: 10.117.127.128
  gateway: 10.117.112.1
  nodeSelector:
    nodeSelectorTerms:
    - matchExpressions:
      - key: node-role.kubernetes.io/worker
        operator: Exists
  perNodeBlockSize: 8
  subnet: 10.117.112.0/20
status:
  allocations:
  - endIP: 10.117.112.8
    nodeName: node-0a35d6d1
    startIP: 10.117.112.1
  - endIP: 10.117.112.16
    nodeName: node-c74b0c94
    startIP: 10.117.112.9

Reproducibility:
100% — occurs consistently

What are the minimal steps needed to reproduce the bug?

Anything else we need to know?

Component Versions

Logs

NVIDIA k8s IPAM Controller Logs (use kubectl logs $PODNAME)

[rollandf]

Hi,

The behavior is as expected.
The exclusions does not affect the range allocated to the nodes, it is only taken into consideration when a Pod request an IP, then the nv-ipam node will make sure that the allocated IP is not in the exclusions ranges.

See from README:

exclusions (optional, list): contains reserved IP addresses that should not be allocated by nv-ipam node component.

Do you think that the documentation should be updated to be clearer on that feature?

[aroynutanix]

Hi @rollandf, thanks for looking into this. I should have also included the pod event log. From our experience, we notice that when the start of the IP range falls within the exclusion range, the IP allocation always fails. In the example above, we tried bringing up a single pod, and no other pod is requesting a NIC from that network. Here is the relevant event log:

  Warning  FailedCreatePodSandBox  23s                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f46bbc9d80a660200c640c272f5eab496d0a3e26aea30becd8bbbc0efb4ce561": plugin type="multus" name="multus-cni-network" failed (add): [default/roce-test-server/1db43b86-079e-4ee7-b268-9e0e55c18b18:hostdev-rdma-device-default-a-su-1]: error adding container to network "hostdev-rdma-device-default-a-su-1": grpc call failed: rpc error: code = ResourceExhausted desc = no free addresses in the pool "hostdev-pool-a-su-1", poolType "ippool"

Everything works fine if I remove the exclusion block from the IPPool definition

[rollandf]

From your IPPool (8 IPs per nodes)
node-0a35d6d1 : 10.117.112.[1,2,3,4,5,6,7,8]
node-c74b0c94 : 10.117.112.[9,10,11,12,13,14,15,16]

Now, all of these IPs are falling into the first exclusion: 10.117.112.1 -> 10.117.127.31 (more than 3000 IPs are excluded)

exclusions:
  - endIP: 10.117.127.31
    startIP: 10.117.112.1

So, no IP are available to be assigned to pods (no free addresses in the pool) is as expected.

Can you explain what are you trying to achieve?

[aroynutanix]

Our internal network uses a /20 block. The first ~3,000 IP addresses are managed by an IT-controlled DHCP server, and the last ~1,000 IPs are reserved for testing purposes.

For my use case:

• The subnet is 10.117.112.0/20.

• I want to exclude the first 3,000 IPs (managed by DHCP) with:

  - startIP: 10.117.112.1
    endIP: 10.117.127.31

• Out of the remaining ~1,000 IPs, I want NV-IPAM to manage only 50 addresses. To do that, I’ll exclude everything above 10.117.127.127 using:

  - startIP: 10.117.127.128
    endIP: 10.117.127.255

That leaves 10.117.127.32 – 10.117.127.127 as the usable allocation range.
My expectation is that NV-IPAM will carve this usable range into contiguous perNodeBlockSize: 8 chunks for each node.