CNI issue on upgrade to 1.8.11 : invalid CIDR address: null/32

[the-nando]

Nomad version

Nomad v1.8.11+ent
BuildDate 2025-03-11T09:23:02Z
Revision f1d10f7f43b943002a505307ae896f8176c038e4+CHANGES

Issue

I run into an issue while upgrading some nodes from 1.8.5 to 1.8.11.

We use a CNI plugin for macvlan, named discovery and when restarting the agent as part of the upgrade, all existing allocations on the node are terminated with a fatal "Setup Failure" from the pre-run hook:

2025-03-24T19:04:22.722Z [WARN]  client.alloc_runner.runner_hook: failed to configure network: alloc_id=97055031-6928-3826-34c2-b5c955130551 error="plugin type=\"macvlan\" failed (add): failed to rename macvlan to \"eth0\": file exists" attempt=3
2025-03-24T19:04:22.722Z [ERROR] client.alloc_runner: prerun failed: alloc_id=97055031-6928-3826-34c2-b5c955130551 error="pre-run hook \"network\" failed: failed to configure networking for alloc: failed to configure network: plugin type=\"discovery\" failed (add): invalid CIDR address: null/32"

I haven't looked into the details yet but it's something added between 1.8.9 and 1.8.11 as I can upgrade to 1.8.9 without issues (min #25093) and consistently reproduce the issue by upgrading then to 1.8.11.

Reproduction steps

I can share the CNI plugin if that helps.

[ygersie]

Hey there, from what I can tell testing this, Nomad seems to wipe the infra containers completely on startup with the 1.8.11 upgrade. It then calls CNI_COMMAND=ADD for the newly created infra containers which aren't linked to existing runtimes as those were wiped. Instead of proceeding I've adjusted our plugin to abort/exit otherwise it leads to the null/32 assignment issue as no address could be handed out. But the issue is that our host-local call is trying to reassign an already existing address because the plugin is fed the same CNI_CONTAINERID as the one that's already running.

So tldr; Nomad wipes the infra containers on upgrade which is very problematic.

[tgross]

Thanks @ygersie. I've got this on my short list at the moment. I'll report back once I know more.

(Internal ref https://hashicorp.atlassian.net/browse/NET-12399)

[ygersie]

@tgross thanks! We’ve tested sequentially and the issue seems to have first appeared in 1.8.10. Hope that helps narrow it down.

[tgross]

I haven't been able to reproduce yet. But I suspect this is related to #24658 which shipped in Nomad 1.9.5 and Nomad Enterprise 1.8.10. Because Nomad tracks the allocation and CNI state in its state store, we needed to account for the case where the host is rebooted and all the actual network state is gone. So we're emitting the CNI CHECK command and recreating the network namespace if that fails. This should be a no-op on client restart if the host itself hasn't been restarted (at which point all the allocations are gone anyways).

However, we know that older default plugins don't work with this so we check their version before attempting it. Does your custom CNI plugin implement the CHECK command? It might help to have some of the debug-level logs from the host where this is happening -- you can see from #24658 that there should be quite a bit of that.

I can share the CNI plugin if that helps.

It definitely might!

[the-nando]

Thanks @tgross!!! That's the problem indeed: it doesn't implement CHECK as it's something which wasn't used before and it currently errors out when it gets that command.
I saw that commit but I didn't realise it was backported into 1.8 and I thought it landed in 1.9.5 only . I'll need to upgrade CNI version to 1.2.0 too

[tgross]

Good to hear. We normally try to avoid this kind of change in the LTS release. But that also suggests that we need to improve some of our documentation here (as well as flag this as an upgrade note in the upgrade guide). I'll get on that.

[the-nando]

I was reading again the commit you linked. Shouldn't it to fall back to the old behaviour without running CHECK via supportsCNICheck.Check(cniVersion)? We run with "cniVersion": "0.4.0", so I'd expect to see it falling back to the old behaviour but I don't see this message in the debug logs:

network namespace exists but could not check if networking is valid because bridge plugin version was <1.3.0: continuing anyways