Driver support for NVIDIA RTX Pro 6000 Blackwell Server Edition cards

[SurealCereal]

The currently included NVIDIA driver 570.148.08 does not support the NVIDIA RTX Pro 6000 Blackwell Server Edition.

Release notes for each driver include a section Supported NVIDIA Data Center GPUs.
https://docs.nvidia.com/datacenter/tesla/tesla-release-notes-570-148-08/index.html ← In Talos 1.10.5
https://docs.nvidia.com/datacenter/tesla/tesla-release-notes-570-158-01/index.html ← Added last week via PR.
https://docs.nvidia.com/datacenter/tesla/tesla-release-notes-570-172-08/index.html ← No support here either.

Only the very latest release supports this fairly new GPU.
https://docs.nvidia.com/datacenter/tesla/tesla-release-notes-575-57-08/index.html

It seems that only LTS and Production branches are added, while the newer hardware support waits in the "New Feature Branch" driver releases.

Is there another way to install this new driver? Are there plans to include NVIDIA "New Feature Branch" drivers?

[smira]

What is the NVIDIA policy on support for this GPU? Will it be included into 570.x? Or once 575.x becomes production?

[SurealCereal]

@smira As far as I can tell, support will not be back-ported to the 570.x series. There will be a new Data Center production branch based off of 575.x.

The release cadence for production branches is twice per year, which means we are due for a new production branch, as 570 has been with us since at least January 2025, but its unclear to me when that will happen. The NVIDIA website is very light on concrete info about this card; there is no proper data sheet published yet for example.

[smira]

As you are NVIDIA customer, I think it would be nice to ask this question to NVIDIA directly, and get the idea of it.

Our current policy is to build production and LTS branches, and we are happy to move to 575.x as soon as it becomes production.

It is also double-edged sword if 575.x removes support for some older hardware.

[texmachina]

@smira - checking in on an update here. We are hard locked, unable to use new GPU hardware with Talos instances. Alternatively, is there a Talos guide on custom driver installs so that we can be unblocked here?

[frezbo]

Talos 1.12 (alpha's already released) is already on 580

[texmachina]

Thanks @frezbo - any way to deploy a node or cluster at 1.12 using Omni?

[frezbo]

raise a ticket at support.siderolabs.com to support pre-release versions (though I'm not sure if it's supported for hobby tiers too)

[texmachina]

Thanks, @frezbo - I'll just try and build an OSS/non-Omni Talos cluster for the time being on 1.12 alpha. Noticed #836 - which would be the ultimate if possible. Biggest impediment for us migrating from our current k8s infra.