Support for running the driver with fake devices

[mortent]

It can be hard to get access to enough devices for testing at scale and it can get very expensive. So for scale testing, it would be useful to be able to run the NVIDIA DRA driver on regular non-accelerator nodes, but make the driver pretend that it does have access to GPUs.

This means that it would publish ResourceSlices that would be as close as possible to what it would look like on real hardware and handle the allocation for scheduled pods so they get to the running state.

[klueska]

We could accomplish this through a combination of a mock NVML implementation and the management of fake device nodes on the underlying machine to back the devices reported b NVML.

[mortent]

One of the scenarios we are looking to test is alignment between GPU and NIC. So we would like to be able to make the driver expose devices with the resource.kubernetes.io/pcieRoot attribute. My guess is that we need to to add specific support for this.

[klueska]

Yes, that would need to be mocked out as well.

I was thinking maybe we could just provide a mode where the resourceslices are read instead of generated (which seems more straightforward on the surface=, however if we want to test dynamic MIG at some point in the future, this will make doing that more difficult.

[mortent]

Allowing ResourceSlices to be read rather than generated could be a good first step, although I agree that it would probably be even better if the input was somehow the actual physical devices (maybe with some additional information about the topology) and that the ResourceSlices would be generated based on that. It would be useful when we get to MIGs, but it would also make sure that things like attributes and capacities during testing reflects what they would look like with real devices.

[pohly]

We also discussed changing the DRA example driver to emulate "real" hardware by publishing the corresponding ResourceSlices. See kubernetes-sigs/dra-example-driver#92.