CUDA_MPS_PINNED_DEVICE_MEM_LIMIT is not set when using MPS

[harche]

Setting CUDA_MPS_PINNED_DEVICE_MEM_LIMIT before starting the MPS control daemon on the host would be helpful in enforcing the memory boundaries of the MPS partition. Should CUDA_MPS_PINNED_DEVICE_MEM_LIMIT also be set in the MPS control daemon template?

/cc @klueska

[harche]

Actually you are setting it here

k8s-dra-driver-gpu/templates/mps-control-daemon.tmpl.yaml

Line 37 in eb5df6e

echo set_default_device_pinned_mem_limit {{ $id }} {{ $limit }} | nvidia-cuda-mps-control

[harche]

I don't see DefaultPinnedDeviceMemoryLimits getting set a value.

[harche]

Oh it looks like part of MPS config

k8s-dra-driver-gpu/api/nvidia.com/resource/v1beta1/sharing.go

Line 85 in eb5df6e

DefaultPinnedDeviceMemoryLimit *resource.Quantity `json:"defaultPinnedDeviceMemoryLimit,omitempty"`