GPU Operator breaks with Ubuntu Image on EKS 1.29

[mukgarg]

We have been using GPU operator on our EKS clusters with EKS Optimized AMI. This setup has been working fine so far. We updated our node groups to start using Ubuntu based ami for EKS from Canonical. Once we made this change we started seeing below errors in the nvidia-driver-ctr container inside the nvidia-driver-daemonset pod.

Errors from nvidia-driver-ctr container inside the nvidia-driver-daemonset pod:

DRIVER_ARCH is x86_64
Creating directory NVIDIA-Linux-x86_64-550.127.08
Verifying archive integrity... OK
Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 550.127.08..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

WARNING: Unable to determine the default X library path. The path /tmp/null/lib will be used, but this path was not detected in the ldconfig(8) cache, and no directory exists at this path, so it is likely that libraries installed there will not be found by the loader.


WARNING: You specified the '--no-kernel-modules' command line option, nvidia-installer will not install any kernel modules as part of this driver installation, and it will not remove existing NVIDIA kernel modules not part of an earlier NVIDIA driver installation.  Please ensure that NVIDIA kernel modules matching this driver version are installed separately.


========== NVIDIA Software Installer ==========

Starting installation of NVIDIA driver version 550.127.08 for Linux kernel version 5.15.0-1075-aws

Stopping NVIDIA persistence daemon...
Unloading NVIDIA driver kernel modules...
Unmounting NVIDIA driver rootfs...
Checking NVIDIA driver packages...
Updating the package cache...
Resolving Linux kernel version...
Proceeding with Linux kernel version 5.15.0-1075-aws
Installing Linux kernel headers...
Installing Linux kernel module files...
Generating Linux kernel version string...
Compiling NVIDIA driver kernel modules...
warning: the compiler differs from the one used to build the kernel
  The kernel was built by: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
  You are using:           cc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
cc: fatal error: Killed signal terminated program cc1
compilation terminated.
make[2]: *** [scripts/Makefile.build:297: /usr/src/nvidia-550.127.08/kernel/nvidia/nv-nano-timer.o] Error 1
make[2]: *** Waiting for unfinished jobs....
cc: fatal error: Killed signal terminated program cc1
compilation terminated.
make[2]: *** [scripts/Makefile.build:297: /usr/src/nvidia-550.127.08/kernel/nvidia/nv-pci.o] Error 1
cc: fatal error: Killed signal terminated program cc1
compilation terminated.
make[2]: *** [scripts/Makefile.build:297: /usr/src/nvidia-550.127.08/kernel/nvidia/nv-dmabuf.o] Error 1
cc: fatal error: Killed signal terminated program cc1
compilation terminated.
make[2]: *** [scripts/Makefile.build:297: /usr/src/nvidia-550.127.08/kernel/nvidia/nv.o] Error 1
cc: fatal error: Killed signal terminated program cc1
compilation terminated.
make[2]: *** [scripts/Makefile.build:297: /usr/src/nvidia-550.127.08/kernel/nvidia/nv-i2c.o] Error 1
cc: fatal error: Killed signal terminated program cc1
compilation terminated.
make[2]: *** [scripts/Makefile.build:297: /usr/src/nvidia-550.127.08/kernel/nvidia/nv-procfs.o] Error 1
cc: fatal error: Killed signal terminated program cc1
compilation terminated.
make[2]: *** [scripts/Makefile.build:297: /usr/src/nvidia-550.127.08/kernel/nvidia/nv-pat.o] Error 1
cc: fatal error: Killed signal terminated program cc1
compilation terminated.
make[2]: *** [scripts/Makefile.build:297: /usr/src/nvidia-550.127.08/kernel/nvidia/nv-cray.o] Error 1
cc: fatal error: Killed signal terminated program cc1
compilation terminated.
make[2]: *** [scripts/Makefile.build:297: /usr/src/nvidia-550.127.08/kernel/nvidia/nv-mmap.o] Error 1
cc: fatal error: Killed signal terminated program cc1
compilation terminated.
make[2]: *** [scripts/Makefile.build:297: /usr/src/nvidia-550.127.08/kernel/nvidia/nv-acpi.o] Error 1
cc: fatal error: Killed signal terminated program cc1
compilation terminated.
make[2]: *** [scripts/Makefile.build:297: /usr/src/nvidia-550.127.08/kernel/nvidia/nv-dma.o] Error 1
cc: fatal error: Killed signal terminated program cc1
compilation terminated.
make[2]: *** [scripts/Makefile.build:297: /usr/src/nvidia-550.127.08/kernel/nvidia/nv-p2p.o] Error 1
cc: fatal error: Killed signal terminated program cc1
compilation terminated.
make[2]: *** [scripts/Makefile.build:297: /usr/src/nvidia-550.127.08/kernel/nvidia/os-mlock.o] Error 1
cc: fatal error: Killed signal terminated program cc1
compilation terminated.
make[2]: *** [scripts/Makefile.build:297: /usr/src/nvidia-550.127.08/kernel/nvidia/os-usermap.o] Error 1
cc: fatal error: Killed signal terminated program cc1
compilation terminated.
make[2]: *** [scripts/Makefile.build:297: /usr/src/nvidia-550.127.08/kernel/nvidia/nv-vm.o] Error 1
cc: fatal error: Killed signal terminated program cc1
compilation terminated.
make[2]: *** [scripts/Makefile.build:297: /usr/src/nvidia-550.127.08/kernel/nvidia/nv-modeset-interface.o] Error 1
cc: fatal error: Killed signal terminated program cc1
compilation terminated.
make[2]: *** [scripts/Makefile.build:297: /usr/src/nvidia-550.127.08/kernel/nvidia/nv-usermap.o] Error 1
cc: fatal error: Killed signal terminated program cc1
compilation terminated.
make[2]: *** [scripts/Makefile.build:297: /usr/src/nvidia-550.127.08/kernel/nvidia/os-registry.o] Error 1
In file included from ./include/linux/thread_info.h:23,
                 from ./arch/x86/include/asm/preempt.h:7,
                 from ./include/linux/preempt.h:78,
                 from ./include/linux/spinlock.h:55,
                 from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-lock.h:29,
                 from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-linux.h:32,
                 from /usr/src/nvidia-550.127.08/kernel/nvidia/os-interface.c:27:
/usr/src/nvidia-550.127.08/kernel/common/inc/nv-mm.h: In function 'NV_GET_USER_PAGES':
./arch/x86/include/asm/current.h:18:17: warning: passing argument 1 of 'get_user_pages' makes integer from pointer without a cast [-Wint-conversion]
   18 | #define current get_current()
      |                 ^~~~~~~~~~~~~
      |                 |
      |                 struct task_struct *
/usr/src/nvidia-550.127.08/kernel/common/inc/nv-mm.h:109:31: note: in expansion of macro 'current'
  109 |         return get_user_pages(current, current->mm, start, nr_pages, write,
      |                               ^~~~~~~
In file included from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-pgprot.h:30,
                 from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-linux.h:33,
                 from /usr/src/nvidia-550.127.08/kernel/nvidia/os-interface.c:27:
./include/linux/mm.h:1853:35: note: expected 'long unsigned int' but argument is of type 'struct task_struct *'
 1853 | long get_user_pages(unsigned long start, unsigned long nr_pages,
      |                     ~~~~~~~~~~~~~~^~~~~
In file included from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-linux.h:34,
                 from /usr/src/nvidia-550.127.08/kernel/nvidia/os-interface.c:27:
/usr/src/nvidia-550.127.08/kernel/common/inc/nv-mm.h:109:47: warning: passing argument 2 of 'get_user_pages' makes integer from pointer without a cast [-Wint-conversion]
  109 |         return get_user_pages(current, current->mm, start, nr_pages, write,
In file included from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-pgprot.h:30,
                 from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-linux.h:33,
                 from /usr/src/nvidia-550.127.08/kernel/nvidia/os-interface.c:27:
./include/linux/mm.h:1853:56: note: expected 'long unsigned int' but argument is of type 'struct mm_struct *'
 1853 | long get_user_pages(unsigned long start, unsigned long nr_pages,
      |                                          ~~~~~~~~~~~~~~^~~~~~~~
In file included from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-linux.h:34,
                 from /usr/src/nvidia-550.127.08/kernel/nvidia/os-interface.c:27:
/usr/src/nvidia-550.127.08/kernel/common/inc/nv-mm.h:109:60: warning: passing argument 4 of 'get_user_pages' makes pointer from integer without a cast [-Wint-conversion]
  109 |         return get_user_pages(current, current->mm, start, nr_pages, write,
      |                                                            ^~~~~~~~
      |                                                            |
      |                                                            long unsigned int
In file included from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-pgprot.h:30,
                 from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-linux.h:33,
                 from /usr/src/nvidia-550.127.08/kernel/nvidia/os-interface.c:27:
./include/linux/mm.h:1854:46: note: expected 'struct page **' but argument is of type 'long unsigned int'
 1854 |        unsigned int gup_flags, struct page **pages,
      |                                ~~~~~~~~~~~~~~^~~~~
In file included from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-linux.h:34,
                 from /usr/src/nvidia-550.127.08/kernel/nvidia/os-interface.c:27:
/usr/src/nvidia-550.127.08/kernel/common/inc/nv-mm.h:109:70: warning: passing argument 5 of 'get_user_pages' makes pointer from integer without a cast [-Wint-conversion]
  109 |         return get_user_pages(current, current->mm, start, nr_pages, write,
      |                                                                      ^~~~~
      |                                                                      |
      |                                                                      int
In file included from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-pgprot.h:30,
                 from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-linux.h:33,
                 from /usr/src/nvidia-550.127.08/kernel/nvidia/os-interface.c:27:
./include/linux/mm.h:1855:32: note: expected 'struct vm_area_struct **' but argument is of type 'int'
 1855 |        struct vm_area_struct **vmas);
      |        ~~~~~~~~~~~~~~~~~~~~~~~~^~~~
In file included from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-linux.h:34,
                 from /usr/src/nvidia-550.127.08/kernel/nvidia/os-interface.c:27:
/usr/src/nvidia-550.127.08/kernel/common/inc/nv-mm.h:109:16: error: too many arguments to function 'get_user_pages'
  109 |         return get_user_pages(current, current->mm, start, nr_pages, write,
      |                ^~~~~~~~~~~~~~
In file included from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-pgprot.h:30,
                 from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-linux.h:33,
                 from /usr/src/nvidia-550.127.08/kernel/nvidia/os-interface.c:27:
./include/linux/mm.h:1853:6: note: declared here
 1853 | long get_user_pages(unsigned long start, unsigned long nr_pages,
      |      ^~~~~~~~~~~~~~
In file included from ./include/linux/thread_info.h:23,
                 from ./arch/x86/include/asm/preempt.h:7,
                 from ./include/linux/preempt.h:78,
                 from ./include/linux/spinlock.h:55,
                 from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-lock.h:29,
                 from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-linux.h:32,
                 from /usr/src/nvidia-550.127.08/kernel/nvidia/nv-vtophys.c:27:
/usr/src/nvidia-550.127.08/kernel/common/inc/nv-mm.h: In function 'NV_GET_USER_PAGES':
./arch/x86/include/asm/current.h:18:17: warning: passing argument 1 of 'get_user_pages' makes integer from pointer without a cast [-Wint-conversion]
   18 | #define current get_current()
      |                 ^~~~~~~~~~~~~
      |                 |
      |                 struct task_struct *
/usr/src/nvidia-550.127.08/kernel/common/inc/nv-mm.h:109:31: note: in expansion of macro 'current'
  109 |         return get_user_pages(current, current->mm, start, nr_pages, write,
      |                               ^~~~~~~
In file included from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-pgprot.h:30,
                 from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-linux.h:33,
                 from /usr/src/nvidia-550.127.08/kernel/nvidia/nv-vtophys.c:27:
./include/linux/mm.h:1853:35: note: expected 'long unsigned int' but argument is of type 'struct task_struct *'
 1853 | long get_user_pages(unsigned long start, unsigned long nr_pages,
      |                     ~~~~~~~~~~~~~~^~~~~
In file included from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-linux.h:34,
                 from /usr/src/nvidia-550.127.08/kernel/nvidia/nv-vtophys.c:27:
/usr/src/nvidia-550.127.08/kernel/common/inc/nv-mm.h:109:47: warning: passing argument 2 of 'get_user_pages' makes integer from pointer without a cast [-Wint-conversion]
  109 |         return get_user_pages(current, current->mm, start, nr_pages, write,
In file included from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-pgprot.h:30,
                 from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-linux.h:33,
                 from /usr/src/nvidia-550.127.08/kernel/nvidia/nv-vtophys.c:27:
./include/linux/mm.h:1853:56: note: expected 'long unsigned int' but argument is of type 'struct mm_struct *'
 1853 | long get_user_pages(unsigned long start, unsigned long nr_pages,
      |                                          ~~~~~~~~~~~~~~^~~~~~~~
In file included from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-linux.h:34,
                 from /usr/src/nvidia-550.127.08/kernel/nvidia/nv-vtophys.c:27:
/usr/src/nvidia-550.127.08/kernel/common/inc/nv-mm.h:109:60: warning: passing argument 4 of 'get_user_pages' makes pointer from integer without a cast [-Wint-conversion]
  109 |         return get_user_pages(current, current->mm, start, nr_pages, write,
      |                                                            ^~~~~~~~
      |                                                            |
      |                                                            long unsigned int
In file included from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-pgprot.h:30,
                 from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-linux.h:33,
                 from /usr/src/nvidia-550.127.08/kernel/nvidia/nv-vtophys.c:27:
./include/linux/mm.h:1854:46: note: expected 'struct page **' but argument is of type 'long unsigned int'
 1854 |        unsigned int gup_flags, struct page **pages,
      |                                ~~~~~~~~~~~~~~^~~~~
In file included from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-linux.h:34,
                 from /usr/src/nvidia-550.127.08/kernel/nvidia/nv-vtophys.c:27:
/usr/src/nvidia-550.127.08/kernel/common/inc/nv-mm.h:109:70: warning: passing argument 5 of 'get_user_pages' makes pointer from integer without a cast [-Wint-conversion]
  109 |         return get_user_pages(current, current->mm, start, nr_pages, write,
      |                                                                      ^~~~~
      |                                                                      |
      |                                                                      int
In file included from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-pgprot.h:30,
                 from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-linux.h:33,
                 from /usr/src/nvidia-550.127.08/kernel/nvidia/nv-vtophys.c:27:
./include/linux/mm.h:1855:32: note: expected 'struct vm_area_struct **' but argument is of type 'int'
 1855 |        struct vm_area_struct **vmas);
      |        ~~~~~~~~~~~~~~~~~~~~~~~~^~~~
In file included from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-linux.h:34,
                 from /usr/src/nvidia-550.127.08/kernel/nvidia/nv-vtophys.c:27:
/usr/src/nvidia-550.127.08/kernel/common/inc/nv-mm.h:109:16: error: too many arguments to function 'get_user_pages'
  109 |         return get_user_pages(current, current->mm, start, nr_pages, write,
      |                ^~~~~~~~~~~~~~
In file included from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-pgprot.h:30,
                 from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-linux.h:33,
                 from /usr/src/nvidia-550.127.08/kernel/nvidia/nv-vtophys.c:27:
./include/linux/mm.h:1853:6: note: declared here
 1853 | long get_user_pages(unsigned long start, unsigned long nr_pages,
      |      ^~~~~~~~~~~~~~
In file included from ./include/linux/thread_info.h:23,
                 from ./arch/x86/include/asm/preempt.h:7,
                 from ./include/linux/preempt.h:78,
                 from ./include/linux/spinlock.h:55,
                 from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-lock.h:29,
                 from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-linux.h:32,
                 from /usr/src/nvidia-550.127.08/kernel/nvidia/os-pci.c:27:
/usr/src/nvidia-550.127.08/kernel/common/inc/nv-mm.h: In function 'NV_GET_USER_PAGES':
./arch/x86/include/asm/current.h:18:17: warning: passing argument 1 of 'get_user_pages' makes integer from pointer without a cast [-Wint-conversion]
   18 | #define current get_current()
      |                 ^~~~~~~~~~~~~
      |                 |
      |                 struct task_struct *
/usr/src/nvidia-550.127.08/kernel/common/inc/nv-mm.h:109:31: note: in expansion of macro 'current'
  109 |         return get_user_pages(current, current->mm, start, nr_pages, write,
      |                               ^~~~~~~
In file included from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-pgprot.h:30,
                 from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-linux.h:33,
                 from /usr/src/nvidia-550.127.08/kernel/nvidia/os-pci.c:27:
./include/linux/mm.h:1853:35: note: expected 'long unsigned int' but argument is of type 'struct task_struct *'
 1853 | long get_user_pages(unsigned long start, unsigned long nr_pages,
      |                     ~~~~~~~~~~~~~~^~~~~
In file included from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-linux.h:34,
                 from /usr/src/nvidia-550.127.08/kernel/nvidia/os-pci.c:27:
/usr/src/nvidia-550.127.08/kernel/common/inc/nv-mm.h:109:47: warning: passing argument 2 of 'get_user_pages' makes integer from pointer without a cast [-Wint-conversion]
  109 |         return get_user_pages(current, current->mm, start, nr_pages, write,
In file included from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-pgprot.h:30,
                 from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-linux.h:33,
                 from /usr/src/nvidia-550.127.08/kernel/nvidia/os-pci.c:27:
./include/linux/mm.h:1853:56: note: expected 'long unsigned int' but argument is of type 'struct mm_struct *'
 1853 | long get_user_pages(unsigned long start, unsigned long nr_pages,
      |                                          ~~~~~~~~~~~~~~^~~~~~~~
In file included from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-linux.h:34,
                 from /usr/src/nvidia-550.127.08/kernel/nvidia/os-pci.c:27:
/usr/src/nvidia-550.127.08/kernel/common/inc/nv-mm.h:109:60: warning: passing argument 4 of 'get_user_pages' makes pointer from integer without a cast [-Wint-conversion]
  109 |         return get_user_pages(current, current->mm, start, nr_pages, write,
      |                                                            ^~~~~~~~
      |                                                            |
      |                                                            long unsigned int
In file included from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-pgprot.h:30,
                 from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-linux.h:33,
                 from /usr/src/nvidia-550.127.08/kernel/nvidia/os-pci.c:27:
./include/linux/mm.h:1854:46: note: expected 'struct page **' but argument is of type 'long unsigned int'
 1854 |        unsigned int gup_flags, struct page **pages,
      |                                ~~~~~~~~~~~~~~^~~~~
In file included from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-linux.h:34,
                 from /usr/src/nvidia-550.127.08/kernel/nvidia/os-pci.c:27:
/usr/src/nvidia-550.127.08/kernel/common/inc/nv-mm.h:109:70: warning: passing argument 5 of 'get_user_pages' makes pointer from integer without a cast [-Wint-conversion]
  109 |         return get_user_pages(current, current->mm, start, nr_pages, write,
      |                                                                      ^~~~~
      |                                                                      |
      |                                                                      int
In file included from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-pgprot.h:30,
                 from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-linux.h:33,
                 from /usr/src/nvidia-550.127.08/kernel/nvidia/os-pci.c:27:
./include/linux/mm.h:1855:32: note: expected 'struct vm_area_struct **' but argument is of type 'int'
 1855 |        struct vm_area_struct **vmas);
      |        ~~~~~~~~~~~~~~~~~~~~~~~~^~~~
In file included from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-linux.h:34,
                 from /usr/src/nvidia-550.127.08/kernel/nvidia/os-pci.c:27:
/usr/src/nvidia-550.127.08/kernel/common/inc/nv-mm.h:109:16: error: too many arguments to function 'get_user_pages'
  109 |         return get_user_pages(current, current->mm, start, nr_pages, write,
      |                ^~~~~~~~~~~~~~
In file included from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-pgprot.h:30,
                 from /usr/src/nvidia-550.127.08/kernel/common/inc/nv-linux.h:33,
                 from /usr/src/nvidia-550.127.08/kernel/nvidia/os-pci.c:27:
./include/linux/mm.h:1853:6: note: declared here
 1853 | long get_user_pages(unsigned long start, unsigned long nr_pages,
      |      ^~~~~~~~~~~~~~
cc: fatal error: Killed signal terminated program cc1
compilation terminated.
make[2]: *** [scripts/Makefile.build:297: /usr/src/nvidia-550.127.08/kernel/nvidia/os-interface.o] Error 1
cc: fatal error: Killed signal terminated program cc1
compilation terminated.
make[2]: *** [scripts/Makefile.build:297: /usr/src/nvidia-550.127.08/kernel/nvidia/nv-vtophys.o] Error 1
make[2]: *** [scripts/Makefile.build:297: /usr/src/nvidia-550.127.08/kernel/nvidia/os-pci.o] Error 1
make[1]: *** [Makefile:1910: /usr/src/nvidia-550.127.08/kernel] Error 2
make: *** [Makefile:89: modules] Error 2
Stopping NVIDIA persistence daemon...
Unloading NVIDIA driver kernel modules...
Unmounting NVIDIA driver rootfs...

Setup details:

Kubernetes version: v1.28
GPU Operator version: v24.9.1
GPU Driver version: 550.127.08
Ubuntu AMI Name: ubuntu-eks/k8s_1.28/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20250123
Kernel Version: 5.15.0-1075-aws

Helm values used for installing GPU Operator:

dcgmExporter:
                  config:
                    name: nvidia-metrics-config
                  env:
                  - name: DCGM_EXPORTER_COLLECTORS
                    value: /etc/dcgm-exporter/dcgm-metrics.csv
                  tolerations:
                  - key: nvidia.com/gpu
                    operator: Exists
                  resources:
                    requests:
                      cpu: 100m
                      memory: 384Mi
                    limits:
                      cpu: 200m
                      memory: 460Mi
                devicePlugin:
                  tolerations:
                  - key: nvidia.com/gpu
                    operator: Exists
                driver:
                  # Prevent the GPU operator from appending the OS version to the image tag by
                  # specifying it as a digest, based on information in this comment:
                  # https://github.com/NVIDIA/gpu-operator/issues/542#issuecomment-1612215289
                  # This digest value can be determined with crane as follows:
                  # crane digest nvcr.io/nvidia/driver:550.127.08-ubuntu20.04
                  # This will need to be changed when we upgrade to a newer version of the operator and host OS
                  version: sha256:042659b349b3d6d915e551302bce80d2b799918fec31f747ae6a7c7ee4a9fc97
                  upgradePolicy:
                    autoUpgrade: false
                  tolerations:
                  - key: nvidia.com/gpu
                    operator: Exists
                gfd:
                  tolerations:
                  - key: nvidia.com/gpu
                    operator: Exists
                operator:
                  resources:
                    requests:
                      cpu: 400m
                      memory: 800Mi
                    limits:
                      cpu: 500m
                      memory: 1000Mi
                toolkit:
                  env:
                  - name: CONTAINERD_SET_AS_DEFAULT
                    value: "true"
                psp:
                  enabled: false
                node-feature-discovery:
                  # Override name to avoid hitting 63 character limits
                  fullnameOverride: "nfd"
                  master:
                    resources:
                      requests:
                        cpu: 400m
                        memory: 800Mi
                      limits:
                        cpu: 500m
                        memory: 1000Mi
                  worker:
                    nodeSelector:
                      node.abakus.volvocars.ai/has-gpu: "true"
                    tolerations:
                    - key: nvidia.com/gpu
                      operator: Exists
                validator:
                  plugin:
                    env:
                    - name: WITH_WORKLOAD
                      value: "false"

Current status of PODs:

❯ kubectl get pods -n abakus-gpu-support
NAME                                       READY   STATUS     RESTARTS        AGE
gpu-feature-discovery-lcsjj                0/1     Init:0/2   0               53m
nfd-worker-nnzdk                           1/1     Running    0               54m
nvidia-container-toolkit-daemonset-tv9d4   0/1     Init:0/1   0               53m
nvidia-dcgm-exporter-hgqqf                 0/1     Init:0/1   0               53m
nvidia-device-plugin-daemonset-4sd7c       0/1     Init:0/1   0               53m
nvidia-driver-daemonset-tfhwm              0/1     Running    7 (3m51s ago)   54m
nvidia-operator-validator-klzrj            0/1     Init:0/4   0               53m

Events from GPU Operator validator pod:

Events:
  Type     Reason                  Age                  From     Message
  ----     ------                  ----                 ----     -------
  Warning  FailedCreatePodSandBox  28s (x303 over 65m)  kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox runtime: no runtime for "nvidia" is configured

Events from GPU Driver daemonset pod:

Events:
  Type     Reason     Age                    From     Message
  ----     ------     ----                   ----     -------
  Warning  Unhealthy  2m41s (x310 over 64m)  kubelet  Startup probe failed: NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

We suspect that this issue is due to this image being used by us: nvcr.io/nvidia/driver:550.127.08-ubuntu20.04. We couldn't find any precompiled image for aws for 5.15.0-1075-aws kernel version here which may resolve this issue for us. Could someone please confirm what is it that we need to change for Ubuntu images to work in our setup?

[tariq1890]

We couldn't find any precompiled image for aws for 5.15.0-1075-aws kernel version here

The precompiled driver for the 5.15.0-1076-aws kernel has been published. Can you try updating your kernel to that version and retrying? Be sure to set the following flags to enabled precompiled drivers

--set driver.usePrecompiled=true --set driver.version=550

[mukgarg]

We couldn't find any precompiled image for aws for 5.15.0-1075-aws kernel version here

The precompiled driver for the 5.15.0-1076-aws kernel has been published. Can you try updating your kernel to that version and retrying? Be sure to set the following flags to enabled precompiled drivers

--set driver.usePrecompiled=true --set driver.version=550

@tariq1890 - We had to upgrade to EKS 1.29 and we did. We tried using this image: nvcr.io/nvidia/driver@sha256:042659b349b3d6d915e551302bce80d2b799918fec31f747ae6a7c7ee4a9fc97, which is same as nvcr.io/nvidia/driver:550.127.08-ubuntu20.04. We are getting
below messages now. Could you please suggest what image should be used to make this work?

❯ kubectl logs -f nvidia-driver-daemonset-sv46z -n abakus-gpu-support
DRIVER_ARCH is x86_64
Creating directory NVIDIA-Linux-x86_64-550.127.08
Verifying archive integrity... OK
Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 550.127.08..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

WARNING: Unable to determine the default X library path. The path /tmp/null/lib will be used, but this path was not detected in the ldconfig(8) cache, and no directory exists at this path, so it is likely that libraries installed there will not be found by the loader.


WARNING: You specified the '--no-kernel-modules' command line option, nvidia-installer will not install any kernel modules as part of this driver installation, and it will not remove existing NVIDIA kernel modules not part of an earlier NVIDIA driver installation.  Please ensure that NVIDIA kernel modules matching this driver version are installed separately.


========== NVIDIA Software Installer ==========

Starting installation of NVIDIA driver version 550.127.08 for Linux kernel version 6.8.0-1021-aws

Stopping NVIDIA persistence daemon...
Unloading NVIDIA driver kernel modules...
Unmounting NVIDIA driver rootfs...
Checking NVIDIA driver packages...
Updating the package cache...
Resolving Linux kernel version...
Could not resolve Linux kernel version
Stopping NVIDIA persistence daemon...
Unloading NVIDIA driver kernel modules...
Unmounting NVIDIA driver rootfs...

Below are the details of the aws AMI we are using for testing this and all the other amis on the cluster running general purpose workload are the same.

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
$ uname -a
Linux ip-10-63-194-91 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

[mukgarg]

@tariq1890 - Tried below option with the Operator that results into the following error, since the image is not present in the catalog.

--set driver.usePrecompiled=true --set driver.version=550

Failed to pull image "nvcr.io/nvidia/driver:550-6.8.0-1021-aws-ubuntu22.04": rpc error: code = NotFound desc = failed to pull and unpack image "nvcr.io/nvidia/driver:550-6.8.0-1021-aws-ubuntu22.04": failed to resolve reference "nvcr.io/nvidia/driver:550-6.8.0-1021-aws-ubuntu22.04": nvcr.io/nvidia/driver:550-6.8.0-1021-aws-ubuntu22.04: not found

[mukgarg]

Tried building our image following the instructions mentioned here. However, that ends up with below. Just to add we were using Amazon Linux so far but the only drawback is that we needed to rely on the driver provided by the underlying machine. Not sure, what should be the best way forward at this point but would be beneficial to get some expert guidance on this.

[+] Building 3.5s (11/20)                                                                                                                                                                                                        
 => [internal] load build definition from Dockerfile                                                                                                                                                                        0.0s
 => => transferring dockerfile: 3.45kB                                                                                                                                                                                      0.0s
 => [internal] load metadata for nvcr.io/nvidia/cuda:12.4.0-base-ubuntu22.04                                                                                                                                                1.6s
 => [internal] load .dockerignore                                                                                                                                                                                           0.0s
 => => transferring context: 2B                                                                                                                                                                                             0.0s
 => [ 1/16] FROM nvcr.io/nvidia/cuda:12.4.0-base-ubuntu22.04@sha256:80d4d9ac041242f6ae5d05f9be262b3374e0e0b8bb5a49c6c3e94e192cde4a44                                                                                        0.0s
 => [internal] load build context                                                                                                                                                                                           0.0s
 => => transferring context: 35B                                                                                                                                                                                            0.0s
 => CACHED [ 2/16] RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections                                                                                                                       0.0s
 => CACHED [ 3/16] RUN apt-key del 7fa2af80 &&     apt-key adv --fetch-keys "https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/3bf863cc.pub"                                                       0.0s
 => [ 4/16] RUN cat /etc/apt/sources.list                                                                                                                                                                                   0.1s
 => [ 5/16] RUN sed -i 's|http://ports.ubuntu.com|http://archive.ubuntu.com|g' /etc/apt/sources.list                                                                                                                        0.1s
 => [ 6/16] RUN cat /etc/apt/sources.list                                                                                                                                                                                   0.1s 
 => ERROR [ 7/16] RUN dpkg --add-architecture i386 &&     apt-get update && apt-get install -y --no-install-recommends         apt-utils         build-essential         ca-certificates         curl         kmod          1.4s 
------                                                                                                                                                                                                                           
 > [ 7/16] RUN dpkg --add-architecture i386 &&     apt-get update && apt-get install -y --no-install-recommends         apt-utils         build-essential         ca-certificates         curl         kmod         file         libelf-dev         libglvnd-dev         pkg-config &&     rm -rf /var/lib/apt/lists/*:                                                                                                                                           
#6 0.262 Ign:1 http://archive.ubuntu.com/ubuntu-ports jammy InRelease                                                                                                                                                            
#6 0.307 Ign:2 http://archive.ubuntu.com/ubuntu-ports jammy-updates InRelease                                                                                                                                                    
#6 0.353 Ign:3 http://archive.ubuntu.com/ubuntu-ports jammy-backports InRelease                                                                                                                                                  
#6 0.401 Ign:4 http://archive.ubuntu.com/ubuntu-ports jammy-security InRelease
#6 0.457 Err:5 http://archive.ubuntu.com/ubuntu-ports jammy Release
#6 0.457   404  Not Found [IP: 185.125.190.81 80]
#6 0.502 Err:6 http://archive.ubuntu.com/ubuntu-ports jammy-updates Release
#6 0.502   404  Not Found [IP: 185.125.190.81 80]
#6 0.542 Err:7 http://archive.ubuntu.com/ubuntu-ports jammy-backports Release
#6 0.542   404  Not Found [IP: 185.125.190.81 80]
#6 0.587 Err:8 http://archive.ubuntu.com/ubuntu-ports jammy-security Release
#6 0.587   404  Not Found [IP: 185.125.190.81 80]
#6 1.025 Get:9 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/sbsa  InRelease [1579 B]
#6 1.156 Get:10 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/sbsa  Packages [1110 kB]
#6 1.356 Reading package lists...
#6 1.388 E: The repository 'http://archive.ubuntu.com/ubuntu-ports jammy Release' does not have a Release file.
#6 1.388 E: The repository 'http://archive.ubuntu.com/ubuntu-ports jammy-updates Release' does not have a Release file.
#6 1.388 E: The repository 'http://archive.ubuntu.com/ubuntu-ports jammy-backports Release' does not have a Release file.
#6 1.388 E: The repository 'http://archive.ubuntu.com/ubuntu-ports jammy-security Release' does not have a Release file.
------
process "/bin/sh -c dpkg --add-architecture i386 &&     apt-get update && apt-get install -y --no-install-recommends         apt-utils         build-essential         ca-certificates         curl         kmod         file         libelf-dev         libglvnd-dev         pkg-config &&     rm -rf /var/lib/apt/lists/*" did not complete successfully: exit code: 100`

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.

[rahulait]

Hi @mukgarg, are you still hitting this issue with the latest version of gpu-operator?

Given this issue has been open for over 6 months now with no recent updates, the context may now be outdated. To keep the issue tracker clean and focused on current and actionable topics, I am going to close this issue. If you still see this issue with latest version of gpu-operator, please re-open it so that someone from team can take a look.