After setup config to crio, crio can not stop container anymore.

[stefenson]

Hello there, I found a Fatal Error in nvidia-ctk.
After run the config command as README says:

nvidia-ctk runtime configure --runtime=crio --set-as-default --config=/etc/crio/crio.conf.d/99-nvidia.conf

I get the config file as the example says.
Then I restart my crio service.

systemctl restart crio

Everything looks like peaceful.
But when I restart some deploys or delete some pods in k8s, the process will stuck in Terminating.

NAME                      READY  STATUS       RESTARTS  AGE
...
backend-6b7945bc64-jqwl7  1/1    Running      0         19m
backend-ccfff5ccc-ktngm   1/1    Terminating  0         22m             <---------- After 19 minutes still running
...

After a long time, I finally found this config will cause this problem:

...
  [crio.runtime]
    default_runtime = "nvidia"
...

This config changes the crio default user to "nvidia" not root, so the permission blocks all the action that crio wants to do.

After delete this config, crio returns to normal, however the new container can not use nvidia plugin anymore.

Therefore, I have these questions:

• Why nvidia user is necessary?
• Why root user can not use nvidia driver in container?
• Any other way to setup config for crio that make it work funcationally?

It will be really helpful for any suggestion you provide.

Thank you very much! <3

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.

[elezar]

Our installation and configuration process for CRI-O has changed. See https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#configuring-cri-o

I am closing this issue. If you're still having problems after following these instructions with the latest version of the NVIDIA Container Toolkit, please reopen it or create a new issu.