Error trying to use NVIDIA runtime with rootless Docker

[muety]

I got a rootless Docker setup on Ubuntu 20.04 and set no-cgroups = true in /etc/nvidia-container-runtime/config.toml. However, when trying to run

docker run --rm --gpus all nvidia/cuda:12.0.0-base-ubuntu20.04 nvidia-smi

I'm getting the following error:

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: privilege change failed: invalid argument: unknown.

Possibly related to #85.

Inspired by NVIDIA/nvidia-docker#1565 (comment), I realized that the above issue only occurs for users logging in via LDAP (using SSSD), not for "real" local user. Those users have very high (> 65536) IDs. Might be a coincidence, though.

Any ideas how to work around this?

[dnns92]

I have a similar issue, see: #106. Where you able to resolve it?

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.

[ArangoGutierrez]

The error privilege change failed: invalid argument when using high UIDs (> 65536) from LDAP/SSSD was a known bug in older versions of libnvidia-container. The logic for setting up the user namespace in rootless mode did not correctly handle 32-bit UIDs.

This issue was resolved in subsequent releases. Full support for rootless Docker, including fixes for user namespaces and high UIDs, was introduced and stabilized in versions 1.16.0 and 1.17.0.

We recommend upgrading to the latest version of the nvidia-container-toolkit to get all rootless-related fixes.

We are closing this issue as resolved.