Fix ldcache update when host and container distributions do not match #1444

elezar · 2025-11-11T11:42:05Z

This change fixes the update of the ldcache when running a non-Debian-like (e.g. RHEL, Fedora) container on a Debian-like (e.g. Debian, Ubuntua) system. Here the ldcache in the container does not initially include system libraries from /lib64 or /usr/lib64 since this ldconfig from the host that is executed in the container's namespace does not process those paths by default.

Testing

The expected behaviour when using runc directly:

$ docker run --rm --runtime=runc redhat/ubi9 bash -c "ldconfig -p | grep libc.so."
        libc.so.6 (libc6,AArch64, OS ABI: Linux 3.7.0) => /lib64/libc.so.6

The current behaviour:

$ docker run --rm --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=all redhat/ubi9 bash -c "ldconfig -p | grep libc.so."

(note that libc.so.6 is not present in the ldcache since /lib64 (linked to /usr/lib64) is not processed).

The updated behaviour:

$ docker run --rm --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=all redhat/ubi9 bash -c "ldconfig -p | grep libc.so."
        libc.so.6 (libc6,AArch64, OS ABI: Linux 3.7.0) => /lib64/libc.so.6

elezar · 2025-11-11T11:44:20Z

/cherry-pick release-1.18

cdesiniotis

I left a couple of comments / questions, but no blockers.

internal/ldconfig/ldconfig.go

tests/e2e/nvidia-container-toolkit_test.go

internal/ldconfig/ldconfig.go

jfroy · 2025-11-14T19:15:56Z

internal/ldconfig/ldconfig.go

-func (l *Ldconfig) getSystemSerachPaths() []string {
+func (l *Ldconfig) getSystemSearchPaths() []string {
 	if l.isDebianLikeContainer {
 		debianSystemSearchPaths()


I've been debugging issues on my homelab cluster with the latest CTK release, and I just spotted this code here. I think it's missing a return. I can send a PR for the existing code, or we can roll that fix into this PR.

Thanks for the fix. In order to minimize backport churn, let's add the commit to this PR.

jfroy · 2025-11-17T18:50:04Z

I'm preparing a follow up / alternative to this patch to handle additional cases I've ran into recently. Should have it out as a branch in an hour or so.

Additional commits.

ArangoGutierrez · 2025-11-20T13:46:19Z

internal/ldconfig/ldconfig.go

+// TODO: Add other architectures that have custom `add_system_dir` macros (e.g. riscv)
+// TODO: Replace with executing the container's dynamlic linker with `--list-diagnostics`?


Once we merge this PR we should turn this into GitHub issues to track this TODO's and get them implemented

ArangoGutierrez · 2025-11-20T13:46:31Z

internal/ldconfig/ldconfig.go

+// TODO: Add other architectures that have custom `add_system_dir` macros (e.g. riscv)
+// TODO: Replace with executing the container's dynamlic linker with `--list-diagnostics`?


Once we merge this PR we should turn this into GitHub issues to track this TODO's and get them implemented

ArangoGutierrez · 2025-11-20T14:05:05Z

Thanks a lot for your contributions on this topic @jfroy !

jgehrcke · 2025-11-20T14:10:37Z

internal/ldconfig/ldconfig.go

 	ldsoconfdFilenamePattern = "00-nvcr-*.conf"
+	// ldsoconfdSystemDirsFilenamePattern specifies the filename pattern for the drop-in conf file
+	// that includes the expected system directories for the container.
+	// This is chosen to have a high likelihood of being lexographically last in


"lexicographically" :)

We could also use the prefix "yolo-" 🤡

I believe this was me. Updating this and references to Debian and non-Debian for clarity.

jgehrcke

Fascinating, thank you.

non-debian (including ubuntu)

From the perspective of this PR, Ubuntu is actually non-Debian?

elezar · 2025-11-20T15:43:18Z

Fascinating, thank you.

non-debian (including ubuntu)

From the perspective of this PR, Ubuntu is actually non-Debian?

No. Ubuntu is debian in this context. Let me update the description to be clearer.

jfroy · 2025-11-20T15:43:19Z

Fascinating, thank you.

non-debian (including ubuntu)

From the perspective of this PR, Ubuntu is actually non-Debian?

@elezar is that what you meant (I think this is quoting #1444 (comment))?

@jgehrcke Ubuntu is "debian-like" because /etc/debian_version exists on Ubuntu systems. This PR does not change that.

This change ensures that updating the ldcache in a container also includes the system search paths for the container ldconfig. In most cases, the hook will be executing a host ldconfig that may be configured widely differently from what the container image expects. The common case is Debian-like (e.g. Debian, Ubuntu) vs non-Debian-like (e.g. RHEL, Fedora). But there are also hosts that configure ldconfig to search in a glibc prefix (e.g. /usr/lib/glibc). To avoid all these cases, write the container's expected system search paths to a drop-in conf file that is likely to be last in lexicographic order. Entries in the top-level ld.so.conf file may be processed after this drop-in, but this hook does not modify the top-level file if it exists. Signed-off-by: Jean-Francois Roy <[email protected]> Signed-off-by: Evan Lezar <[email protected]>

elezar · 2025-11-20T15:52:27Z

@elezar is that what you meant (I think this is quoting #1444 (comment))?

Yes, the code also used Debian and non-Debian in comments (and a function name). I have updated to use Debian-like and non-Debian-like in the comments with examples of distros to make this clearer.

github-actions · 2025-11-20T20:17:46Z

🤖 Backport PR created for release-1.18: #1488 ✅

elezar added this to the v1.18.1 milestone Nov 11, 2025

github-actions bot added the cherry-pick/release-1.18 label Nov 11, 2025

elezar force-pushed the fix-ldcache-update-on-non-debian branch 2 times, most recently from 9a32323 to fb95f58 Compare November 11, 2025 12:36

ArangoGutierrez self-requested a review November 11, 2025 13:30

ArangoGutierrez assigned elezar Nov 11, 2025

elezar force-pushed the fix-ldcache-update-on-non-debian branch 3 times, most recently from aa0c6fa to a854d3d Compare November 12, 2025 07:22

elezar requested review from cdesiniotis and tariq1890 November 12, 2025 12:32

cdesiniotis approved these changes Nov 14, 2025

View reviewed changes

internal/ldconfig/ldconfig.go Show resolved Hide resolved

internal/ldconfig/ldconfig.go Outdated Show resolved Hide resolved

tests/e2e/nvidia-container-toolkit_test.go Outdated Show resolved Hide resolved

internal/ldconfig/ldconfig.go Outdated Show resolved Hide resolved

elezar force-pushed the fix-ldcache-update-on-non-debian branch 2 times, most recently from 2794eaa to 0b28791 Compare November 14, 2025 10:49

jfroy reviewed Nov 14, 2025

View reviewed changes

This was referenced Nov 15, 2025

Add missing return in getSystemSerachPaths for debian like containers #1457

Closed

ldconfig: Determine container "debian-ness" after root pivot #1458

Closed

ldconfig: Create ld.so.conf file if missing #1459

Merged

elezar mentioned this pull request Nov 17, 2025

Some containers worked well with legacy mode but no longer work since the switch to CDI mode in version 1.18.0 #1456

Open

elezar force-pushed the fix-ldcache-update-on-non-debian branch from f432507 to 70cca03 Compare November 17, 2025 15:50

ArangoGutierrez previously approved these changes Nov 17, 2025

View reviewed changes

jfroy mentioned this pull request Nov 17, 2025

Write system paths to lexicographically last ld.so.conf.d drop-in #1462

Closed

cdesiniotis previously approved these changes Nov 18, 2025

View reviewed changes

elezar mentioned this pull request Nov 18, 2025

fixes to ldconfig update #1466

Merged

elezar force-pushed the fix-ldcache-update-on-non-debian branch 2 times, most recently from 7f73f39 to 22d7b5d Compare November 19, 2025 10:32

elezar changed the title ~~Fix ldcache update on non debian systems~~ Fix ldcache update when host and container distributions do not match Nov 19, 2025

elezar force-pushed the fix-ldcache-update-on-non-debian branch 2 times, most recently from d3dcb86 to 1e52761 Compare November 19, 2025 13:51

jfroy mentioned this pull request Nov 19, 2025

Write system paths to lexicographically last ld.so.conf.d drop-in v2 #1473

Closed

elezar force-pushed the fix-ldcache-update-on-non-debian branch from 1e52761 to 8e56cf7 Compare November 20, 2025 10:03

ArangoGutierrez self-requested a review November 20, 2025 10:26

elezar requested a review from cdesiniotis November 20, 2025 11:12

ArangoGutierrez approved these changes Nov 20, 2025

View reviewed changes

jgehrcke reviewed Nov 20, 2025

View reviewed changes

elezar force-pushed the fix-ldcache-update-on-non-debian branch from 8e56cf7 to 6592021 Compare November 20, 2025 15:49

jfroy mentioned this pull request Nov 20, 2025

Pass driver root to nvinfo.New in device plugin main NVIDIA/k8s-device-plugin#1505

Merged

ArangoGutierrez approved these changes Nov 20, 2025

View reviewed changes

elezar merged commit dcb05d0 into NVIDIA:main Nov 20, 2025
30 of 33 checks passed

github-actions bot mentioned this pull request Nov 20, 2025

[release-1.18] Fix ldcache update when host and container distributions do not match #1488

Merged

elezar deleted the fix-ldcache-update-on-non-debian branch December 2, 2025 11:51

		// TODO: Add other architectures that have custom `add_system_dir` macros (e.g. riscv)
		// TODO: Replace with executing the container's dynamlic linker with `--list-diagnostics`?

Fix ldcache update when host and container distributions do not match #1444

Fix ldcache update when host and container distributions do not match #1444

Uh oh!

Conversation

elezar commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing

Uh oh!

elezar commented Nov 11, 2025

Uh oh!

cdesiniotis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jfroy commented Nov 17, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ArangoGutierrez commented Nov 20, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jgehrcke left a comment

Choose a reason for hiding this comment

Uh oh!

elezar commented Nov 20, 2025

Uh oh!

jfroy commented Nov 20, 2025

Uh oh!

elezar commented Nov 20, 2025

Uh oh!

Uh oh!

github-actions bot commented Nov 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

elezar commented Nov 11, 2025 •

edited

Loading