Dedicated GPUs per bundle

[andlaz]

Hi,

apologies if this isn't the right forum for a feature request/question

I've been poking around, trying to find ways to automatically assign a number of Nvidia accelerators ( unique devices connected to the host ) to a specific, Nvidia Container Runtime enhanced bundle. So far i'm coming up with very little and i'm not too certain if this is in or out of scope for NVCR itself.

Before i jump in to a novel oci shim that could work together with NVCR envvars, such as VISIBLE_DEVICES, and other features of the container toolkit ( discovery and filtering on other envvars ) to maintain a bundle to device/s "lease", i'd like to ask

• if this already is easily attainable and i just haven't searched thoroughly enough,
• if this should live in NVCR or elsewhere in the toolkit, or
• it should be indeed essentially pre-processing of the bundle json before so by the time the oci create reaches NVCR, it already has a specific VISIBLE_DEVICES value set, even though the overlap with NVCR ( parsing envvar expressed requirements ) may be large

my use case is coordinating containerized workloads on nodes with many accelerators, that are a poor fit for kubernetes scheduling at this moment ( partially because some of them are containerized kubelets themselves, but i'll also accept an "all of this should really be done over kubernetes" solution as well )

thanks!
Andras

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.

[ArangoGutierrez]

You are correct in your third assumption: this scheduling logic must live outside and before the OCI runtime (NVCR) is invoked.

The nvidia-container-toolkit (and its runtime components) is designed to be a simple, stateless tool. Its job is to execute a request (e.g., "give this container GPU 2"), not to decide on the request (e.g., "which GPU is free?").

• This "leasing" or scheduling logic is explicitly out of scope for the toolkit itself.

• Your idea of a "novel oci shim" or another pre-processing agent is exactly the right architectural layer. That agent would be responsible for checking GPU availability (perhaps via nvidia-smi), "leasing" a device (by tracking its state), and then setting the NVIDIA_VISIBLE_DEVICES environment variable in the OCI bundle before the runtime is called.

And while you noted it's a "poor fit" for your current use case, it is worth confirming that this is exactly the problem that the nvidia-device-plugin for Kubernetes is designed to solve. It handles the discovery, advertising, and scheduling of individual GPUs as schedulable resources.

As this is an architectural question about scope, we are closing this as answered.