[nvidia-ctk-installer] do not revert cri-o config on shutdown #1360
Conversation
cdesiniotis
force-pushed
the
do-not-cleanup-crio-drop-in
branch
8 times, most recently
from
7d12529 to
c9c7aa7
Compare
cdesiniotis
changed the title
|
One question: Even if we don't delete the drop-in file, do we no still remove the binaries? |
| if !o.SetAsDefault { | ||
| return nil | ||
| } |
There was a problem hiding this comment.
Don't we still need to check whether we're using a drop-in file? We could be modifying the top-level config directly.
There was a problem hiding this comment.
I am not sure I follow why that matters? Regardless of whether we use a drop-in file or not, removing the nvidia runtime from the cri-o config can lead to the issues described in the PR description.
The reason I thought to add this conditional is that we must attempt to revert the config if nvidia is the default runtime, or else all future pods will fail to function (after the nvidia runtime binaries are removed).
I believe we still do, yes. |
cdesiniotis
force-pushed
the
do-not-cleanup-crio-drop-in
branch
3 times, most recently
from
e6d4b6e to
521c243
Compare
This commit updates the behavior of the nvidia-ctk-installer for cri-o. On shutdown, we no longer delete the drop-in config file as long as none of the nvidia runtime handlers are set as the default runtime. This change was made to workaround an issue observed when uninstalling the gpu-operator -- management containers launched with the nvidia runtime handler would get stuck in the terminating state with the below error message: ``` failed to find runtime handler nvidia from runtime list map[crun:... runc:...], failed to "KillPodSandbox" for ... ``` There appears to be a race condition where the nvidia-ctk-installer removes the drop-in file and restarts cri-o. After the cri-o restart, if there are still pods / containers to terminate that were started with the nvidia runtime, then cri-o fails to terminate them. The behavior of cri-o, and its in-memory runtime handler cache, appears to differ from that of containerd as we have never encountered such an issue with containerd. This commit can be considered a stop-gap solution until more robust solution is developed. Signed-off-by: Christopher Desiniotis <[email protected]>
…eanup Signed-off-by: Christopher Desiniotis <[email protected]>
cdesiniotis
force-pushed
the
do-not-cleanup-crio-drop-in
branch
from
42a4ad6 to
9fab81c
Compare
Just to close the loop on this. In checking the implementation yesterday, we do unconfigure the runtime, but do NOT remove the binaries. This is the reason that containerd still functions and why CRIO functions with this change. This should be a sufficient workaround for the time being, but the implications of being able to trigger the |
3 participants
This commit updates the behavior of the nvidia-ctk-installer for cri-o.
On shutdown, we no longer delete the drop-in config file as long as
none of the nvidia runtime handlers are set as the default runtime.
This change was made to workaround an issue observed when uninstalling
the gpu-operator -- management containers launched with the nvidia
runtime handler would get stuck in the terminating state with the below
error message:
There appears to be a race condition where the nvidia-ctk-installer removes the drop-in file
and restarts cri-o. After the cri-o restart, if there are still pods / containers to terminate
that were started with the nvidia runtime, then cri-o fails to terminate them. The behavior
of cri-o, and its in-memory runtime handler cache, appears to differ from that of containerd as
we have never encountered such an issue with containerd.
This commit can be considered a stop-gap solution until more robust solution is developed.
Signed-off-by: Christopher Desiniotis [email protected]