Freeze container during live checkpoint for consistency#28963
Open
rst0git wants to merge 1 commit into
Open
Conversation
|
[NON-BLOCKING] Packit jobs failed. @containers/packit-build please check. Everyone else, feel free to ignore. |
Contributor
|
Arguably a breaking change? But, given we did the same to |
Contributor
|
After discussion, let's get this in 6.0 |
Luap99
reviewed
Jun 18, 2026
Luap99
left a comment
Luap99
left a comment
Member
There was a problem hiding this comment.
seems reasonable overall.
some nits for the test
Also please squash the commits into one, we like feature/bug and test in one commit.
baude
reviewed
Jun 18, 2026
baude
reviewed
Jun 18, 2026
baude
reviewed
Jun 18, 2026
baude
reviewed
Jun 18, 2026
Contributor
Author
@mheon It shouldn't be a breaking change. The only difference is that we keep a container paused until the checkpoint is fully written when cc: @adrianreber |
rst0git
force-pushed
the
checkpoint-leave-running-fs-consistency
branch
2 times, most recently
from
065ebd0 to
068d2c2
Compare
When checkpointing a container with --leave-running, libpod dumps the container's memory via the OCI runtime (CRIU) first and only captures the rootfs diff and named volumes afterwards. CRIU thaws the container as soon as the memory dump finishes, so the processes inside the container continue to run between the memory snapshot and the file-system capture. As a result, the checkpoint can be inconsistent: have CRIU images and a file system that reflect different points in time. To fix this, we freeze the container's cgroup before invoking the OCI runtime and thaw it again only after the checkpoint image/archive has been written. The OCI runtime calls CRIU with the freezer cgroup and restores it to its previous state once the dump completes, so a container that was already frozen stays frozen across the dump and the file system is captured at the same instant as the CRIU images. This mirrors the approach other engines (e.g. CRI-O and containerd). The default (stopping) checkpoint functionality is not affected by this issue because CRIU leaves the tasks dead after the dump. This patch also adds a regression test for the consistency of live (--leave-running) checkpoints. The container runs a workload that keeps an in-memory counter in sync with a value written to a file on its root file system, maintaining the invariant that the on-disk value never gets ahead of the in-memory counter. Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
rst0git
force-pushed
the
checkpoint-leave-running-fs-consistency
branch
from
068d2c2 to
44d1e68
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When checkpointing a container with
--leave-running, the processes inside the container continue running after CRIU captures the container's runtime state. Because the rootfs diff and named volumes are saved afterward (inexportCheckpoint()/createCheckpointImage()), this can result in an inconsistent checkpoint state, where the CRIU images reflect an earlier point in time than the captured filesystem state. To fix this, we freeze the container cgroup during the checkpoint operation similar to the approach used with other engines (e.g. CRI-O, containerd).