kola: Add soft-reboot support for external tests #4119

jmarrero · 2025-05-30T19:10:13Z

Implements soft-reboot capabilities for Kola,
it enables tests to use systemd's soft-reboot functionality.

The implementation follows the same pattern as regular reboots but for systemctl soft-reboot, tracks systemd boot
timestamps rather than kernel boot IDs for state detection.

openshift-ci · 2025-05-30T19:10:17Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

depends on: coreos/coreos-assembler#4119

dustymabe · 2025-05-30T19:17:01Z

Implements soft-reboot capabilities for Kola,
it enables tests to use systemd's soft-reboot functionality.

Is there a case where a soft-reboot would be a better test than a hard reboot?

Is the goal here to reduce test time?

jmarrero · 2025-05-30T19:21:09Z

Right now soft-reboots are not really supported by ostree but we are implementing soft-reboots for ostree on : ostreedev/ostree#3420
We wanted to use Kola for the tests as we already depend on it. The idea is to reduce reboot time and we only would support if no Kernel is present on the deployment we are going to soft-reboot to. But it's still in progress. Just that having Kola support it would make our testing cycle easier too.

dustymabe · 2025-05-30T19:22:17Z

Ok, Just comment here when you move this out of draft

cgwalters

Looks good to me overall! Main issue is code duplication with the "hard" reboot path but that's probably hard to fix and not worth doing.

cgwalters · 2025-05-30T19:27:35Z

tests/kola-ci-self/tests/kola/autopkgtest-soft-reboot

+    echo "test beginning"
+    # Check that boot_id stays the same across soft-reboot
+    INITIAL_BOOT_ID=$(cat /proc/sys/kernel/random/boot_id)
+    echo "Initial boot ID: $INITIAL_BOOT_ID" | logger


Don't need the |logger, we always log tests to the journal

cgwalters · 2025-05-30T19:28:30Z

tests/kola-ci-self/tests/kola/autopkgtest-soft-reboot

+    # Verify boot_id is the same (soft-reboot should not change it)
+    CURRENT_BOOT_ID=$(cat /proc/sys/kernel/random/boot_id)
+    echo "Current boot ID after soft-reboot: $CURRENT_BOOT_ID" | logger


How is this verified though? I think we need to save the boot id to a persistent file like /var/cache/kola-boot-id or something in the first boot and then compare it here.

cgwalters · 2025-05-30T19:29:01Z

tests/kola-ci-self/tests/kola/autopkgtest-soft-reboot

+  mark2) 
+    echo "test in mark2"
+    FINAL_BOOT_ID=$(cat /proc/sys/kernel/random/boot_id)
+    echo "Final boot ID after forced soft-reboot: $FINAL_BOOT_ID" | logger


jmarrero · 2025-06-03T14:23:18Z

OK test now dies with a timeout which is the same behavior I see when I do systemctl soft-reboot without any of our changes to ostree. I will try to see if I can get farther with our changes plus this test.

edit: looking at the console log the VM appears to come back. It's just cosa that loses the ability to ssh back in, even during a cosa run. I been trying to figure out why.

jmarrero · 2025-06-04T22:38:38Z

OK needed to modify qemu.go to get the test to comeback too.

Now need to figureout this failed service.

× coreos-ignition-firstboot-complete.service - CoreOS Mark Ignition Boot Complete
     Loaded: loaded (/usr/lib/systemd/system/coreos-ignition-firstboot-complete.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Wed 2025-06-04 22:34:52 UTC; 18s ago
   Duration: 11.967s
 Invocation: 97458820b4e8426184180f42cd04bf13
       Docs: https://docs.fedoraproject.org/en-US/fedora-coreos/
    Process: 2729 ExecStart=/usr/libexec/coreos-ignition-firstboot-complete (code=exited, status=1/FAILURE)
   Main PID: 2729 (code=exited, status=1/FAILURE)
   Mem peak: 2.7M
        CPU: 11ms

Jun 04 22:34:52 cosa-devsh systemd[1]: Starting coreos-ignition-firstboot-complete.service - CoreOS Mark Ignition Boot Complete...
Jun 04 22:34:52 cosa-devsh coreos-ignition-firstboot-complete[2755]: rm: cannot remove '/boot/ignition.firstboot': No such file or directory
Jun 04 22:34:52 cosa-devsh systemd[1]: coreos-ignition-firstboot-complete.service: Main process exited, code=exited, status=1/FAILURE
Jun 04 22:34:52 cosa-devsh systemd[1]: coreos-ignition-firstboot-complete.service: Failed with result 'exit-code'.
Jun 04 22:34:52 cosa-devsh systemd[1]: Failed to start coreos-ignition-firstboot-complete.service - CoreOS Mark Ignition Boot Complete.

jmarrero · 2025-06-04T23:21:06Z

https://github.com/coreos/fedora-coreos-config/blob/20feb176f19c3142b7256c1eb5bf1cb7c53b29b9/overlay.d/05core/usr/libexec/coreos-ignition-firstboot-complete#L14

I am not sure of the story of that service but it looks like we could ignore that failed service without any real issues.

cgwalters · 2025-06-05T00:45:49Z

My offhand guess as to what's happening here is that service isn't prepared for soft reboots. https://github.com/coreos/fedora-coreos-config/blob/20feb176f19c3142b7256c1eb5bf1cb7c53b29b9/overlay.d/05core/usr/lib/systemd/system/coreos-ignition-firstboot-complete.service#L4 is still triggered because we didn't change the kernel commandline across the soft reboot.

I think CoreOS probably does need to fix this because soft rebooting from the initial boot is a sane thing to do and I see real world use cases for it. It is perhaps as simple as ConditionPathExists=!/boot/ignition.firstboot in the unit.

On coreos/coreos-assembler#4119 surfaced that this service would fail on soft-reboots, it's non fatal but would make the Kola tests fail. It looks like originally this was set to fail instead of risking not running it: https://github.com/coreos/fedora-coreos-config/blob/20feb176f19c3142b7256c1eb5bf1cb7c53b29b9/overlay.d/05core/usr/libexec/coreos-ignition-firstboot-complete#L14

coreos/coreos-assembler#4119 surfaced that this service would fail on soft-reboots, it's non fatal but would make the Kola tests fail. It looks like originally this was set to fail instead of risking not running it: https://github.com/coreos/fedora-coreos-config/blob/20feb176f19c3142b7256c1eb5bf1cb7c53b29b9/overlay.d/05core/usr/libexec/coreos-ignition-firstboot-complete#L14

cgwalters · 2025-06-05T21:04:58Z

Holy cow that was painful to figure out but fixes in #4133 get us improved error message handling to the point where I now see

2025-06-05T20:53:19Z kola: dropping to shell: kolet failed: kolet run-test-unit failed:  mkfifo: cannot create fifo '/run/kolet-reboot': File exists

Which was the real bug here - with soft reboots /run persists (this will be a big trap!) and so we need to change the mkfifo to be idempotent, which I'll do after rebasing this PR on the prep one.

coreos/coreos-assembler#4119 surfaced that this service would fail on soft-reboots, it's non fatal but would make the Kola tests fail. It looks like originally this was set to fail instead of risking not running it: https://github.com/coreos/fedora-coreos-config/blob/20feb176f19c3142b7256c1eb5bf1cb7c53b29b9/overlay.d/05core/usr/libexec/coreos-ignition-firstboot-complete#L14

coreos/coreos-assembler#4119 surfaced that this service would fail on soft-reboots, it's non fatal but would make the Kola tests fail. It looks like originally this was set to fail instead of risking not running it: https://github.com/coreos/fedora-coreos-config/blob/20feb176f19c3142b7256c1eb5bf1cb7c53b29b9/overlay.d/05core/usr/libexec/coreos-ignition-firstboot-complete#L14 Co-authored-by: Colin Walters <[email protected]>

…t-reboots coreos/coreos-assembler#4119 surfaced that this service would fail on soft-reboots, it's non fatal but would make the Kola tests fail. It looks like originally this was set to fail instead of risking not running it: https://github.com/coreos/fedora-coreos-config/blob/20feb176f19c3142b7256c1eb5bf1cb7c53b29b9/overlay.d/05core/usr/libexec/coreos-ignition-firstboot-complete#L14 Co-authored-by: Colin Walters <[email protected]>

coreos/coreos-assembler#4119 surfaced that this services using ConditionKernelCommandLine=ignition.firstboot would fail on soft-reboots, it's non fatal but would make the Kola tests fail. Co-authored-by: Colin Walters <[email protected]>

Implements soft-reboot capabilities for Kola, it enables tests to use systemd's soft-reboot functionality. The implementation follows the same pattern as regular reboots but for `systemctl soft-reboot`, tracks systemd boot timestamps rather than kernel boot IDs for state detection. Co-Authored-By: Colin Walters <[email protected]> Co-Authored-By: Claude <[email protected]> Signed-off-by: Colin Walters <[email protected]> Signed-off-by: Joseph Marrero Corchado <[email protected]>

jmarrero · 2025-06-08T01:07:49Z

/test rhcos

openshift-ci · 2025-06-08T05:08:03Z

@jmarrero: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/rhcos	`6796892`	link	true	`/test rhcos`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

coreos/coreos-assembler#4119 surfaced that this services using ConditionKernelCommandLine=ignition.firstboot would fail on soft-reboots, it's non fatal but would make the Kola tests fail. Co-authored-by: Jonathan Lebon <[email protected]> Co-authored-by: Colin Walters <[email protected]>

openshift-ci bot added the do-not-merge/work-in-progress label May 30, 2025

jmarrero added a commit to jmarrero/ostree that referenced this pull request May 30, 2025

add kola test

1c51dcf

depends on: coreos/coreos-assembler#4119

cgwalters requested changes May 30, 2025

View reviewed changes

jmarrero force-pushed the soft-reboot branch from 23c3678 to 025b07e Compare June 2, 2025 19:38

jmarrero force-pushed the soft-reboot branch 2 times, most recently from 4944160 to d0743f6 Compare June 4, 2025 23:20

jmarrero mentioned this pull request Jun 5, 2025

overlay.d: Update ignition.firstboot services for soft-reboots coreos/fedora-coreos-config#3536

Merged

cgwalters mentioned this pull request Jun 5, 2025

Soft reboot prep #4133

Merged

cgwalters force-pushed the soft-reboot branch from d0743f6 to 01dc0a6 Compare June 5, 2025 21:20

cgwalters force-pushed the soft-reboot branch from 01dc0a6 to 1800086 Compare June 6, 2025 15:20

jmarrero force-pushed the soft-reboot branch from 1800086 to 5c37650 Compare June 6, 2025 16:26

jmarrero marked this pull request as ready for review June 6, 2025 17:36

openshift-ci bot removed the do-not-merge/work-in-progress label Jun 6, 2025

jmarrero force-pushed the soft-reboot branch from 5c37650 to 6796892 Compare June 6, 2025 18:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

kola: Add soft-reboot support for external tests #4119

kola: Add soft-reboot support for external tests #4119

Uh oh!

jmarrero commented May 30, 2025

Uh oh!

openshift-ci bot commented May 30, 2025

Uh oh!

dustymabe commented May 30, 2025

Uh oh!

jmarrero commented May 30, 2025

Uh oh!

dustymabe commented May 30, 2025

Uh oh!

cgwalters left a comment

Uh oh!

cgwalters May 30, 2025

Uh oh!

cgwalters May 30, 2025

Uh oh!

cgwalters May 30, 2025

Uh oh!

jmarrero commented Jun 3, 2025 •

edited

Loading

Uh oh!

jmarrero commented Jun 4, 2025

Uh oh!

jmarrero commented Jun 4, 2025

Uh oh!

cgwalters commented Jun 5, 2025

Uh oh!

cgwalters commented Jun 5, 2025

Uh oh!

jmarrero commented Jun 8, 2025

Uh oh!

openshift-ci bot commented Jun 8, 2025

Uh oh!

Uh oh!

kola: Add soft-reboot support for external tests #4119

Are you sure you want to change the base?

kola: Add soft-reboot support for external tests #4119

Uh oh!

Conversation

jmarrero commented May 30, 2025

Uh oh!

openshift-ci bot commented May 30, 2025

Uh oh!

dustymabe commented May 30, 2025

Uh oh!

jmarrero commented May 30, 2025

Uh oh!

dustymabe commented May 30, 2025

Uh oh!

cgwalters left a comment

Choose a reason for hiding this comment

Uh oh!

cgwalters May 30, 2025

Choose a reason for hiding this comment

Uh oh!

cgwalters May 30, 2025

Choose a reason for hiding this comment

Uh oh!

cgwalters May 30, 2025

Choose a reason for hiding this comment

Uh oh!

jmarrero commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jmarrero commented Jun 4, 2025

Uh oh!

jmarrero commented Jun 4, 2025

Uh oh!

cgwalters commented Jun 5, 2025

Uh oh!

cgwalters commented Jun 5, 2025

Uh oh!

jmarrero commented Jun 8, 2025

Uh oh!

openshift-ci bot commented Jun 8, 2025

Uh oh!

Uh oh!

jmarrero commented Jun 3, 2025 •

edited

Loading