Rpm-ostree in unrecoverable read-only state after failed upgrade

[atonal440]

Describe the bug

After a failed update, my computer is unable to write to any files anywhere. Even rpm-ostree and bootc, run as root in single user mode, are unable to make any changes. This is true on both deployments, so rolling back or other repair actions are not possible.

What did you expect to happen?

I don't understand how the update failed exactly, but after it failed, I expected to be able to roll back to a working configuration of some kind instead of being left with a completely immutable computer.

Output of bootc status

error: Status: Initializing storage: Acquiring Sysroot: No space left on device

This is obviously unhelpful, but "rpm-ostree status" still works:

        State: idle
        Warning: failed to finalize previous deployment
                 error: Finalizing deployment: Finalizing SELinux policy: Child process exited with code 1
                 check `journalctl -b -1 -u ostree-finalize-staged.service`
        AutomaticUpdates: stage; rpm-ostreed-automatic.timer: inactive
        Deployments:
        ● ostree-image-signed:docker://ghcr.io/ublue-os/bluefin-dx:latest
                           Digest: sha256:b0601cb52760625bbd17be97e410bc20de70235100064bbb30945ef040512743
                          Version: latest-42.20250922.1 (2025-09-22T12:25:09Z)
         
          ostree-image-signed:docker://ghcr.io/ublue-os/bluefin-dx:latest
                           Digest: sha256:81d2207d094b945627f4e14d8fafca8d989dd0d3eb516cc2af03aec640c3bb59
                          Version: latest-42.20250825.1 (2025-08-25T14:29:15Z)

The output of the recommended command above is: 

Nov 17 12:59:08 latitude5591 systemd[1]: Finished ostree-finalize-staged.service - OSTree Finalize Staged Deployment.
Nov 17 15:37:16 latitude5591 systemd[1]: Stopping ostree-finalize-staged.service - OSTree Finalize Staged Deployment...
Nov 17 15:37:16 latitude5591 ostree[43197]: Finalizing staged deployment
Nov 17 15:37:21 latitude5591 ostree[43197]: Copying /etc changes: 31 modified, 0 removed, 115 added
Nov 17 15:37:21 latitude5591 ostree[43197]: Copying /etc changes: 31 modified, 0 removed, 115 added
Nov 17 15:37:23 latitude5591 ostree[43197]: Refreshing SELinux policy
Nov 17 15:37:35 latitude5591 ostree[43221]: libsemanage.semanage_make_sandbox: Could not copy files to sandbox /etc/selinux/targeted/tmp. (No space left on device).
Nov 17 15:37:37 latitude5591 ostree[43221]: semodule:  Could not begin transaction:  No space left on device
Nov 17 15:37:37 latitude5591 ostree[43197]: Refreshed SELinux policy in 14400 ms
Nov 17 15:37:37 latitude5591 ostree[43197]: error: Finalizing deployment: Finalizing SELinux policy: Child process exited with code 1
Nov 17 15:37:37 latitude5591 systemd[1]: ostree-finalize-staged.service: Control process exited, code=exited, status=1/FAILURE
Nov 17 15:37:37 latitude5591 systemd[1]: ostree-finalize-staged.service: Failed with result 'exit-code'.
Nov 17 15:37:37 latitude5591 systemd[1]: Stopped ostree-finalize-staged.service - OSTree Finalize Staged Deployment.
Nov 17 15:37:37 latitude5591 systemd[1]: ostree-finalize-staged.service: Consumed 2.776s CPU time, 141M memory peak.

Output of groups

in single user mode, it's obviously just "root" 

groups vloring
vloring: vloring wheel quemu libvirt docker lxd incus-admin

Extra information or context

output of df -h

        Filesystem      Size  Used Avail Use% Mounted on
        composefs        36M   36M     0 100% /
        /dev/dm-0       237G  207G   31G  88% /etc
        devtmpfs        7.6G     0  7.6G   0% /dev
        tmpfs           7.8G     0  7.8G   0% /dev/shm
        efivarfs        384K   87K  293K  23% /sys/firmware/efi/efivars
        tmpfs           3.1G   18M  3.1G   1% /run
        tmpfs           1.0M     0  1.0M   0% /run/credentials/systemd-cryptsetup@luks\x2dc8b2858c\x2dbcf0\x2d4451\x2d8026\x2d713c12f19dc5.service
        tmpfs           1.0M     0  1.0M   0% /run/credentials/systemd-journald.service
        /dev/nvme0n1p2  974M  415M  492M  46% /boot
        /dev/dm-0       237G  207G   31G  88% /var/home
        /dev/nvme0n1p1  599M   36M  563M   7% /boot/efi
        tmpfs           7.8G     0  7.8G   0% /tmp
        tmpfs           1.0M     0  1.0M   0% /run/credentials/systemd-resolved.service

My disk is not full and even if /boot is too full, I can't do anything about it now, because rpm-ostree cleanup --rollback fails with this:
error: cleanup: Invoking cleanup: GDBus.Error:org.gtk.GDBus.UnmappedGError.Quark._g_2dio_2derror_2dquark.Code12: No space left on device

[castrojo]

Does it free up space if you do a cleanup with a -p to remove the pending update?

[atonal440]

Both cleanup operations fail with that same "No space left on device" error.

[travier]

Disk full errors can be disk space or inode space. You can check that with:

$ df -i /sysroot
# or for btrfs:
$ btrfs filesystem df -h /sysroot

Then I would check if I have more than two deployments in /sysroot/ostree/deploy/fedora/deploy/.

[travier]

You might also be hitting https://gitlab.com/fedora/ostree/sig/-/issues/77

[atonal440]

Yeah it was that: I came pretty close to filling my disk a while ago and apparently after I cleared space btrfs never really reclaimed any of it so I was out of both unallocated blocks and metadata space.

Things were so wedged that the only solution I could find was to grab a thumb drive and btrfs device add it to my filesystem, then white-knuckle my way through a series of balance commands as described. Things seem good now though, thanks to everyone who pointed me in the right direction!