Failure to unmount loop device causes build failures

[mtalexan]

Problem

Running either a --type=anaconda-iso --rootfs xfs or --type=qcow2 --rootfs xfs build fails 100% of the time in the same way.

While the output from the actual execution is extremely limited, it appears that the org.osbuild.bootc.install-to-filesystem inexplicably fails to unmount one of the loop devices it mounted (no way to determine which one or why), which instead of being caught as a failure continues on to later stages anyway. The next stage is therefore unable to lock the same loop device, and tries to associate a different loop device with the disk. But since the old loop device partitions are still mounted, it fails when there's a disk UUID conflict with the still unmounted partitions.

Command that was run:

sudo podman pull quay.io/centos-bootc/bootc-image-builder:latest quay.io/fedora/fedora-bootc:42

sudo podman run --rm -it --privileged --security-opt=label=type:unconfined_t -v $(pwd)/output:/output -v /var/lib/containers/storage:/var/lib/containers/storage quay.io/centos-bootc/bootc-image-builder:latest --type qcow2 --rootfs xfs quay.io/fedora/fedora-bootc:42

Console output

[-] Disk image building step
[3 / 5] Pipeline image [------------------------------------------------------------------------------------------------------------>________________________________________________________________________] 60.00%
[7 / 12] Stage org.osbuild.systemd.unit.create [------------------------------------------------------------------------------------------->_________________________________________________________________] 58.33%
Message: Starting module org.osbuild.systemd.unit.create
2025/09/08 21:19:58 error: cannot run osbuild: error running osbuild: exit status 1
BuildLog:
starting -
Starting pipeline source org.osbuild.containers-storage
Finished module source org.osbuild.containers-storage
Finished pipeline org.osbuild.containers-storage
Starting pipeline build
Starting module org.osbuild.container-deploy
Copying blob sha256:8f42b6769f71d516f01f5de6931fda2bef8b45f595b7799130a9d5da5c87dfc3
Copying blob sha256:e4e9e3254539d4d8c2c9bab6355c98e86bcc9c46638d2743e8ac2a1745e12057
Copying blob sha256:71d3fe0f4c16feb178a16d36fc1fe3c27b5e9e1cb5d4e93980ada11f0d2774a1
Copying blob sha256:b1f33a901ddf08c648e2a9a77af6c65488059f092b70cb6ff083617771ce866c
Copying blob sha256:fee468cd2a0e57a1d51a37e6de8267b58b5e35c7b3908d894c1b66789254ce30
Copying blob sha256:d34f56390ac5bbe0336abb77131e64ac8b570e7c7bc6913b1be1896da2cb3856
Copying blob sha256:81bce3ee3026d7ec2eac4f224416b85657c0c1e8d3dd6cd6cba7a9dd76d44884
Copying blob sha256:d950ec128ff1068e10221cf0366b253c08ba4d1bb8fd195f6ed819c1b78c16ec
Copying blob sha256:8feeb16432f7450206a05eec9f053e33bfc2802ca3b5d2cfc35c231bd458b1e6
Copying blob sha256:66fc5734949691c62b487c9aefd5afea4a81c6035b30ac1f7db4907a5242dfcf
Copying blob sha256:54d6d2bb899395ab7cc8705d721b104067569321b3ee20194df76110b99caf02
Copying blob sha256:bcb0a306a0b357640cdaae1a1d49799307286de41d02697e7b10694eba80e32c
Copying blob sha256:7484e79fab3fa05af331ca222962fabc1fe65878cf68aa92e61090dc88309e47
Copying blob sha256:5b8972b24f719a3c07f5b28f9c38b6a7d65c05a2678f0b1be6432d80df2948d0
Copying blob sha256:1c347b4b1b9b3ab6818256f39d63cf3335a98ce22bc3be8f0eb661c2c009d5e8
Copying blob sha256:3c89171804a693d44a65557df52eac05f2c1a8ac7c22bb6a7ff7dd1c46758a31
Copying blob sha256:dd401e8d4d5ade539a650acbdfad81cd6a4275699657d786500cc0b06da13132
Copying blob sha256:db0d04f2a9db0245e4a5cb4be5add769c4e072c361aed0a69fda4372b99bb150
Copying blob sha256:cd2476929c1cdc7273a560c05b8a1da95d638454cd1b3449ffb4b07493e9ee38
Copying blob sha256:07223c993e66c9ff6ca9ed1aa95a16ff26ca553ca751acc1dd6075e6f303fac9
Copying blob sha256:94eb8ec6b8ceec858250695cd6ee5046e6d5f14a6c3702356248be0db701cc58
Copying blob sha256:b2fbbae958fcc9582bce179f255ae645f61ca575a8f64200fe8732a56d22ce5b
Copying blob sha256:567590cee07169823876bde380a8a1d8e9964f41336b97ea56b32decb584ac6c
Copying blob sha256:dd8c098bdd211c9ece56c330cdec38063669719e2210b218f7929f679c62c33e
Copying blob sha256:b141be4e6eadd12f8a28b0f4c9aa68e4a11d54f1d869df16afbabd5922791a50
Copying blob sha256:a679f4928fd8b10c44d76783edc9ecc10477215264a082d7cb60536bb104c2a7
Copying blob sha256:ecf1520d7b76c09bca0cdf71e8d0d9934f477f05e0f35d49b7743fde90e2daa9
Copying blob sha256:5b838c4944f61899159c02b2d6c8a5ebcb533389dd75ad3c03f7c69fa5c73a1e
Copying blob sha256:6a1c16201d0c13329071a1bd479ee606067046da6c31ae278f849e02e9a7d114
Copying blob sha256:3b26c2bde9c96d52fbe3768881eb750f61c86ddf50e0b5e00ca0b4e1f82ac3d0
Copying blob sha256:d7b4af81d04ec8f6a73b7fac1857f933036194ed6914beae214905e0b89ca6ad
Copying blob sha256:8123e96b7539cf844db06cd5b8f4135a6fb841d7a0ce717372b5604aedca9cea
Copying blob sha256:00645b645b0968cf79f744983756426409af22fd7899510f0365a10b55fe4937
Copying blob sha256:089eaff08ce04b25419f2590fa4b1fb1d282b825c7e04b6a2ff3349c469cf2e3
Copying blob sha256:cfdc89f7ea439adadf5343a0cc469f99639cca7d4c2f868ac411bbbfedcaf060
Copying blob sha256:9ccaf84e0c9f34d22049743a36afcc62a426b2d665c4b60cc77c369dc8fea4bc
Copying blob sha256:e0522cd26916e554410c5b0c2bf00985691b5d18e3e586b39bfee4cde3559188
Copying blob sha256:338c113924074a7384612bc33e68c39956c5dceebe8617b771f29e84b0d1c7df
Copying blob sha256:98bbaa7281f5fce5f8dfc095146bd748007de6c628995dce37e3383701f6f337
Copying blob sha256:ca93d63800edcaa4c59e88dcb862e71c69921f17fedaedb4adc027b2dab07fa1
Copying blob sha256:38b9620a62a3b615ef04884d89f7d7f29974fd35def1b581fede6ad6b715ac19
Copying blob sha256:ee2dbd00bf3517e867fe45c2a1bafefe48c05fd5f732cf47f403829d9e0a558c
Copying blob sha256:ad9fa312064073d066d1b611b4243ca795ff862ed54dfb035e4199cf6010c565
Copying blob sha256:aaf5c2922583c6b7fadcfdef0705c22bff9e58da9c26c5d9c100dcd74db4e80b
Copying blob sha256:3dc72e07cb44b47f167dd71672ba2af3d4fb564f0e8c5a1debbadef6a01f1323
Copying blob sha256:414465c0346a3cb7ffa6246cbaf044644f34905608316aad8aec59396d1bef05
Copying blob sha256:1521d358a8278563e920c7900b6e7c9c56bee7f260c904a8b7eccf3cdac526e6
Copying blob sha256:3b0aba1aebbd20795960a9af00f3b8542a58dadca2a1781c887f9ae0ae843c7d
Copying blob sha256:d70be2b9f89fdb8b1feef1d7206b902035643a42e06f26cca3397595babf4085
Copying blob sha256:a4fd312aa77c204852ff56011aff16e8eeb2c846fcb057bf95d89ed4250e7164
Copying blob sha256:367a54691d8d7953cc809cb54e98e67fec818b911d9093189ef4d14a35f7f61c
Copying blob sha256:2baa9273f86e6d9f3ba81f7980ef86bfc0c06a84f02cc86db4086ab54914a13c
Copying blob sha256:98921ce2dbf1ffac8e55f3833ff6c313aa8cdbe8e93bd9d2f17b640569641417
Copying blob sha256:3808a7f04e7348d1e8d65f3f7290788be5fdbc9a3a56ef0e638896c79b868537
Copying blob sha256:402fcc23766d9e2190f40480003c8e250043875e1d774e80afdbfd0f2c27c7b9
Copying blob sha256:3f854678beb6914898d07137dccef378edce84217abcbad822bc056455144a10
Copying blob sha256:9b8b50778db2a21fd29d4ab9febb4c2606c2a30babc7e6ebcb47d33794eb7a40
Copying blob sha256:7eb778b9662b1c51f9aaa1825313996ba62328165bca569e62f58b348032f5ae
Copying blob sha256:b60cff754a8b808c29c709b3a155a413d47fa3fd27e74d0e97fcbaaf61d50145
Copying blob sha256:abb4a36511074dd2d2d199f00bf5db8f812e212a97288eb17c3a493cdb1acae5
Copying blob sha256:638594298b6d8066bd198c0891399091f9a68d0ddccccfdbf4da7bf1f0ea7f53
Copying blob sha256:abf01cef94820ec041ddd0c25ad571943eba740ebf51dee677e8a09fdf4621fb
Copying blob sha256:3c79b2b59e6b62c96c80c8b697fe638afdd6f426fc6f3a2f7f5b797a64b2f6a3
Copying blob sha256:aade6c647fc599eeb236c2792ba2f9d33ee5338dd8b5021876dd11b53eeb251b
Copying blob sha256:12787d84fa137cd5649a9005efe98ec9d05ea46245fdc50aecb7dd007f2035b1
Copying config sha256:8bd413d59f3d20770b849d37fc2858c45d45c1bb14cb133944feef5fb968a61b
Writing manifest to image destination
8bd413d59f3d20770b849d37fc2858c45d45c1bb14cb133944feef5fb968a61b
Untagged: docker.io/library/tmp-container-deploy-16296744577203:latest
Deleted: 8bd413d59f3d20770b849d37fc2858c45d45c1bb14cb133944feef5fb968a61b
Finished module org.osbuild.container-deploy
Starting module org.osbuild.selinux
Finished module org.osbuild.selinux
Finished pipeline build
Starting pipeline image
Starting module org.osbuild.truncate
Finished module org.osbuild.truncate
Starting module org.osbuild.sfdisk
device/device (org.osbuild.loopback): loop18 acquired (locked: True)
label: gpt
label-id: D209C89E-EA5E-4FBD-B161-B461CCE297E0
start="2048", size="2048", type="21686148-6449-6E6F-744E-656564454649", uuid="FAC7F1FB-3E8D-4137-A512-961DE09A5549", bootable
start="4096", size="1026048", type="C12A7328-F81F-11D2-BA4B-00A0C93EC93B", uuid="68B2905B-DF3E-4FB3-80FA-49D1E773AA33"
start="1030144", size="2097152", type="0FC63DAF-8483-4772-8E79-3D69D8477DE4", uuid="CB07C243-BC44-4717-853E-28852021225B"
start="3127296", size="17844191", type="0FC63DAF-8483-4772-8E79-3D69D8477DE4", uuid="6264D520-3FB9-423F-8AB8-7A0A8E3D3562"
{
   "partitiontable": {
      "label": "gpt",
      "id": "D209C89E-EA5E-4FBD-B161-B461CCE297E0",
      "device": "/dev/loop18",
      "unit": "sectors",
      "firstlba": 2048,
      "lastlba": 20971486,
      "sectorsize": 512,
      "partitions": [
         {
            "node": "/dev/loop18p1",
            "start": 2048,
            "size": 2048,
            "type": "21686148-6449-6E6F-744E-656564454649",
            "uuid": "FAC7F1FB-3E8D-4137-A512-961DE09A5549"
         },{
            "node": "/dev/loop18p2",
            "start": 4096,
            "size": 1026048,
            "type": "C12A7328-F81F-11D2-BA4B-00A0C93EC93B",
            "uuid": "68B2905B-DF3E-4FB3-80FA-49D1E773AA33"
         },{
            "node": "/dev/loop18p3",
            "start": 1030144,
            "size": 2097152,
            "type": "0FC63DAF-8483-4772-8E79-3D69D8477DE4",
            "uuid": "CB07C243-BC44-4717-853E-28852021225B"
         },{
            "node": "/dev/loop18p4",
            "start": 3127296,
            "size": 17844191,
            "type": "0FC63DAF-8483-4772-8E79-3D69D8477DE4",
            "uuid": "6264D520-3FB9-423F-8AB8-7A0A8E3D3562"
         }
      ]
   }
}
Finished module org.osbuild.sfdisk
Starting module org.osbuild.mkfs.fat
device/device (org.osbuild.loopback): loop18 acquired (locked: True)
mkfs.fat 4.2 (2021-01-31)
Finished module org.osbuild.mkfs.fat
Starting module org.osbuild.mkfs.xfs
device/device (org.osbuild.loopback): loop18 acquired (locked: True)
meta-data=/dev/loop18            isize=512    agcount=4, agsize=65536 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=1
         =                       reflink=1    bigtime=1 inobtcount=1 nrext64=1
         =                       exchange=0  
data     =                       bsize=4096   blocks=262144, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1, parent=0
log      =internal log           bsize=4096   blocks=44964, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
Discarding blocks...
Done.
Finished module org.osbuild.mkfs.xfs
Starting module org.osbuild.mkfs.xfs
device/device (org.osbuild.loopback): loop18 acquired (locked: True)
meta-data=/dev/loop18            isize=512    agcount=4, agsize=557631 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=1
         =                       reflink=1    bigtime=1 inobtcount=1 nrext64=1
         =                       exchange=0  
data     =                       bsize=4096   blocks=2230523, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1, parent=0
log      =internal log           bsize=4096   blocks=51300, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
Discarding blocks...
Done.
Finished module org.osbuild.mkfs.xfs
Starting module org.osbuild.bootc.install-to-filesystem
device/disk (org.osbuild.loopback): loop18 acquired (locked: False)
mount/- (org.osbuild.xfs): mounting /dev/loop18p4 -> /store/tmp/buildroot-tmp-dvclbw6t/mounts/
mount/boot (org.osbuild.xfs): mounting /dev/loop18p3 -> /store/tmp/buildroot-tmp-dvclbw6t/mounts/boot
mount/boot-efi (org.osbuild.fat): mounting /dev/loop18p2 -> /store/tmp/buildroot-tmp-dvclbw6t/mounts/boot/efi
Installing image: docker://quay.io/fedora/fedora-bootc:42
Initializing ostree layout
layers already present: 0; layers needed: 65 (1.0 GB)
Deploying container image...
done (5 seconds)
Installing bootloader via bootupd
Added 01_users.cfg
Added 10_blscfg.cfg
Added 14_menu_show_once.cfg
Added 30_uefi-firmware.cfg
Added 41_custom.cfg
Installed: grub.cfg
Installed: bootuuid.cfg
Installed: "fedora/grub.cfg"
Installed: "fedora/bootuuid.cfg"
Trimming root
.: 8.5 GiB (9136222208 bytes) trimmed
Finalizing filesystem root
Trimming boot
boot: 1 GiB (1073741824 bytes) trimmed
Finalizing filesystem boot
Installation complete!
mount/boot-efi (org.osbuild.fat): umount: /store/tmp/buildroot-tmp-dvclbw6t/mounts/boot/efi unmounted
mount/boot (org.osbuild.xfs): umount: /store/tmp/buildroot-tmp-dvclbw6t/mounts/boot unmounted
mount/- (org.osbuild.xfs): umount: /store/tmp/buildroot-tmp-dvclbw6t/mounts/ unmounted
device/disk (org.osbuild.loopback): Traceback (most recent call last):
device/disk (org.osbuild.loopback):   File "/usr/lib/osbuild/devices/org.osbuild.loopback", line 184, in <module>
device/disk (org.osbuild.loopback):     main()
device/disk (org.osbuild.loopback):     ~~~~^^
device/disk (org.osbuild.loopback):   File "/usr/lib/osbuild/devices/org.osbuild.loopback", line 180, in main
device/disk (org.osbuild.loopback):     service.main()
device/disk (org.osbuild.loopback):     ~~~~~~~~~~~~^^
device/disk (org.osbuild.loopback):   File "/usr/lib/python3.13/site-packages/osbuild/host.py", line 252, in main
device/disk (org.osbuild.loopback):     self.stop()
device/disk (org.osbuild.loopback):     ~~~~~~~~~^^
device/disk (org.osbuild.loopback):   File "/usr/lib/python3.13/site-packages/osbuild/devices.py", line 124, in stop
device/disk (org.osbuild.loopback):     self.close()
device/disk (org.osbuild.loopback):     ~~~~~~~~~~^^
device/disk (org.osbuild.loopback):   File "/usr/lib/osbuild/devices/org.osbuild.loopback", line 165, in close
device/disk (org.osbuild.loopback):     self.lo.clear_fd_wait(self.fd, 30)
device/disk (org.osbuild.loopback):     ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
device/disk (org.osbuild.loopback):   File "/usr/lib/python3.13/site-packages/osbuild/loop.py", line 265, in clear_fd_wait
device/disk (org.osbuild.loopback):     raise TimeoutError("waiting for loop device timed out")
device/disk (org.osbuild.loopback): TimeoutError: waiting for loop device timed out
Finished module org.osbuild.bootc.install-to-filesystem
Starting module org.osbuild.systemd.unit.create
device/disk (org.osbuild.loopback): loop19 acquired (locked: False)
mount/- (org.osbuild.xfs): mounting /dev/loop19p4 -> /store/tmp/buildroot-tmp-n608pr12/mounts/
mount/- (org.osbuild.xfs): already unmounted: /store/tmp/buildroot-tmp-n608pr12/mounts/
Output:
Traceback (most recent call last):
  File "/usr/bin/osbuild", line 33, in <module>
    sys.exit(load_entry_point('osbuild==160', 'console_scripts', 'osbuild')())
             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/site-packages/osbuild/main_cli.py", line 185, in osbuild_cli
    r = manifest.build(
        object_store,
    ...<4 lines>...
        stage_timeout=stage_timeout
    )
  File "/usr/lib/python3.13/site-packages/osbuild/pipeline.py", line 525, in build
    res = pl.run(store, monitor, libdir, debug_break, stage_timeout)
  File "/usr/lib/python3.13/site-packages/osbuild/pipeline.py", line 399, in run
    results = self.build_stages(store,
                                monitor,
                                libdir,
                                debug_break,
                                stage_timeout)
  File "/usr/lib/python3.13/site-packages/osbuild/pipeline.py", line 371, in build_stages
    r = stage.run(tree,
                  self.runner,
    ...<4 lines>...
                  debug_break,
                  stage_timeout)
  File "/usr/lib/python3.13/site-packages/osbuild/pipeline.py", line 243, in run
    data_mnt = mntmgr.mount(mount)
  File "/usr/lib/python3.13/site-packages/osbuild/mounts.py", line 98, in mount
    path = client.call("mount", args)
  File "/usr/lib/python3.13/site-packages/osbuild/host.py", line 348, in call
    ret, _ = self.call_with_fds(method, args)
             ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/osbuild/host.py", line 384, in call_with_fds
    raise error
osbuild.host.RemoteError: RuntimeError: b'mount: /store/tmp/buildroot-tmp-n608pr12/mounts: wrong fs type, bad option, bad superblock on /dev/loop19p4, missing codepage or helper program, or other error.\n       dmesg(1) may have more information after failed mount system call.' (code: 32)
   File "/usr/lib/python3.13/site-packages/osbuild/host.py", line 268, in serve
    reply, reply_fds = self._handle_message(msg, fds)
                       ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/osbuild/host.py", line 301, in _handle_message
    ret, fds = self.dispatch(name, args, fds)
               ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/osbuild/mounts.py", line 131, in dispatch
    r = self.mount(args)
  File "/usr/lib/python3.13/site-packages/osbuild/mounts.py", line 199, in mount
    raise RuntimeError(f"{msg} (code: {code})") from e

journalctl is full of host system actions that the bootc-image-builder is trying to trigger, like bumblebee, cups, zscaler, crowdstrike, etc since it's completely uniisolated from the host system, but the only relevant error in the journal is:

Sep 08 15:19:57 golw-12t4k74 kernel: XFS (loop19p4): Filesystem has duplicate UUID cf88fa83-d614-464c-b75e-d53fd6fa4056 - can't mount

---

Almost certainly this is being caused by something in the host system that's automatically triggering on the fact that there's a rootfs mounted, and is causing the (singular?) attempt to unmount the partition to be blocked due to the partition still being in use. There are lots of reports in the journal log of various intrinsic system services trying to access the partition(s) that are failing to unmount, none of which can be disabled on a functioning host system.

[mtalexan]

It's worth noting that this is the very reason why no one in the decades development teams have been working on it, has successfully created a tool that can do device partitioning, file system creation, and file system content population with an un-isolated loop device. It's been worked on for decades, by 10s to 100s of development teams unsuccessfully: there are just too many host system variable effects that can't be avoided or accounted for.

Having the loop devices exposed un-isolated from the host is the primary cause of this issue reported here. Hosts will have myriad of predefined behaviors that are looking for certain files to be available. Mounting a loop device with a rootfs in it will trigger all of them, there's no way to disable or avoid that. In almost every case, one of these thousands of possible versions of host defined automatic behaviors will cause a race condition with unmounting the loop device when you're done with it, and may also directly interfere the content when it's being written and corrupt it. The failure to unmount can only be handled by going into an infinite loop retrying the unmount until (hopefully) none of the processes causing the block happen to be blocking it during the moment a retry is attempted. This can't be bounded, it could be seconds or days depending on what's blocking it. Since there's thousands of possible tools/services that can cause this, there's also no way to check for or disable them. Often they're mandatory system services so they can't even be halted to avoid the problem without crashing the entire host.

Additionally, you have to contend with auto-mounting behaviors, which come in dozens of varieties, and are often enabled by default. Right after you partition the loop device and trigger partition redetection, you get a race condition between your attempt to mount the partitions you just created so you can write to them, and the auto-mounting that will try to do the same. You can't mount the same partition twice, so whoever looses the race fails to mount the partition.

---

This is a very well known problem in the embedded system development space, and so far the only consistently successful solutions I've seen are to either do all the work in QEMU or similar, or to use something like libguestfs.

I see there's already related discussion here: #98

[supakeen]

It's indeed a pretty common problem; especially in the bootable container world where cross-distribution (or even cross-version) building is common, tooling from the container itself is being used against the host kernel, and thus these issues get triggered even more commonly.

There have been a few proof of concept implementations (including libguestfs, and qemu) but I see you've already found the discussion ticket where some of them have been discussed. I know @alexlarsson and @mvo5 were discussing this again very recently so perhaps they can chime in with the current status and/or ideas.