Skip to content

research: Could 1.16.13/14 BLS installs lack the composefs ? marker without fs-verity #1391

Open

Description

@cgwalters-bot

Summary

P0, for the operator: could bootc 1.16.13 or 1.16.14 have made a composefs BLS (non-UKI) install with a strict composefs= karg, without the ? marker, on a filesystem without fs-verity such as XFS? If so, those systems would fail to boot after upstream bootc PR 2629.

Two runs:

  • Code analysis (astra): cgwalters-forge/tracker run 42231673679. Answers here with a yes/no and file:line evidence.
  • VM reproduction (devspace, claude, 64 cores): bootc-dev/cgwalters-devspace-sandbox run 42231608357. Installs 1.16.13/14 on XFS, records the BLS options line, then boots PR 2629's head. Hands back in the devspace repo; link its result here.

Done when this issue has a yes/no with evidence, the affected configurations and the recovery path.

Generated-by: https://github.com/cgwalters/#llms

Activity

  1. cgwalters-bot commented on Nov 6, 2026

    @cgwalters-bot
    CollaboratorAuthor

    ✅ dispatch ended well. Result: #1391 (comment)

  2. cgwalters-bot commented on Nov 6, 2026

    @cgwalters-bot
    CollaboratorAuthor

    Yes, a strict BLS install without working fs-verity was possible in both releases—but not the ordinary XFS case. The source-supported case is a filesystem classified as capable (ext4/btrfs), with fs-verity actually unavailable, and no --allow-missing-verity. This is a source-level conclusion, not a VM reproduction.

    • XFS gets ?, in both tags. supports_fsverity() is a filesystem-type test: only ext4/btrfs return true (v1.16.13 install/config.rs:48–51). Non-UKI composefs installs automatically set allow_missing_verity = !supports_fsverity() (v1.16.13 install.rs:1751–1754, v1.16.14:1779–1782). BLS generation uses that flag (both tags, boot.rs:675–689); formatting inserts ? iff true (status.rs:111–118). Thus no to “these versions normally omitted ? on XFS.”
    • The wider hole is real: before import, the disk-space preflight initializes the destination repository unconditionally insecure (v1.16.13 install.rs:2024–2038, v1.16.14:2052–2066). Subsequent initialization requests strict mode when the flag is false (repo.rs:104–114), but the pinned composefs-rs reopens matching existing metadata and derives security from its verity state (repository.rs:1468–1512). Insecure writes tolerate FilesystemNotSupported (same file:2343–2356). Therefore an ext4/btrfs target lacking effective verity support can retain insecure storage while the BLS argument remains strict. Insecure metadata alone does not prove missing image verity: writes still try enabling it.
    • Main vs PR: main at c3be5244 instead initializes a strict provisional repository on a classified-capable filesystem (install.rs:2289–2305). Its initramfs still only relaxes repository policy for ?; absence does not force strictness (initramfs/src/lib.rs:363–374). PR #2629 head 5ce8120a explicitly requires verity when ? is absent and mounts by digest (same file:364–383). Inferred consequence: a strict entry pointing to an unverified image will fail with that initramfs; an ordinary XFS entry retaining ? is not that case.

    Verified by reading both tags, pinned dependency, main and PR head and comparing release diffs. No VM/log verification or runtime tests performed. Requested sol delegation was unavailable (sol is not installed/exposed); this source audit was performed directly.

    Generated-by: https://github.com/cgwalters/#llms

    Generated by dispatch · ◷

  3. cgwalters commented on Nov 6, 2026

    @cgwalters
    Contributor

    @cgwalters-bot go ahead and post in bootc-dev/bootc#2629 but before you do reread all existing comments resynthesize with existing data

  4. cgwalters-bot commented on Nov 6, 2026

    @cgwalters-bot
    CollaboratorAuthor

    Draft: yes for ext4 BLS without effective fs-verity; no for ordinary XFS BLS installs. Nothing posted upstream. I reread all currently returned comments, reviews and inline replies on #2629 (including the c53c2fcc dependency update), plus composefs-rs #439. However, the earlier bootc PR quoted in this comment is not identified by a URL; I could not locate it, so cannot honestly claim to have reread that entire discussion. Please supply its URL before treating this as the completed all-comments synthesis. sol is not available in this runner, so execution could not be delegated; this is a directly prepared draft, not a sol result.

    Exact proposed upstream text:

    **Yes: 1.16.13/1.16.14 could create strict composefs BLS entries on ext4 without effective fs-verity, notably installs onto an existing ext4 filesystem lacking the verity feature. No: ordinary XFS BLS installs already get `?`.** Sealing is not the dividing line; this concerns native composefs entries without `?` whose image/data lack verity, not OSTree or correctly verified strict installations.
    
    I agree with [jmarrero's correction](https://github.com/bootc-dev/bootc/pull/2629#issuecomment-6686909823), and disagree with the XFS example in the [earlier quoted review](https://github.com/bootc-dev/bootc/pull/2629#issuecomment-6686771753). Historically, BLS strictness was chosen by filesystem *type*, not by probing working verity: [v1.16.13 install.rs:1751–1754](https://github.com/bootc-dev/bootc/blob/v1.16.13/crates/lib/src/install.rs#L1751-L1754), [v1.16.14 install.rs:1779–1782](https://github.com/bootc-dev/bootc/blob/v1.16.14/crates/lib/src/install.rs#L1779-L1782), and [config.rs:48–51](https://github.com/bootc-dev/bootc/blob/v1.16.14/crates/lib/src/install/config.rs#L48-L51). Thus ext4/btrfs default to strict unless explicitly relaxed, while XFS is automatically relaxed. [boot.rs:675–689](https://github.com/bootc-dev/bootc/blob/v1.16.14/crates/lib/src/bootc_composefs/boot.rs#L675-L689) passes that flag to the formatter, which inserts `?` [at status.rs:111–118](https://github.com/bootc-dev/bootc/blob/v1.16.14/crates/lib/src/bootc_composefs/status.rs#L111-L118). This confirms cgwalters' concern about externally prepared ext4; it does not establish what Anaconda/image-builder actually formatted. UKIs instead take policy from their embedded cmdline and reject strict XFS [at install.rs:1754–1770](https://github.com/bootc-dev/bootc/blob/v1.16.14/crates/lib/src/install.rs#L1754-L1770).
    
    The mismatch comes from the unconditional insecure preflight ([1.16.13 install.rs:2024–2038](https://github.com/bootc-dev/bootc/blob/v1.16.13/crates/lib/src/install.rs#L2024-L2038), [1.16.14:2052–2066](https://github.com/bootc-dev/bootc/blob/v1.16.14/crates/lib/src/install.rs#L2052-L2066)); subsequent initialization reopens existing metadata and detects its security mode ([pinned composefs-rs repository.rs:1468–1512](https://github.com/composefs/composefs-rs/blob/f720d6c79ac6b082fb1e182eed3b291f88e2a7f4/crates/composefs/src/repository.rs#L1468-L1512)). Insecure writes still attempt verity, but tolerate unsupported filesystems [at :2343–2356](https://github.com/composefs/composefs-rs/blob/f720d6c79ac6b082fb1e182eed3b291f88e2a7f4/crates/composefs/src/repository.rs#L2343-L2356): unprotected metadata alone does not imply an unverified image. Current main retains the BLS type-based default [at install.rs:2003–2008](https://github.com/bootc-dev/bootc/blob/c3be524427b80b20ad1716b2370bbc501876a121/crates/lib/src/install.rs#L2003-L2008), but initializes a strict provisional repo on classified-capable filesystems [at :2289–2305](https://github.com/bootc-dev/bootc/blob/c3be524427b80b20ad1716b2370bbc501876a121/crates/lib/src/install.rs#L2289-L2305). I would qualify the blanket btrfs exemption: the source condition is working verity, not merely the filesystem name.
    
    At current PR head, absence of `?` explicitly requires verity independently of metadata ([c53c2fcc initramfs/src/lib.rs:364–383](https://github.com/bootc-dev/bootc/blob/c53c2fccf0a3af59b58b1a99e8c8033d4ad287af/crates/initramfs/src/lib.rs#L364-L383)). An affected strict image therefore fails in the initramfs; temporarily recover by adding `?` immediately after `composefs=` in the BLS entry, explicitly accepting loss of integrity enforcement. Alternatively select a retained previous deployment manually; that recovery remains untested here. For a UKI, rebuild its embedded cmdline with `--allow-missing-verity` (and re-sign as required), or reinstall onto working fs-verity storage for protected operation. I agree with the reported failure mechanism, but have independently verified source only: [jmarrero-bot reports an ext4 VM reproduction and the broader 1.16.3–1.16.14 range](https://github.com/bootc-dev/bootc/pull/2629#issuecomment-6687049249); I have not inspected its logs or independently verified that entire range. Our [separate VM run](https://github.com/bootc-dev/cgwalters-devspace-sandbox/actions/runs/42231608357) failed in setup with execution jobs skipped, so supplies no boot evidence. Automatically relaxing unsigned BLS is a separate policy proposal, not what this PR currently implements.
    
    Generated-by: https://github.com/cgwalters/#llms

    Generated by dispatch · ◷

  5. cgwalters-bot commented on Nov 6, 2026

    @cgwalters-bot
    CollaboratorAuthor

    btrfs follow-up (1.16.x installs onto btrfs) is running as bootc-dev/cgwalters-devspace-sandbox#36, run 42280467569.

  6. cgwalters-bot commented on Nov 7, 2026

    @cgwalters-bot
    CollaboratorAuthor

    Triage of cgwalters's comment: claimed; started.

    Generated-by: https://github.com/cgwalters/#llms

  7. cgwalters-bot commented on Nov 7, 2026

    @cgwalters-bot
    CollaboratorAuthor

    Tracking the operator's request in #1391.

    Generated-by: https://github.com/cgwalters/#llms

  8. cgwalters-bot commented on Nov 7, 2026

    @cgwalters-bot
    CollaboratorAuthor

    The upstream posting request is already fulfilled; the compatibility bug is valid and VM-reproduced on ext4, not ordinary XFS. I reread this thread and all currently returned PR comments, reviews and inline replies. Our source synthesis and VM confirmation are already upstream, and PR #2629 is merged, so another copy there would be redundant.

    The reconciled affected set is composefs installs made by 1.16.0–1.16.14, using to-filesystem or to-existing-root onto ext4 without verity, with a strict BLS argument or strict UKI. The 1.16.13/14 VM report records strict ext4 entries failing after c53c2fcc and XFS entries retaining ? and booting. The later upstream synthesis extends the historical range and reports tested BLS recovery. Later btrfs tests also pass: unprotected metadata alone is not sufficient to trigger failure. These are existing test reports, not new VM runs here.

    The code matches: 1.16.14 chooses BLS policy by filesystem type, while its preflight creates an insecure repository; #2629 requires verity for strict boots independently of metadata.

    Remaining work is compatibility guidance, a pre-upgrade check, clearer errors and pinned-version upgrade coverage (#2632), not reverting the integrity fix. Recovery is selecting the previous deployment manually or adding ? immediately after composefs= in BLS, explicitly accepting no integrity enforcement. Protected operation requires working verity storage and verified objects. The upstream report still leaves strict-to-relaxed UKI recovery and Secure Boot untested; installed-system prevalence is unknown.

    Generated-by: https://github.com/cgwalters/#llms

    Generated by dispatch · ◷

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions