ostree pull-local corrupts commit objects

[roshan20039]

Hi @cgwalters ,

I've integrated meta-updater with my yocto build to enable ostree. When ostree --repo=ota-sysroot/ostree/repo pull-local ostree_repo <branch> is done, it is corrupting the repo - ostree --repo=ota-sysroot/ostree/repo fsck is failing with metadata mismatch error.
Whereas, ostree --repo=ostree_repo fsck is passing.

Is this a bug in ostree pull-local? Is there any other alternative to check file system health with ostree enabled system/ is there any other way to create deployment in offline way(without pull-local) so that it doesn't corrupt the file system?

[codeshravan]

Same issue I am also facing. Would you please help here @cgwalters

[ricardosalveti]

Trying to reproduce with LmP (scarthgap), and this is working fine:

build-lmp/tmp-lmp/work/intel_corei7_64-lmp-linux/lmp-base-console-image/1.0/recipe-sysroot-native/usr/bin/ostree --repo=build-lmp/deploy/images/intel-corei7-64/ostree_repo fsck
Validating refs...
Validating refs in collections...
Enumerating commits...
Verifying content integrity of 3 commit objects...
fsck objects (13719/13719) [=============] 100%
object fsck of 3 commits completed successfully - no errors found.

build-lmp/tmp-lmp/work/intel_corei7_64-lmp-linux/lmp-base-console-image/1.0/recipe-sysroot-native/usr/bin/ostree --repo=build-lmp/tmp-lmp/work/intel_corei7_64-lmp-linux/lmp-base-console-image/1.0/ota-sysroot/ostree/repo pull-local --remote=lmp build/lmp-secondary/build-lmp/deploy/images/intel-corei7-64/ostree_repo 6e407c71133a442a753e0f12727c88a3157a8b56a3be030c289aeea9ef66ec4f
1229 metadata, 12464 content objects imported; 2.7 GB content written

build-lmp/tmp-lmp/work/intel_corei7_64-lmp-linux/lmp-base-console-image/1.0/recipe-sysroot-native/usr/bin/ostree --repo=build-lmp/deploy/images/intel-corei7-64/ostree_repo fsck
Validating refs...
Validating refs in collections...
Enumerating commits...
Verifying content integrity of 3 commit objects...
fsck objects (13719/13719) [=============] 100%
object fsck of 3 commits completed successfully - no errors found.

build-lmp/tmp-lmp/work/intel_corei7_64-lmp-linux/lmp-base-console-image/1.0/recipe-sysroot-native/usr/bin/ostree --repo=build-lmp/tmp-lmp/work/intel_corei7_64-lmp-linux/lmp-base-console-image/1.0/ota-sysroot/ostree/repo fsck
Validating refs...
Validating refs in collections...
Enumerating commits...
Verifying content integrity of 1 commit objects...
fsck objects (13695/13695) [=============] 100%
object fsck of 1 commits completed successfully - no errors found.

libostree:
 Version: '2024.5'
 Features:
  - inode64
  - initial-var
  - libcurl
  - composefs
  - ex-fsverity
  - libarchive
  - release
  - p2p

[cgwalters]

Yes; we have unit and integration tests using pull-local, so it's either some ostree bug or a bug in one of its dependencies or the environment.

(I know yocto uses a filesystem-operations-interception tool for example)

For the corrupted objects, try comparing the two - do you see a size/content mismatch or is it just metadata?

[ricardosalveti]

Was able to reproduce the issue locally, and indeed, it only happens when the sysroot gets created via yocto, probably because it is not calling the commands as root (I can reproduce with fakeroot).

Noticed the error when trying a pull-local as a normal user, and got:
error: Writing content object: fchownat: Operation not permitted

Calling with fakeroot made pull-local pass, but then ostree fsck failed:
error: In commits 3f294e31582a0687049febe199d13d7598e20d49ac9a2f4277bdb46a986440a1: fsck content object fd6f197474e5a9c583845dbcf23a5e41e9aed63ba66b3674ef35e38ea3ec9b03: Corrupted file object; checksum expected='fd6f197474e5a9c583845dbcf23a5e41e9aed63ba66b3674ef35e38ea3ec9b03' actual='85ee9d9e4ab71a38c5dabdd8ea2df985897be6f4172f819403ee070d062eaf5f'

If I call with sudo instead of fakeroot, everything works as expected.

@cgwalters this is probably not yocto specific (we might have similar issues with other build systems), and I know repo itself has an option for bare-user, but in this case the sysroot gets created with ostree admin, as you can see at https://github.com/uptane/meta-updater/blob/kirkstone/classes/image_types_ota.bbclass#L9

The flow is basically:

• ostree admin init-fs
• ostree admin os-init
• ostree pull-local
• ostree admin deploy

After that we just create the ota-ext4 image, which gets flashed on the device.

[ricardosalveti]

Yeah, systemd and virtualization won't be options for Yocto, so need to investigate this further to understand what exactly is causing the hash mismatch here.

[cgwalters]

The concept of using .proto files or equivalent isn't specific to systemd and can be used outside of it - it's a feature of the mkfs.$fs tools (in various disparate formats). It's just that systemd-repart streamlines and encourages it.

[cgwalters]

so need to investigate this further to understand what exactly is causing the hash mismatch here.

But yes, agreed. Can you briefly describe the high level architecture here: We are going from a bare-user to a bare repo, right? The most likely thing offhand I'd guess is that we're losing access to the user.ostreemeta xattr that is a key aspect of the bare-user repo.

[ricardosalveti]

At the moment we're going from archive-z2 (https://github.com/uptane/meta-updater/blob/kirkstone/classes/image_types_ostree.bbclass#L138) to bare I believe (part of ostree admin init-fs: https://github.com/uptane/meta-updater/blob/kirkstone/classes/image_types_ota.bbclass#L9).

The flow in yocto is the following:

Yocto rootfs, local ostree repo and ostree commit:

• Yocto rootfs created (via packages)
• converted to an ostree-compatible rootfs format (https://github.com/uptane/meta-updater/blob/kirkstone/classes/image_types_ostree.bbclass#L38)
• local repo initialized with archive-z2, rootfs files commited (part of the ostreecommit task: https://github.com/uptane/meta-updater/blob/kirkstone/classes/image_types_ostree.bbclass#L137)

Ostree-compatible rootfs:

• ota-sysroot create with ostree admin init-fs
• boot files created manually (based on systemd-boot, u-boot, etc)
• ostree pull-local (from the ostree repository generated after the Yocto rootfs was created)
• ostree deploy
• ext4 image creation out of the ota-sysroot folder

What is failing here is the pull-local step.

[ricardosalveti]

By failing I mean that ostree fsck will point out that there is a corrupted file, pull-local itself doesn't give any error. If I run pull-local as root then fsck says everything is correct and fine.

[codeshravan]

Approach1 i tried and its working fine.

ostree --repo=shravan init --mode=archive-z2
ostree --repo=shravan pull-local --remote=poky actual_repo_created_on_content(image_types_ostree.bbclass)
ostree --repo=shravan/ fsck
Validating refs...
Validating refs in collections...
Enumerating objects...
Verifying content integrity of 1 commit objects...
fsck objects (20407/20407) [=============] 100%
object fsck of 1 commits completed successfully - no errors found.

Approach2 Ricardo already tried and its not working and it will work with sudo.

ostree admin --sysroot="shravan1" init-fs --modern "shravan1"
ostree admin --sysroot="shravan1" os-init "poky"
ostree --repo=shravan1/ostree/repo pull-local --remote=poky actual_repo_created_on_content(image_types_ostree.bbclass)
error: Writing content object: fchownat: Operation not permitted

It means problem is not with pull-local command, problem with "ostree admin --sysroot="shravan1" init-fs --modern "shravan1"" command where we initialized the sysroot along with repo.

@cgwalters Would you please help here

[jlebon]

The main difference is that in the first case, you're pulling into an archive repo and so don't need to actually chown files. In the latter, you're pulling into a bare repo and so it wants to chown files, which will fail if you're not root.

I didn't look deeply here but ISTM the issue is that you're using fakeroot to pull into a bare repo. A e.g. root-owned file will not actually be owned by root, which throws off the checksum, hence why fsck fails.

[codeshravan]

@jlebon if i change the mode from bare to archive then its working fine.

ostree --repo=shravan1/ostree/repo init --mode=archive-z2
ostree admin --sysroot="shravan1" init-fs --modern "shravan1"
ostree admin --sysroot="shravan1" os-init "poky"
ostree --repo=shravan1/ostree/repo pull-local --remote=poky actual_repo_created_on_content(image_types_ostree.bbclass)

So can we use same command here as well before line9:https://github.com/uptane/meta-updater/blob/kirkstone/classes/image_types_ota.bbclass#L9 to resolve this issue??

[jlebon]

You really want your sysroot repo to be bare because that's the hardlink farm. It may somewhat work, though that's a pretty baked assumption in libostree.

[codeshravan]

Then only way to maintain bare because that's the fundamental requirement of libostree is to create the originally repo with bare mode: uptane/meta-updater@646b792

[codeshravan]

Build failed as expected with this :error: Writing content object: fchownat: Operation not permitted