Incorrect `mtime` when generating EROFS from OCI image

[Johan-Liebert1]

While parsing tar entries, we are most probably not taking into account the latest modification time for a directory.
Issue in detail

The layers of a certain image

"layers": [
    {
        "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
        "digest": "sha256:adad76f273c267201338df472d73aa2bfd8e5c5cdb72bd3abfe9a07de1b5d592",
        "size": 2260670
    },

    // other layers
    // ...

    {
        "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
        "digest": "sha256:0c31ce11ea42f8eabc0e18d7887bda77fea345403fe4ee9f5e2325f0e0bd85ef",
        "size": 36261536
    },
]

The directory in question is /usr/share/zoneinfo/Etc

mtime of said directory in layer adad76f273c267201338df472d73aa2bfd8e5c5cdb72bd3abfe9a07de1b5d592 is 1 Jan 1970.
mtime of said directory in layer 0c31ce11ea42f8eabc0e18d7887bda77fea345403fe4ee9f5e2325f0e0bd85ef is 20 May 2025.

In the final generated EROFS image, the mtime of said directory is recorded as 1 Jan 1970, which is incorrect as 0c31... is the newer layer.

This happens with other directories as well, not just the one mentioned above.

On an initial observation, this seems to only happen for directories and not for files.

The diff between an EROFS image generated by parsing an OCI image vs mounting a filesystem.

1c1
< / 0 40755 14 0 0 0 0.0 - - - security.selinux=system_u:object_r:root_t:s0
---
> / 0 40555 14 0 0 0 1747671178.0 - - - security.selinux=system_u:object_r:root_t:s0
10c10
< /etc/NetworkManager 0 40755 7 0 0 0 0.0 - - - security.selinux=system_u:object_r:NetworkManager_etc_t:s0
---
> /etc/NetworkManager 0 40755 7 0 0 0 1747211423.0 - - - security.selinux=system_u:object_r:NetworkManager_etc_t:s0
13c13
< /etc/NetworkManager/dispatcher.d 0 40755 5 0 0 0 0.0 - - - security.selinux=system_u:object_r:NetworkManager_dispatcher_script_t:s0
---
> /etc/NetworkManager/dispatcher.d 0 40755 5 0 0 0 1747211423.0 - - - security.selinux=system_u:object_r:NetworkManager_dispatcher_script_t:s0
21c21
< /etc/X11 0 40755 5 0 0 0 0.0 - - - security.selinux=system_u:object_r:etc_t:s0
---
> /etc/X11 0 40755 5 0 0 0 1747211422.0 - - - security.selinux=system_u:object_r:etc_t:s0
24,25c24,25
< /etc/X11/xinit 0 40755 4 0 0 0 0.0 - - - security.selinux=system_u:object_r:bin_t:s0
< /etc/X11/xinit/xinitrc.d 0 40755 2 0 0 0 0.0 - - - security.selinux=system_u:object_r:bin_t:s0
---
> /etc/X11/xinit 0 40755 4 0 0 0 1747211422.0 - - - security.selinux=system_u:object_r:bin_t:s0
> /etc/X11/xinit/xinitrc.d 0 40755 2 0 0 0 1747211422.0 - - - security.selinux=system_u:object_r:bin_t:s0
30c30
< /etc/alternatives 0 40755 2 0 0 0 0.0 - - - security.selinux=system_u:object_r:etc_t:s0


... goes on for a while

[Johan-Liebert1]

This could very well be an issue with the tar crate. Creating a small script to read all entries from both archives, and I see the mtime for everything is 0.

While, extracting the archive to disk and checking the mtime yields a different result.

[allisonkarlitskaya]

We chatted about this and did a bit of screen sharing. The issue is that if you don't have a directory entry in the "second" tar then when you extract it, but there are files contained in the directory, you need to make up a random mtime for the directory, which ends up being the time that it was extracted. When you then use overlayfs to stitch the filesystems back together, it'll use the random-time-of-extraction-mtime from the upper layer instead of the real timestamp from the lower layer because it has no idea that this was chosen out of thin air...

So this is basically a tension introduced by a strange interaction between the tar format, the form that podman stores its layers (ie: directories on the disk), and the technology it uses to stitch them back together (overlay). There's no way to have files present in an overlay without them being inside of some directory, and that directory needs metadata...

Because composefs-rs does all of this in memory and creates the tree in its final form it doesn't suffer from this limitation: you get whatever timestamp was on most-recently-seen version of the directory...

I have a sneaky suspicion that this issue is caused by the layer splitting code that bootc uses. @jeckersb could maybe look into this. For each file present in a layer tar stream there should always be all leading directory components of that file present in the same stream in order to avoid forcing the OS to choose a random timestamp and exposing that to the container at runtime. This is really important: we can't deal with this indeterminism if we want to do UKIs with composefs hashes built in...

[allisonkarlitskaya]

containers/buildah#5837 would help with this...

[jeckersb]

I have a sneaky suspicion that this issue is caused by the layer splitting code that bootc uses. @jeckersb could maybe look into this.

Yeah I'll take a look at this.

[cgwalters]

Yeah, I would agree that we should include all leading components as a general rule. I think the rpm-ostree splitting puts all the base directories in the first layer only today.

(This all interacts with dropping ostree/ with the tar stream which will simplify things a lot, ref https://gitlab.com/fedora/bootc/base-images/-/issues/58)

All this becomes clearly easier to handle if we avoid materializing individual layers to disk at all, as you said.

This is really important: we can't deal with this indeterminism if we want to do UKIs with composefs hashes built in...

That all said, I think we could also address this by starting with

diff --git i/src/tree.rs w/src/tree.rs
index f5cd26f..1a39e9e 100644
--- i/src/tree.rs
+++ w/src/tree.rs
@@ -43,7 +43,7 @@ pub struct Leaf<ObjectID: FsVerityHashValue> {
 
 #[derive(Debug)]
 pub struct Directory<ObjectID: FsVerityHashValue> {
-    pub stat: Stat,
+    pub stat: Option<Stat>,
     entries: BTreeMap<Box<OsStr>, Inode<ObjectID>>,
 }

And then we'd only fail when computing the final squashed tree if a parent was really missing from all layers.

[allisonkarlitskaya]

I don't think composefs-rs needs to be fixed: it's already doing the right thing. podman has the bug because of the way it uses overlayfs... basically if we could somehow apply your suggested fix to overlayfs by having a flag that instructed it to ignore stat information on a particular directory and look at the next lowerdir instead... then we could set this flag when creating an implicit parent directory that never explicitly appeared in the image and we'd skip the bogus metadata on that directory when viewed from inside the container...

[cgwalters]

basically if we could somehow apply your suggested fix to overlayfs by having a flag that instructed it to ignore stat information on a particular directory and look at the next lowerdir instead...

Yes, that's one way.

Yet another way is to also generate flattened merged filesystem trees beforehand instead of stitching together with overlayfs dynamically (this of course scope creeps into something like composefs), but if one has reflinks at least it's still just O(metadata).

Bigger picture I think it's actually a lot cleaner if we don't require all parent directories in OCI layers because it really helps lend itself towards things like "dynamic linking" as flatpak does it. Flatpak of course just makes apps live in /app in a totally different root to paper over this problem, but I do think it'd make sense to support such dynamic linking in general with OCI.

I don't think composefs-rs needs to be fixed: it's already doing the right thing.

So I think it would still make sense to enhance the composefs-rs model for this.

[cgwalters]

I think it's actually a lot cleaner if we don't require all parent directories in OCI layer

A quick search for "parent" in issues there turns up
opencontainers/image-spec#1221

[allisonkarlitskaya]

So I think it would still make sense to enhance the composefs-rs model for this.

What's the suggested change? With the exception of the root directory we currently never create directories implicitly...

[cgwalters]

Hmm right sorry, I guess the problem here is actually that the tar stream generated by podman is nondeterministic, and there's definitely nothing we can do about that here; and as you said earlier:

Because composefs-rs does all of this in memory and creates the tree in its final form it doesn't suffer from this limitation: you get whatever timestamp was on most-recently-seen version of the directory...

This sounds correct to me!

[cgwalters]

Another way to fix this is in containers/storage to track (xattr?) which directories are synthetic like this and then also omit them in the tar stream. Or maybe we actually walk the tar archive instead of the local filesystem to determine what to tar?
cc @giuseppe