Cache submodule into git db #16246

avnyu · 2025-11-12T16:33:56Z

What does this PR try to resolve?

My attempt to continue #10279.

This:

Moves the code for creating git db into GitSource::fetch_db
Creates GitSource for each submodules
Replaces fetch inside update_submodule by GitSource::fetch_db and db.copy_to
Removes recursive update_submodules calls cos db.copy_to already recursive.

How to test and review this PR?

I tested using the original pull method:

~/.cargo/target/debug/cargo update -p boring --precise 46787b7b6909cadf81cf3a8cd9dc351c9efdfdbd
~/.cargo/target/debug/cargo update -p boring --precise c037a438f8d7b91533524570237afcfeffffe496

and confirmed that the time to do the second update is negligible.
Also test if it can fetch submodule offline using the downloaded git db

git clone https://github.com/pop-os/cosmic-files && cd cosmic-files
~/.cargo/target/debug/cargo fetch --locked --target=$(rustc --print host-tuple)
rm -rf ~/.cargo/git/checkout ~/.cargo/target/debug/cargo fetch --locked --target=$(rustc --print host-tuple) --offline

rustbot · 2025-11-12T16:34:02Z

r? @weihanglo

rustbot has assigned @weihanglo.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

epage · 2025-11-12T16:50:16Z

It would help reviewers if you break this down into atomic commits. For example, by making a commit for extracting update_db, it becomes clear whether it was purely an extraction or there are also other changes.

See https://doc.crates.io/contrib/process/working-on-cargo.html#submitting-a-pull-request

avnyu · 2025-11-12T17:59:26Z

Done, the "extract code into update_db" part is a single commit now. I think the remains part needs to be in the same commit

epage · 2025-11-12T18:04:59Z

src/cargo/sources/git/utils.rs

            .with_context(|| {
                let name = child.name().unwrap_or("");
                format!("failed to fetch submodule `{name}` from {child_remote_url}",)
            })?;


Seems like we should have this also on update_db and copy_to

I tried and the testsuite failed. Should I also adjust testsuite more? I do try to keep it unchanged as much as possible.

Depends on the failure

How about put it only after db.copy_to? I tested and it do pass the testsuite without further modification. The original fetch was replaced by (gitsource create, update_db and db.copy_to), so put the error context to after db.copy_to seem enough?

aeda351
Done, and adjust the testsuite.

See an alternative in #16246 (comment)

epage · 2025-11-12T18:06:33Z

src/cargo/sources/git/utils.rs

+                SourceId::from_url(&format!("git+{child_remote_url}#{head}"))?,
                gctx,
-                RemoteKind::GitDependency,
            )


What impact does this have on our progress bars?

Mostly the same? The old fetch will have a single progress bar, which now changes into a progress bar for update_db and a brief progress bar after for db.copy. This progress bars behavior should be the same as when update git source. Most of the time I don't even notice the brief one.

ea4eb86 should fix when submodule database exist but still print "updating submodule"

src/cargo/sources/git/utils.rs

weihanglo

Thanks for the contribution!

We should have tests for this new caching behavior, especially for nested submodules. See https://doc.crates.io/contrib/process/working-on-cargo.html#making-a-change for how, and also mind the atomic commit pattern (adding test first capturing the existing behavior, and the fix commit showing behavior change through test diffs)

View changes since this review

src/cargo/sources/git/source.rs

weihanglo · 2025-11-29T17:17:21Z

src/cargo/sources/git/utils.rs

-                &child_remote_url,
-                &reference,
+            let mut source = GitSource::new(
+                SourceId::from_url(&format!("git+{child_remote_url}#{head}"))?,


Instead of constructing URL manually, we have a SourceId::for_git for the purpose.

a474768, I notice the GitSource created by for_git always has precise=None, result to locked_rev being Deferred, so fetch_db always try to fetch if online. Is it appropriate to use with_git_precise for this?

Nice catch! Yeah would be good if we could add a comment why we need to have with_git_precise there.

02b8bf9 Done.

src/cargo/sources/git/utils.rs

weihanglo · 2025-11-29T18:11:22Z

src/cargo/sources/git/utils.rs

                    let name = child.name().unwrap_or("");
                    format!("failed to fetch submodule `{name}` from {child_remote_url}",)
                })?;
+            guard.mark_ok()?;


The checkout is marked ready then we recurse into submodules. I feel like that means the checkout is not ready yet.

Would this situation happens? _User cancels the operation before recursing to nested submodules. They later they dependency on this submodule directly and Cargo assumes it is fresh so never recurse into nested submodules.

I overlook when inline db.copy_to, my bad. But now the code almost the same as db.copy_to, with db.copy has additional check for freshness, should I just use db.copy_to instead for reduce code dups?

Yeah, if that works (does look like will work).

Do I need to push this myself? #16246 (review) If you can do it, please do. Thanks.

I am offering reorder commits, not writing a fix. Would appreciate if you can help write the fix :)

02b8bf9 Done.

src/cargo/sources/git/utils.rs

src/cargo/sources/git/source.rs

weihanglo · 2025-12-09T14:06:30Z

@rustbot author

rustbot · 2025-12-09T14:06:36Z

Reminder, once the PR becomes ready for a review, use @rustbot ready.

rustbot · 2025-12-10T03:54:35Z

This PR was rebased onto a different master commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

avnyu · 2025-12-10T04:46:39Z

Sorry for the late response, I force-pushed the commits. About new tests, if I am not mistaking, it should be only about the git db cache, not the correctness of git repo checkouts, as it should be already covered? Currently I can't find api for manipulating the cached git db, or example about testing the git cache db

weihanglo

Overall looks like. Let me know if you plan to reorganize the commits, or I can do that for you. Thanks!

View changes since this review

tests/testsuite/git.rs

src/cargo/sources/git/source.rs

src/cargo/sources/git/utils.rs

weihanglo · 2025-12-13T06:48:11Z

src/cargo/sources/git/utils.rs

                    let name = child.name().unwrap_or("");
                    format!("failed to fetch submodule `{name}` from {child_remote_url}",)
                })?;
+            guard.mark_ok()?;


Yeah, if that works (does look like will work).

weihanglo · 2025-12-13T06:48:34Z

src/cargo/sources/git/utils.rs

-                &child_remote_url,
-                &reference,
+            let mut source = GitSource::new(
+                SourceId::from_url(&format!("git+{child_remote_url}#{head}"))?,


Nice catch! Yeah would be good if we could add a comment why we need to have with_git_precise there.

src/cargo/sources/git/source.rs

avnyu · 2025-12-13T12:33:34Z

Overall looks like. Let me know if you plan to reorganize the commits, or I can do that for you. Thanks!

View changes since this review

If you can do it, please do. Thanks!

fetch each submodules using git database then checkout

weihanglo

@avnyu I rearranged the commits to reflect behavior change in the fix commit. See 4f02e47.

Thanks for working with us!

View changes since this review

rustbot assigned weihanglo Nov 12, 2025

rustbot added A-git Area: anything dealing with git S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 12, 2025

This comment has been minimized.

Sign in to view

avnyu force-pushed the cached-submodules branch from 70350b1 to ffd78e6 Compare November 12, 2025 17:52

This comment has been minimized.

Sign in to view

epage reviewed Nov 12, 2025

View reviewed changes

src/cargo/sources/git/utils.rs Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

weihanglo reviewed Nov 29, 2025

View reviewed changes

rustbot added S-waiting-on-author Status: The marked PR is awaiting some action (such as code changes) from the PR author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Dec 9, 2025

avnyu force-pushed the cached-submodules branch from 8705c44 to a474768 Compare December 10, 2025 03:54

weihanglo reviewed Dec 13, 2025

View reviewed changes

refactor: extract into GitSource::fetch_db

1774e76

avnyu force-pushed the cached-submodules branch from a474768 to fcb41ba Compare December 14, 2025 14:15

avnyu added 3 commits December 14, 2025 10:29

test: cache git submodule tests

9e9d3db

feat: recursive cache git submodules

4f02e47

fetch each submodules using git database then checkout

fix: remove extra 'update git submodule' when the submodule is cached

cf022a5

weihanglo force-pushed the cached-submodules branch from fcb41ba to cf022a5 Compare December 14, 2025 15:36

weihanglo approved these changes Dec 14, 2025

View reviewed changes

weihanglo enabled auto-merge December 14, 2025 15:39

weihanglo added this pull request to the merge queue Dec 14, 2025

Merged via the queue into rust-lang:master with commit 0101bde Dec 14, 2025
28 checks passed

Cache submodule into git db #16246

Cache submodule into git db #16246

Conversation

avnyu commented Nov 12, 2025 • edited by weihanglo Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR try to resolve?

How to test and review this PR?

Uh oh!

rustbot commented Nov 12, 2025

Uh oh!

epage commented Nov 12, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

avnyu commented Nov 12, 2025

Uh oh!

epage Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment has been minimized.

weihanglo left a comment • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

weihanglo commented Dec 9, 2025

Uh oh!

rustbot commented Dec 9, 2025

Uh oh!

rustbot commented Dec 10, 2025

Uh oh!

avnyu commented Dec 10, 2025

Uh oh!

weihanglo left a comment • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

avnyu commented Nov 12, 2025 •

edited by weihanglo

Loading

epage Nov 12, 2025 •

edited

Loading

weihanglo left a comment •

edited by rustbot

Loading

weihanglo left a comment •

edited by rustbot

Loading

weihanglo left a comment •

edited by rustbot

Loading