Skip to content

ci(macos,windows): drop submodules:true, use selective submodule list#5992

Merged
Fedr merged 5 commits intomasterfrom
ci/checkout-third-party-submodules-selective
Apr 27, 2026
Merged

ci(macos,windows): drop submodules:true, use selective submodule list#5992
Fedr merged 5 commits intomasterfrom
ci/checkout-third-party-submodules-selective

Conversation

@Fedr
Copy link
Copy Markdown
Contributor

@Fedr Fedr commented Apr 26, 2026

Summary

Bring build-test-macos.yml and build-test-windows.yml in line with the selective-list pattern the other six workflows in this repo (build-test-emscripten, build-test-linux-vcpkg, build-test-ubuntu-arm64, build-test-ubuntu-x64, pip-build, update-docs-manual) already use, and refine the pattern in all eight to recurse only into thirdparty/mrbind/deps/cppdecl — the one nested submodule MeshLib's build actually consumes — instead of recursing into every entry's .gitmodules blindly.

After this PR, every workflow that follows the Checkout + Checkout third-party submodules two-step pattern has the same shape: parent actions/checkout without submodules: true, followed by a selective git submodule update --init --depth 1 of just the submodules that workflow consumes, plus an explicit git -C thirdparty/mrbind submodule update --init --depth 1 deps/cppdecl for the one path that needs recursion.

Why

Both workflows previously had:

- name: Checkout
  uses: actions/checkout@v6
  with:
    submodules: true             # ← inits all 29 top-level submodules

- name: Checkout third-party submodules
  run: |
    git submodule update --init --recursive --depth 1 thirdparty/mrbind

submodules: true clones every entry in .gitmodules (29 top-level submodules — including heavyweights like openvdb, GDCM, libE57Format, mbedtls, onetbb, libzip, c-blosc) regardless of whether the build will use the source. macOS gets most of those from Homebrew or builds them via build_thirdparty.sh's thirdparty CMakeLists; Windows gets them from vcpkg. Neither workflow actually consumes most of those submodules' source.

This PR drops submodules: true and lists exactly the submodules each platform actually consumes:

Platform Submodules listed Why
macOS (16) imgui, eigen, parallel-hashmap, expected header-only, consumed directly
googletest, OpenCTM-git, libE57Format, glad, tinygltf, laz-perf built via thirdparty/CMakeLists.txt's Apple branch
clip built standalone via scripts/build_thirdparty.shclip.sh
fastmcpp, nlohmann-json, cpp-httplib add_subdirectory(${MESHLIB_THIRDPARTY_DIR}/fastmcpp); nlohmann-json + cpp-httplib are fastmcpp's transitive deps
mrbind, mrbind-pybind11 binding-generator build tools
Windows (11) imgui, eigen, parallel-hashmap, expected header-only
OpenCTM-git, laz-perf source/OpenCTM/CMakeLists.txt and source/laz-perf/CMakeLists.txt add_subdirectory them
fastmcpp, nlohmann-json, cpp-httplib same as macOS
mrbind, mrbind-pybind11 binding-generator build tools

Windows lists fewer because vcpkg supplies the equivalents — except for the five that get pulled in via add_subdirectory from source/<x>/CMakeLists.txt and so need source even on the vcpkg path.

Recursion is targeted

Of the 16 submodules on the macOS list, only thirdparty/mrbind has a nested submodule MeshLib's build actually needs (deps/cppdecl, the codegen library mrbind links against). Two other submodules carry nested .gitmodules entries but neither is consumed by MeshLib:

  • thirdparty/expectedcmake/tl-cmake is unreferenced by the cantonios fork's CMakeLists at the pinned commit.
  • thirdparty/libE57Formattest/extern/googletest is gated behind libE57Format's own test target, which MeshLib never builds.

The other 13 have no nested .gitmodules at all — --recursive on them was a strict no-op.

So the workflow does:

- name: Checkout third-party submodules
  run: |
    # Selective init -- parent Checkout drops submodules:true.
    # https://github.com/actions/checkout/issues/1779
    git submodule update --init --depth 1 \
      thirdparty/imgui \
      thirdparty/eigen \
      ...
      thirdparty/mrbind \
      thirdparty/mrbind-pybind11
    # mrbind needs deps/cppdecl; recurse only there
    git -C thirdparty/mrbind submodule update --init --depth 1 deps/cppdecl

Bash workflows use \ continuation; the Windows workflow uses pwsh backtick (`) for the same shape. (See the Windows step.)

Measured impact

Comparing the latest run 24994826587 (764cd258) against master baseline 24987634018 (466884e1). Times = Checkout + Checkout third-party submodules summed (seconds):

Platform Master This PR Δ
macOS x64 Release (macos-15-intel) 88 53 −35
macOS arm64 Debug (macos-latest) 112 47 −65
macOS arm64 Release (self-hosted, warm) 10 8 −2
Win msvc-2019 Release CMake 96 31 −65
Win msvc-2019 Debug CMake (iter-debug) 94 26 −68
Win msvc-2022 Debug MSBuild 86 30 −56
Win msvc-2022 Release CMake 112 30 −82
Sum across 7 legs 598 225 −373 (−62 %)

Per master run that's ~6.2 minutes of cumulative runner-time saved across these 7 GitHub-hosted/self-hosted legs, with ~50–80 s shaved off each Windows leg's critical path.

The self-hosted arm64 Release runner is essentially neutral — its .git/modules/ cache stays warm between runs so neither pre- nor post-PR actually clones much there. The savings show up on the GitHub-hosted ephemeral-VM legs.

Pattern uniformity

All eight workflows that follow the Checkout + Checkout third-party submodules two-step pattern now use the same shape — no submodules: true on the parent, selective list with explicit cppdecl recursion in the follow-up step. The remaining occurrences of submodules: true / submodules: recursive in this repo are in workflows that don't follow that pattern (prepare-images.yml, update-docs.yml, update-win-version.yml, the macOS- and Windows-specific jobs in pip-build.yml) — those genuinely need the full tree for unrelated reasons and are out of scope here.

Iteration history

CI on this branch surfaced changes incrementally:

  • 55816785 — initial drop of submodules: true on macOS + Windows with selective lists.
  • 60fc8e3f — added OpenCTM-git, laz-perf, fastmcpp, nlohmann-json, cpp-httplib after Windows configure-time errors on source/{OpenCTM,laz-perf,fastmcpp}/CMakeLists.txt add_subdirectory calls.
  • d0c12915 — added clip after macOS build_thirdparty.sh failed on the standalone clip.sh step.
  • 0f13185b — dropped --recursive, added explicit git -C thirdparty/mrbind submodule update --init --depth 1 deps/cppdecl. Windows used per-line separate commands at this stage.
  • 764cd258 — switched Windows from per-line separate commands to a single batched call with pwsh backtick continuation, recouping ~7 s/leg of per-invocation overhead.

Replaces

Closed PR #5991, which took the opposite direction (add submodules: true everywhere, trim the second step to just thirdparty/mrbind). The selective-list approach in this PR is what the repo had been using on its other six workflows; extending it to macOS and Windows is the simpler and faster end state.

🤖 Generated with Claude Code

Fedr and others added 3 commits April 26, 2026 16:38
Bring macos and windows workflows in line with the selective-list
pattern the other six workflows already follow:

- Remove `submodules: true` from the parent `actions/checkout`. That
  flag was inflating each checkout to all 29 top-level submodules,
  most of which neither workflow actually needs in source form.
- Keep the existing "Checkout third-party submodules" step name and
  expand its `git submodule update` argument list to the submodules
  that are actually consumed by the build:
    macos:   imgui, eigen, parallel-hashmap, expected,
             googletest, OpenCTM-git, libE57Format, glad,
             tinygltf, laz-perf, mrbind, mrbind-pybind11.
             (header-only consumed directly + ones built from source
             via thirdparty/CMakeLists.txt's Apple branch + build
             tools.)
    windows: imgui, eigen, parallel-hashmap, expected,
             mrbind, mrbind-pybind11.
             (header-only consumed directly + build tools. The heavy
             stack on windows comes from vcpkg.)

Saves the parent checkout from cloning ~17–23 unused submodules per
matrix leg.

If a submodule is missing from either list, the build will fail with
a clear cmake / cpath error pointing at the missing path; the list
can be expanded incrementally.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…it+laz-perf

CI run 24958011966 on the previous commit failed at cmake configure
on Windows with three "source directory does not contain a
CMakeLists.txt" errors -- for thirdparty/fastmcpp, thirdparty/laz-perf,
and thirdparty/OpenCTM-git.

The three source/<x>/CMakeLists.txt files that explicitly pull these in
via add_subdirectory(${MESHLIB_THIRDPARTY_DIR}/<y>):

  source/fastmcpp/CMakeLists.txt:20 -> thirdparty/fastmcpp
  source/laz-perf/CMakeLists.txt:7 -> thirdparty/laz-perf
  source/OpenCTM/CMakeLists.txt:1  -> thirdparty/OpenCTM-git

Add fastmcpp + its dependency siblings (nlohmann-json, cpp-httplib --
they travel with fastmcpp per the existing linux-vcpkg selective list)
plus OpenCTM-git and laz-perf to the Windows list. macOS already has
OpenCTM-git and laz-perf; just add fastmcpp/nlohmann-json/cpp-httplib
there for symmetry.

Updated lists:

  Windows (11): imgui, eigen, parallel-hashmap, expected,
                OpenCTM-git, laz-perf,
                fastmcpp, nlohmann-json, cpp-httplib,
                mrbind, mrbind-pybind11

  macOS (15):   imgui, eigen, parallel-hashmap, expected,
                googletest, OpenCTM-git, libE57Format, glad,
                tinygltf, laz-perf,
                fastmcpp, nlohmann-json, cpp-httplib,
                mrbind, mrbind-pybind11

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Run 24958327115 / job 73080428199 failed at:

    CMake Error: The source directory
        "/Users/runner/work/MeshLib/MeshLib/thirdparty/clip"
    does not appear to contain CMakeLists.txt.

build_thirdparty.sh on macOS runs scripts/thirdparty/clip.sh against
${MESHLIB_THIRDPARTY_DIR}/clip as a separate build step (line 155),
so thirdparty/clip's source must be checked out. Only macOS needs
this -- Windows gets clip from the vcpkg overlay port at
thirdparty/vcpkg/ports/clip/.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@Fedr Fedr requested a review from Grantim April 27, 2026 08:23
Fedr added 2 commits April 27, 2026 13:13
Of the 16 submodules in the macOS list (and equivalents elsewhere), only
mrbind has a nested submodule MeshLib actually needs (deps/cppdecl).
expected and libE57Format have nested .gitmodules entries but their
content is unused by MeshLib's build.

Replace `--recursive` with an explicit `git -C thirdparty/mrbind
submodule update --init --depth 1 deps/cppdecl` so the intent reads
straight off the workflow.

Each submodule on its own line. macOS, Ubuntu, Emscripten, linux-vcpkg,
pip-build, update-docs-manual use bash `\` continuation; Windows uses
one git command per line (pwsh-friendly without backtick).

Also shorten the verbose comment block I added on macOS and Windows
to a 2-line one matching the style of the other workflows.
Recoups ~5-10s/leg of per-invocation overhead from running 12 separate
git submodule update commands. Single batched call matches the macOS
workflow's bash backslash continuation style; only the line-continuation
character differs (pwsh backtick vs bash backslash).
@Fedr Fedr merged commit c4452c0 into master Apr 27, 2026
35 checks passed
@Fedr Fedr deleted the ci/checkout-third-party-submodules-selective branch April 27, 2026 13:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants