[macOS] Fix Metal RHI 3D viewer crashes and enable qtAliceVision plugins

[NeverGET]

Summary

• Fix Metal RHI 3D viewer crashes on macOS caused by missing vertexNormal attributes in custom Qt3D geometries
• Compute flat normals via dFdx/dFdy derivatives instead of requiring normals as vertex attributes in SphericalHarmonics shaders
• Add ensureNormals() to dynamically patch loaded meshes (OBJ/PLY) that lack normals
• Add PySide6 bundled QML module path discovery (Qt3D, QtQuick.Scene3D)
• Add macOS support to setupEnvironment() (Metal RHI backend default, Qt plugin path existence guards)
• Add groupDesc backward compatibility for AliceVision node descriptors

Acknowledgments

Huge thanks to @music-dsp-collection for the incredible work on MTL-AliceVision (alicevision/AliceVision#2019) — bringing Metal GPU support and the complete macOS build pipeline to AliceVision is a massive achievement. And to the entire @alicevision team for building such an amazing open-source photogrammetry ecosystem. This PR is a small contribution building on top of their foundational work to help get the Meshroom 3D viewer running smoothly on macOS.

Context

While testing Meshroom on macOS 26.3 (Apple Silicon M4 Max, Mac15,6) with the MTL-AliceVision bundle from PR alicevision/AliceVision#2019, we found that the 3D viewer crashed due to Metal's strict vertex descriptor validation when geometries lacked normal attributes that built-in Qt3D materials expect.

Root cause: Metal RHI validates that all vertex shader inputs (layout(location = N) in ...) are satisfied by the geometry's vertex descriptor. When a shader declares vertexNormal but the geometry only provides vertexPosition, Metal's pipeline state creation fails, crashing the application.

Fixes applied:

1. SphericalHarmonics shaders: Remove vertexNormal input entirely; compute flat normals from dFdx/dFdy derivatives of world position in the fragment shader
2. Custom geometries (Grid3D, BoundingBox, Locator3D): Add default (0, 1, 0) normal attributes
3. Loaded meshes: New Scene3DHelper.ensureNormals() dynamically adds normals to geometries missing them
4. Platform support: Metal RHI default, Qt plugin path guards

Test Plan

• Launch Meshroom on macOS with Metal RHI (no QT3D_RENDERER override needed)
• Enable 3D viewer — no crash
• Load mesh from Texturing node — renders correctly
• Resize window — no crash
• Switch render modes (Solid, Wireframe, Textured, SH)
• Verify qtAliceVision plugin loads (no "Missing plugin" warning)
• Verify EXR textures load via QtAliceVisionImageIO
• Grid, BoundingBox, Locator render without crash
• All pipeline nodes execute correctly (camera init, feature extraction, etc.)

Tested on

• macOS 26.3 (Tahoe) / Apple Silicon M4 Max (Mac15,6)
• Python 3.13 / PySide6 6.8.x
• Qt 6.8.x with Metal RHI backend
• MTL-AliceVision bundle from PR constructing a mesh from an AI generated set of images. #2019

🤖 Generated with Claude Code

[philippremy]

This is AWESOME! I ran into all these issues and did some very dirty hacking to get a bare minimum of it working 😅.
That would have been a follow-up project for me eventually but honestly: I am really not into Qt and QML... you are a lifesaver! I'll make sure to do some testing the following days ^^.
So a big thank you for looking into this!!!

[NeverGET]

You're welcome, BTW I bundled the app and using daily 😄.
I didnt test all the features in the app, i am still investigating the app, if you have a test list for it, i can try on and keep fixing the issues for MacOS

[fabiencastan]

What's the reason to change that?

[NeverGET]

Actually the mesh preview is looking a bit phased off, like whiteish. I tried to make it look like full saturated colour but it didn't worked either and I was feeling very happy about the app is working, I directly go into PR.

Tldr; it just a mess I forgot to revert

[fabiencastan]

Could you fix that?

[codecov]

Codecov Report

❌ Patch coverage is 15.00000% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.21%. Comparing base (b913d50) to head (4097642).
⚠️ Report is 12 commits behind head on develop.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
meshroom/__init__.py	13.33%	13 Missing ⚠️
meshroom/core/cgroup.py	0.00%	3 Missing ⚠️
meshroom/core/desc/attribute.py	50.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3030      +/-   ##
===========================================
- Coverage    83.35%   83.21%   -0.14%     
===========================================
  Files           73       73              
  Lines         9882     9901      +19     
===========================================
+ Hits          8237     8239       +2     
- Misses        1645     1662      +17     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[fabiencastan]

Suggested change

	@deprecated.depreciateParam("group", "Param 'group' on {name} should not be used anymore. Please use 'commandLineGroup' instead")
	@deprecated.depreciateParam("group", "Param argument 'group' on {name} should not be used anymore. Please use 'commandLineGroup' instead")
	@deprecated.depreciateParam("groupDesc", "GroupAttribute argument 'groupDesc' on {name} should not be used anymore. Please use 'items' instead")

[fabiencastan]

@NeverGET Could you rebase this PR onto the develop branch (instead of a merge)?
@philippremy Could you review and test this PR on macos?

[philippremy]

@philippremy Could you review and test this PR on macos?

Will do. I think I have a valid AdaptiveCpp build now and will report any issues asap. I'll be on a train ride tonight, so maybe I can take it for a spin then.

[philippremy]

I just went ahead and did the testing:

(+): It does not crash anymore, models and images are loaded correctly and (afaict) the 3D Viewer now works correctly on macOS (🚀)1

(-): I don't know what to think about the DYLD_FALLBACK_LIBRARY_PATH and AV_BUNDLE parts.

1. Regarding DYLD_FALLBACK_LIBRARY_PATH, using environment variables which directly affect the dynamic linker is really not a good idea and an inherent safety hazard (and I assume it already is/soon will be nonfunctional because of macOS sandbox restrictions and System Integrity Protection). I would rather prefer to modify the relevant libraries (namely AliceVision and QtAliceVision) and set the correct rpaths when building them. For AliceVision, I guess we are already fine since my PR got merged. The resulting bundle folder can just be renamed to aliceVision and works out of the box. Remaining are the libraries for QtAliceVision, which would be a very simple PR I could submit.

2. The AV_BUNDLE environment variable assumes the bundle structure from my heavily-modified MTL-AliceVision fork (i.e., a macOS .bundle folder) afaik. Since the relevant macOS PR got merged over at AliceVision, macOS now also uses a standard Unix root bundle (i.e., lib, bin, share). So I guess this is not needed anymore, since macOS now essentially behaves like Linux.

Everything else LGTM. But I would really prefer to not use/introduce more environment variables but instead try to reduce them in general.

[NeverGET]

Hi @fabiencastan @philippremy — apologies for the long silence on this one. The past couple of months turned out to be unexpectedly busy and I only now got the time to come back and properly address the review. Thanks a lot for the detailed feedback and for taking the time to test it on macOS.

I've pushed an updated branch that addresses every point:

• Rebased onto the latest develop — the merge commit is gone, it's a clean linear history now.
• MaterialSwitcher.qml: reverted the ambient/shininess/specular override. It was a leftover experiment that didn't actually change anything visually, so the textured material is back to using the mesh's own material properties.
• attribute.py: applied the suggested @deprecated.depreciateParam("groupDesc", ...) decorator.
• DYLD_FALLBACK_LIBRARY_PATH: removed. Agreed it shouldn't be set from Python — library resolution should rely on the rpaths baked into the AliceVision bundle, which works now that the macOS PR is merged. Happy to follow up once the QtAliceVision rpath PR lands.
• AV_BUNDLE: removed. Since the macOS support PR was merged, macOS uses the standard Unix bundle layout, so the special-case path discovery is no longer needed.

What remains is the macOS RHI/shader work — Metal normal fixes, OpenGL fallback techniques, the cgroup fix, and the Qt plugin-path existence guards. I re-tested locally on macOS (Apple Silicon, Metal RHI): the full pipeline recomputes end to end and the 3D viewer works.

Let me know if anything else needs adjusting.

[servantftransperfect]

Hello,

Thanks for your contribution.

• I don't think you understood the cgroup usage. This is passed to the command line as a user defined cap on the number of cpu/ max memory to use. The command line are already using the necessary logic to detect the real number of cpus (available memory). This chunk of code is necessary for large shared computers which have per user limitations. It should return -1 if cgroup does not exists. See this link : https://github.com/alicevision/AliceVision/blob/develop/src/aliceVision/system/hardwareContext.cpp .

• Why are you creating alternatives to RHI shaders ? can you describe a real use case ?

Thanks !

[NeverGET]

Hi @servantftransperfect — thanks, both points are fair.

cgroup: you're right, I had the semantics backwards. Looking at the hardwareContext.cpp you linked, --maxCores is meant to express a user-imposed cap, not the detected core count — and the existing OSError-catching path in getCgroupCpuCount() already returns -1 correctly on non-Linux. So the early-return I added was both unnecessary and semantically wrong.

For context on why I touched that file in the first place: while debugging the macOS crashes I also noticed Meshroom wasn't using all CPU threads on my machine, and I tried a few things to nudge it — that change was one of those experiments and ended up being a leftover. With it dropped, AliceVision's own detection takes over, which is what should happen.

OpenGL fallback shaders: no real-world use case. The actual macOS Metal fix is in the modified SphericalHarmonics.vert/frag (RHI path); the parallel _gl technique I added was speculative — it would only kick in if someone explicitly set QT3D_RENDERER=opengl to force Qt3D onto its legacy non-RHI OpenGL backend, which isn't something I had a concrete reason for. Dropped.

Branch is force-pushed; the PR now has 6 focused commits.

[servantftransperfect]

I am not sure to know why, but without your PR, the bounding box and the widgets were correctly lighted using the directional light (in the 3D viewer).

Now using your PR, as you explicitely added wrong normals, the lighting is weird.

I am not sure we wanted (in the first place) those widgets to be affected by the lighting.

Would you have time finding a solution to make them unlit ? I can see in the doc there is a PerVertexColorMaterial which requires to replace normals with color.