meson2hermetic: the world's premiere build system converter - #15462
meson2hermetic: the world's premiere build system converter#15462gurchetansingh wants to merge 7 commits into
Conversation
c9ebe87 to
d3dff63
Compare
|
As relevant context, this tool was previously suggested by @gurchetansingh at #14134 /cc @dcbaker based on past discussions. |
|
I haven't yet had an opportunity to take a close look at this. I see that it adds two new commands, one being the command to actually generate the hermetic buildsystem e.g. bazel, and another "check-toolchain" command that appears to be similar in spirit to the existing env2mfile command. I'd like to hear a high-level explanation for what this tool does, to help better understand it. (And in particular, what are the challenges that make using machine files directly, not sufficient for your purposes? It looks like maybe the main differentiator is a lack of support for specifying the results of |
|
For reference, you can run the tool given the directions at: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39279 Right now, only
One thing I like about |
|
I will make a point to look at this in the near future. As a bit of an aside: I'm stilling working on the ability to have meson definitions for LLVM, but I need to get more changes into Meson's dependency representation to be able to model that. |
|
One thing I saw implementation wise from a quick glance, the new |
|
@gurchetansingh please add stubs to docs/markdown/Commands.md so that CI passes. To build the documentation you need to install hotdoc, then you can use |
d3dff63 to
f8081d8
Compare
Nice. LLVM to hermetic is challenging as well like Mesa. Plus, I think many devs would enjoy an improvement over CMake.
Added: But should IncludeType only be required with T.TYPE_CHECKING? Could be related to recent refactors. |
|
@gurchetansingh: interpreter/kwargs.py should only be imported inside a |
dcbaker
left a comment
There was a problem hiding this comment.
I've started looking purely at the implementation details of this, but I haven't looked too much at the design yet. It's a lot of code and this is giving me a chance to try to get familiar with it.
I haven't looked at the soong implementation yet, and I'm about half way through the instance code of the third patch. I still need to get to reference/ and to top level convert/ file and mconvert.py
One thing I'd really like to have is some more comments, particularly docstrings to help explain how all of the code fits together, and what each thing is for, there's a lot of code here and it would really help me to review it (and for us to maintain it).
| parser.add_argument("-o", "--output", default=None, help="Output file name.") | ||
|
|
||
|
|
||
| def run(options: argparse.Namespace) -> int: |
There was a problem hiding this comment.
We do have some uses of Protocol in these front end functions to help with accurate typing vs using Namespace, I'm not sure how practical that is in this case.
There was a problem hiding this comment.
left as is since most other commands use namespace
| class AbstractCompiler(Compiler): | ||
| def __init__(self, conf: T.Dict[T.Any, T.Any], *args: T.Any, **kwargs: T.Any): | ||
| self.conf = conf | ||
| super().__init__(*args, **kwargs) |
There was a problem hiding this comment.
I'm not crazy about the heavy use of Any here, but I need to think more about whether there's another good option.
There was a problem hiding this comment.
Indeed, why can't the list of arguments just be copied from Compiler?
| def find_library( | ||
| self, | ||
| libname: str, | ||
| extra_dirs: T.List[str], | ||
| libtype: LibType = LibType.PREFER_SHARED, | ||
| lib_prefix_warning: bool = True, | ||
| ignore_system_dirs: bool = False, | ||
| ) -> T.Optional[T.List[str]]: |
There was a problem hiding this comment.
Please do not use the black style especially this thing it does, it's a really weird style for python and we don't use it elsewhere, the ) -> T.Optional[...]: should not be dedented tot he same level as def
one of:
def find_library(self,
libname: str,
...): ...
def find_library(self, libname: str, ...): ...
def find_library(
self,
libname: str,
...):
...Would be consistent with our style elsewhere.
There was a problem hiding this comment.
I added a simple .style.yapf based on pep8 . It dedents to the same level as the function definition much less than ruff (the prior tool I used). It does in a few select cases, but I think those cases are reasonable. LMK otherwise, and I can skip the linters and hand-code to match the style you want.
f8081d8 to
a2f97c3
Compare
Added more docstring comment. I would recommend reviewers start in |
ac3a82c to
65b375a
Compare
Changelog since last update1. meson-to-Bazel for FuchsiaA Bazel backend functional enough to compile gfxstream_vk for Fuchsia. If you have:
here is the command to test: 2. Enhanced meson check-toolchainNow supports downloading from a [compiler_binaries.wrap] section with standard parameters (source_url, source_hash, source_filename). 3. Unit tests for CI/CDTo make sure the tool works as expected. Looks like the unittests are failing on platforms without Python 3.11, and that will be addressed in a future update. 4. formatter issues fixedThis style uses ruff as the formatter, but it's turned off for parts Meson developers especially hate (like the function defintions). One thing that would be nice if Meson had an officially supported formatter or style file (regardless of which one it is). meson2hermetic: a guide for reviewersAt its core, meson2hermetic is a two-step build system transpiler. 1. meson check-toolchainThis tool probes a toolchain (like the Android NDK or Fuchsia SDK) to see which headers, functions, and flags are supported, saving the results for the conversion process. 2. meson convertThis tool takes a standard Meson project and translates it into a different build system (Soong or Bazel) using the metadata gathered by the toolchain checker. FAQ1. Does the tool handle the entire Meson API?Not yet. build.Executable is one notable area where I haven't had a way to test yet (but it's on the radar). configuration-data too 2. How stable are the new APIs?The tool is useful for developers integrating FOSS projects into gigantic monorepos. This is a niche audience. So the tool's API will likely be considered "experimental" and subject to change for a while (six months to 1 year). Specifically, this flow uses TOML files to handle multiple compilers/sysroots, but I'm unsure if the approach is exactly right (though I don't think Bazel handles that well either). If a better representation emerges, this tool will likely migrate. 3. Is this only useful for Mesa3D?Some people also need hermetic QEMU builds, and someone was playing around with DPDK + Bazel (magma-gpu/mesonbuild#2). It's a niche audience, but not limited to Mesa3D alone. 4. Why not use AI to transpile?The project was conceived before generative AI tools were widely available. But for the same reasons clang/gcc are better than AI-to-bytecode conversion, a dedicated tool offers better utility. Specifically:
|
65b375a to
b1c12f5
Compare
|
@gurchetansingh: I need to apologize for being rather slow on reviewing this. I'll be offline for a bit, but this is on the top of my list to get back to when I return to work. |
b1c12f5 to
8904ffe
Compare
Newest updateWe are using the tool right now for Shower thoughtsI was reading: https://blogsystem5.substack.com/p/bazel-next-generation
In many ways, meson2hermetic makes the same observations. meson is the preferred build system of local Linux-based open-source projects. However, integrating open-source projects is increasingly done in the hermetic Bazel style. It's also good to think about Meson eventually handling monorepo style projects one day -- will it continue to stay in its niche or have a unique take on the hermeticization issue? 🤔 |
8904ffe to
3f1ca29
Compare
I have done some thinking about this, you may be interested in #14147, on that issue. |
3f1ca29 to
7578d53
Compare
That issue and the related issues were useful. As a result, I changed the API from A platform is in line with the Bazel concept: a collection of sysroots, OS, architecture and toolchains which define a machine. For example,
where Meson itself would understand what that means. |
7578d53 to
424140d
Compare
dcbaker
left a comment
There was a problem hiding this comment.
I made it about half way through a first read through of this in a while. I'll have more comments and it's a large enough PR that I'll probably clone it locally and have a more hands on look. I'll try to get through the rest of the first re-read tomorrow.
|
I'll wait for the complete review before uploading a new version. Though, a question about the formatting: is the Meson style to ignore all hanging parenthesis, or just ones in specific places (function definitions, conditionals/with statements)? For example, is a statement like allowed? Or should it be: Though, I'll note all formatters (ruff, black, yapf) all use hanging parenthesis somewhere, although in different places. |
Sounds good. |
240cbff to
6d6bad4
Compare
|
I've removed Most of the project-specific knowledge is gone and will be put into the revelant project. For example, here is a PR to Mesa for the TOML files: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/42932 There is a small amount of project specific knowledge in The most significant change is the introduction of |
af1a6e7 to
9217da5
Compare
These are TOML files for meson2hermetic: mesonbuild/meson#15462 This makes the Android build of rutabaga_gfx a bit more maintainable and cleaner.
670f48b to
2345b42
Compare
|
The latest version added support for That should help allay concerns that the tool is Mesa3D specific. I was thinking about doing |
2345b42 to
58acafc
Compare
|
[No major changes; just a rebase awaiting further review] |
2c64b00 to
6b92c7c
Compare
|
CI failures/hang look to Github Actions being down today: https://www.githubstatus.com/. Everything works locally. @bonzini @dcbaker ping again for re-review. There are four downstream meson-to-{X} solutions out there (example 1, example 2, example 3). Someone just pinged me about Mesa-to-GN for Chrome (yes, they have their own hermetic builds system too). Taken against this err .. um .. less than optimal status quo, this is a definite win! |
6b92c7c to
d395b9a
Compare
|
Some of the test cases are put in I agree with bonzini's comment from above:
This feels a bit like a feature that would be useful as a standalone thing. For example if people want to do Windows development and use prebuilt libs without using pkg-config or something similar. Then you could have a file that says "dependency named X exists and to use it you add these compiler flags and these linker flags". |
This will allow overrides (used by convert) of the Rust module to leverage these.
Fix some easy-to-see type checking errors.
This distills what I've learned about the Meson formatting rules into a [tool.ruff]. This is only applicable to the new meson2hermetic code right now, and others who want to use some formatter in their contributions has well. This of course is not meant to be adopted project-wide. There is a disclaimer about trusting the tool and the "Black" style without auditing it's output.
The testname is useful as a key to precomputed compiler checks dictionary: [platform.cpp.links.fails] "GNU qsort_r" = true "BSD qsort_r" = true
This API converts meson.build into their corresponding hermetic representation (Android Blueprint, Bazel and maybe Buck2 in the future). This is motivated by the need to integrate Mesa3D into AOSP and Fuchsia trees. The question "how to build and update Mesa3D drivers for Android?" [1] in particular has led to several methods over the years, none of which used Android's native build system (Soong). This has been an obstacle to adoption of open-source drivers, which everyone knows are more secure, maintainable and faster than closed alternatives. By integrating into Mesa3D's native build system (Meson), "meson convert" brings shocking and jaw-dropping clarity to the question. Technically speaking, the tool works by via series of TOML files. Python 3.11 has tomlib in the standard library, and mconvert.py uses a conditional import strategy to prevent issues on older Python versions. These TOML files specify: * the Meson project that is being converted * where to find the dependencies in a hermetic tree * which compilers that a hermetic tree supports These TOML files are used to run the Meson interpreter multiple times. For example, the set of C/C++ flags may be different if the compiler targets x86_64 or ARM64. Data from each of meson intrepreter runs is collated to reconstruct the full set of Soong/Bazel rules. Although the initial implementation is focused with converting to a existing hermetic build system, this introduces infrastructure that could be useful if Meson itself takes a look at remote-executed, hermetic builds. For example, Meson can download prebuilts from NixPkgs or grow the ability to handle other projects. Used in Android17+ mesa3d now [2]. [1] https://gitlab.freedesktop.org/mesa/mesa/-/issues/13776 [2] https://android.googlesource.com/platform/external/mesa3d/+/refs/heads/android17-release/Android.bp Co-developed-by: Craig Stout <cstout@google.com> Co-developed-by: Brandon Nguyen <bpnguyen@google.com>
For convert: - add to mesonmain.py - add Commands.md
This adds CI/CD + unittest for 'meson convert'. Idea is simple: - have meson.build files - have "golden" Soong/Bazel files - run meson convert - compare with the goldens. Test command: - python3 run_unittests.py ConvertTests Regenerate goldens when needed: >> cd ~/meson/test cases/unit/138 hermetic basic >> ~/meson/test cases/unit/138 hermetic basic $ python3 ~/meson/meson.py convert test basic_soong >> ~/meson/test cases/unit/138 hermetic basic $ python3 ~/meson/meson.py convert test basic_bazel
d395b9a to
65ff55f
Compare
Ack, fixed in the latest version. The test data is now in
Agreed and that was fixed in the July revision. The only project specific logic is in 50-line
Yes, I think it should be fairly simple if someone wants to reuse the TOML structure to define a |
This API converts meson.build into their corresponding hermetic representation (Android Blueprint, Bazel and maybe Buck2 in the future).
This is motivated by the need to integrate Mesa3D into AOSP and Fuchsia trees.
The question "how to build and update Mesa3D drivers for Android?" [1] in particular has led to several methods over the years, none of which used Android's native build system (Soong). This has been an obstacle to adoption of open-source drivers, which everyone knows are more secure, maintainable and faster than closed alternatives.
By integrating into Mesa3D's native build system (Meson), "meson convert" brings shocking and jaw-dropping clarity to the question.
Technically speaking, the tool works via a series of TOML files. Python 3.11 has tomlib in the standard library, and
mconvert.pyuses a conditional import strategy to prevent issues on older Python versions.These TOML files specify:
These TOML files are used to run the Meson interpreter multiple times. For example, the set of C/C++ flags may be different if the compiler targets x86_64 or ARM64. Data from each of of the meson interpreter runs is collated to reconstruct the full set of Soong/Bazel rules.
Although the initial implementation is focused with converting to a existing hermetic build system, this introduces infrastructure that could be useful if Meson itself takes a look at remote-executed, hermetic builds. For example, Meson can download prebuilts from NixPkgs.
[1] https://gitlab.freedesktop.org/mesa/mesa/-/issues/13776