GH-46797: [C++] Support Meson on Windows #47282

WillAyd · 2025-08-07T20:19:02Z

Rationale for this change

This allows users on the Windows platform to use the Meson build system

What changes are included in this PR?

Updates to the Meson configurations and source files to accommodate Windows

Are these changes tested?

Yes

Are there any user-facing changes?

No

GitHub Issue: [C++][CI] Add Windows CI Job for Meson #46797

github-actions · 2025-08-07T20:19:29Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

WillAyd · 2025-08-07T20:20:36Z

cpp/src/arrow/compute/kernels/ree_util_internal.h

@@ -347,7 +347,7 @@ Result<std::shared_ptr<ArrayData>> PreallocateRunEndsArray(
 /// \param has_validity_buffer a validity buffer must be allocated
 /// \param length the length of the values array
 /// \param data_buffer_size the size of the data buffer for string and binary types
-ARROW_COMPUTE_EXPORT Result<std::shared_ptr<ArrayData>> PreallocateValuesArray(
+ARROW_EXPORT Result<std::shared_ptr<ArrayData>> PreallocateValuesArray(


Although this file is in the compute/kernels tree, the associated module is built as part of the arrow lib irrespective of whether compute is enabled or not. As such, I figured it made more sense to use the EXPORT semantics from the arrow lib rather than arrow_compute

Really?
It seems that it's used by libarrow_compute not libarrow:

arrow/cpp/src/arrow/CMakeLists.txt

Line 767 in 97d7cdb

compute/kernels/ree_util_internal.cc

Ah you are right - I can fix this

WillAyd · 2025-08-07T20:22:19Z

cpp/src/arrow/compute/CMakeLists.txt

@@ -52,6 +52,7 @@ if(ARROW_TESTING AND ARROW_COMPUTE)
    target_link_libraries(arrow_compute_testing
                          PUBLIC $<TARGET_OBJECTS:arrow_compute_core_testing>
                          PUBLIC ${ARROW_GTEST_GTEST_MAIN})
+    target_compile_definitions(arrow_compute_testing PRIVATE COMPILED_BY_CMAKE)


Not a huge fan of this, but there appears to be an issue with the registry on MSVC with the Meson configuration that isn't in the CMake config.

I think this has to do with the CMake config forcing a certain type of linkage, but I'm not sure yet. I am also under the impression that the Meson/non-MSVC path is preferred, so we might prefer to "fix" CMake rather than make the Meson configuration match it

Hmm. What is the difference between Meson build and CMake build? GTest version?

While there is a gtest version difference, I don't think that is the problem. The CMake configuration has had a few issues dating back to #46839 and #47049 and it currently special cases the gtest dependency linkage (the Meson configuration does not, but also seems OK with it...)

WillAyd · 2025-08-07T20:23:16Z

cpp/src/arrow/util/meson.build

@@ -224,18 +222,25 @@ utility_test_srcs = [
 ]

 if host_machine.system() == 'windows'
+    # meson does not natively support manifest files like CMake does


I don't think the Meson support for manifest files is as nice as CMake's, but I'm also not super familiar with them. Need to research this a bit more...

mesonbuild/meson#13041 may be related.

/MANIFEST:EMBED /MANIFESTINPUT:io_util_test.manifest linker options may work.
See also: https://learn.microsoft.com/en-us/cpp/build/reference/manifestinput-specify-manifest-input?view=msvc-170

kou

Could you open an issue for this?

kou · 2025-08-08T05:36:25Z

.github/workflows/cpp_extra.yml

+          key: cpp-ccache-windows-${{ env.CACHE_VERSION }}-${{ hashFiles('cpp/**') }}
+          restore-keys: cpp-ccache-windows-${{ env.CACHE_VERSION }}-


Suggested change

key: cpp-ccache-windows-${{ env.CACHE_VERSION }}-${{ hashFiles('cpp/**') }}

restore-keys: cpp-ccache-windows-${{ env.CACHE_VERSION }}-

key: cpp-ccache-windows-meson-${{ env.CACHE_VERSION }}-${{ hashFiles('cpp/**') }}

restore-keys: cpp-ccache-windows-meson-${{ env.CACHE_VERSION }}-

kou · 2025-08-08T05:37:10Z

.github/workflows/cpp_extra.yml

+        shell: cmd
+        run: |
+          call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64
+          bash -c "ARROW_USE_MESON=ON; ci/scripts/cpp_build.sh $(pwd) $(pwd)/build ${{ matrix.run-options || '' }}"


Suggested change

bash -c "ARROW_USE_MESON=ON; ci/scripts/cpp_build.sh $(pwd) $(pwd)/build ${{ matrix.run-options || '' }}"

bash -c "ci/scripts/cpp_build.sh $(pwd) $(pwd)/build"

kou · 2025-08-08T05:37:20Z

.github/workflows/cpp_extra.yml

+        run: |
+          # For ORC
+          export TZDIR=/c/msys64/usr/share/zoneinfo
+          ARROW_USE_MESON=ON ci/scripts/cpp_test.sh $(pwd) $(pwd)/build ${{ matrix.run-options || '' }}


Suggested change

ARROW_USE_MESON=ON ci/scripts/cpp_test.sh $(pwd) $(pwd)/build ${{ matrix.run-options || '' }}

ci/scripts/cpp_test.sh $(pwd) $(pwd)/build

kou · 2025-08-08T05:37:43Z

.github/workflows/cpp_extra.yml

+          # We can invalidate the current cache by updating this.
+          CACHE_VERSION: "2022-09-13"
+      - name: Build
+        shell: cmd


Suggested change

shell: cmd

shell: cmd

env:

ARROW_USE_MESON: ON

kou · 2025-08-08T05:38:29Z

.github/workflows/cpp_extra.yml

+          call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64
+          bash -c "ARROW_USE_MESON=ON; ci/scripts/cpp_build.sh $(pwd) $(pwd)/build ${{ matrix.run-options || '' }}"
+      - name: Test
+        shell: bash


Suggested change

shell: bash

shell: bash

env:

ARROW_USE_MESON: ON

kou · 2025-08-08T05:42:23Z

cpp/src/arrow/compute/kernels/ree_util_internal.h

@@ -347,7 +347,7 @@ Result<std::shared_ptr<ArrayData>> PreallocateRunEndsArray(
 /// \param has_validity_buffer a validity buffer must be allocated
 /// \param length the length of the values array
 /// \param data_buffer_size the size of the data buffer for string and binary types
-ARROW_COMPUTE_EXPORT Result<std::shared_ptr<ArrayData>> PreallocateValuesArray(
+ARROW_EXPORT Result<std::shared_ptr<ArrayData>> PreallocateValuesArray(


Really?
It seems that it's used by libarrow_compute not libarrow:

arrow/cpp/src/arrow/CMakeLists.txt

Line 767 in 97d7cdb

compute/kernels/ree_util_internal.cc

kou · 2025-08-08T05:44:08Z

cpp/src/arrow/meson.build

+if host_machine.system() != 'windows'
+  dl_dep = dependency('dl')
+else
+  dl_dep = declare_dependency()
+endif


Could you swap these branches for easy to read? (In general, no "not" is easier to read.)

Suggested change

if host_machine.system() != 'windows'

dl_dep = dependency('dl')

else

dl_dep = declare_dependency()

endif

if host_machine.system() == 'windows'

dl_dep = declare_dependency()

else

dl_dep = dependency('dl')

endif

kou · 2025-08-08T05:46:21Z

cpp/src/arrow/meson.build

 )

+arrow_compile_args = []
+if host_machine.system() == 'windows' and get_option('default_library') != 'shared'


We can always define ARROW_STATIC:

Suggested change

if host_machine.system() == 'windows' and get_option('default_library') != 'shared'

if get_option('default_library') != 'shared'

BTW, does this work with default_library=both?

I haven't gotten default_library=both to work yet, but haven't put a ton of effort into debugging it either. Happy to do so after getting shared/static support stabilized

kou · 2025-08-08T05:50:42Z

cpp/src/arrow/meson.build

@@ -484,14 +505,22 @@ if needs_compute
    arrow_compute_lib = library(
        'arrow-compute',
        sources: arrow_compute_lib_sources,
-        dependencies: arrow_dep,
+        dependencies: [arrow_dep] + int128_deps,


Hmm. Why do we need this?

If libarrow is a static library, it must have int128_deps dependency. libarrow consumers should not care about int128_deps.

This is to build the library itself, not to expose it to consumers. I believe that MSVC/Windows do not offer a native 128 bit integer, so the int128_deps grabs it from the boost multiprecision header. On other platforms, this dependency is empty

Ah, compute also uses arrow::internal::int128_t. Sorry. I thought that decimal in libarrow only uses arrow::internal::int128_t.

kou · 2025-08-08T06:07:06Z

cpp/src/arrow/util/meson.build

@@ -224,18 +222,25 @@ utility_test_srcs = [
 ]

 if host_machine.system() == 'windows'
+    # meson does not natively support manifest files like CMake does


mesonbuild/meson#13041 may be related.

/MANIFEST:EMBED /MANIFESTINPUT:io_util_test.manifest linker options may work.
See also: https://learn.microsoft.com/en-us/cpp/build/reference/manifestinput-specify-manifest-input?view=msvc-170

github-actions · 2025-08-08T13:31:27Z

⚠️ GitHub issue #46797 has been automatically assigned in GitHub to PR creator.

WillAyd · 2025-08-08T13:31:43Z

Could you open an issue for this?

Sorry about that - there is an open issue I just messed up the commit name. I've updated the title now and can squash on next push

WillAyd · 2025-08-08T15:14:15Z

.github/workflows/cpp_extra.yml

+          call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64
+          echo %PATH%
+          dir "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\bin\HostX64\x64"
+          bash -c "ci/scripts/cpp_build.sh $(pwd) $(pwd)/build ${{ matrix.run-options || '' }}"


I believe the current error has to do with the fact that calling `bash -c link ..." invokes the linker that comes bundled with bash, and not the linker that the vcvarsall.bat script is supposed to activate. You can reproduce this locally:

> link /help Microsoft (R) Incremental Linker Version 14.44.35214.0 Copyright (C) Microsoft Corporation. All rights reserved. For help on Linker, type `link /link' or `link' For help on Library Manager, type `link /lib' or `lib' For help on Dumper, type `link /dump' or `dumpbin' For help on Editor, type `link /edit' or `editbin' For help on CvtCIL, type `link /cvtcil' For help on PushThunkObj Generator, type `link /pushthunkobj' > bash -c "link /help" link: missing operand after ‘/help’ Try 'link --help' for more information. > bash -c "link --help" Usage: link FILE1 FILE2 or: link OPTION Call the link function to create a link named FILE2 to an existing FILE1. --help display this help and exit --version output version information and exit GNU coreutils online help: <https://www.gnu.org/software/coreutils/> Report any translation bugs to <https://translationproject.org/team/> Full documentation <https://www.gnu.org/software/coreutils/link> or available locally via: info '(coreutils) link invocation'

I guess the current CMake configuration doesn't care about this, although it makes sense for Meson to be suspect of this environment.

Maybe its worth creating a separate cpp_build.bat script for Windows platforms to use? What do you think @kou ?

How about trying meson setup --vsenv instead of calling vcvarsall.bat manually?

https://mesonbuild.com/Using-with-Visual-Studio.html

If no Visual Studio Command Prompt was detected, and no mingw compilers are detected either, meson will attempt to find "a" Visual Studio installation for you automatically, by asking Microsoft's "vswhere" program. If you want to ignore mingw compilers, pass the --vsenv option on the meson command line. If you need to guarantee a specific Visual Studio version, set it up manually.

WillAyd · 2025-08-15T19:20:31Z

OK looks like this is all green now. The Ubuntu Meson failure is unrelated and will be fixed by #47041

WillAyd · 2025-08-19T22:46:39Z

I don't think the glib & ruby failure is related

kou · 2025-08-20T02:39:36Z

Correct: #46690

WillAyd · 2025-08-20T20:59:58Z

@kou do you have any thoughts on the content of the PR at this stage?

kou · 2025-08-21T02:18:15Z

.github/workflows/cpp_extra.yml

+          restore-keys: cpp-ccache-windows-meson-${{ env.CACHE_VERSION }}-
+        env:
+          # We can invalidate the current cache by updating this.
+          CACHE_VERSION: "2022-09-13"


Could you update this date?

I'm not sure what date this is supposed to be so just picked today's. Hope that is OK!

kou · 2025-08-21T02:19:01Z

ci/scripts/cpp_test.sh

+    --max-lines=0 \
+    --timeout-multiplier 10 \


Can we unify --XXX=V and -XXX V style?

kou · 2025-08-21T02:20:27Z

cpp/src/arrow/compute/CMakeLists.txt

@@ -52,6 +52,7 @@ if(ARROW_TESTING AND ARROW_COMPUTE)
    target_link_libraries(arrow_compute_testing
                          PUBLIC $<TARGET_OBJECTS:arrow_compute_core_testing>
                          PUBLIC ${ARROW_GTEST_GTEST_MAIN})
+    target_compile_definitions(arrow_compute_testing PRIVATE COMPILED_BY_CMAKE)


Hmm. What is the difference between Meson build and CMake build? GTest version?

kou · 2025-08-21T02:23:07Z

cpp/src/arrow/flight/flight_test.cc

+#ifndef ARROW_ENABLE_THREADING
+  GTEST_SKIP() << "This test requires Boost.Process to run";
+#endif


How about adding arrow::testing::Process::IsAvailable() static method or something and use it here?

kou · 2025-08-21T02:27:54Z

cpp/src/arrow/flight/meson.build

@@ -44,7 +44,17 @@ install_headers(
    subdir: 'arrow/flight',
 )

-grpc_dep = dependency('grpc++')
+# We specifically need to use the grpc subproject and cannot use a system installion
+# due to https://github.com/mesonbuild/meson/issues/14905


Can we use https://mesonbuild.com/Reference-manual_returned_external_program.html#external_programfull_path as workaround?
If we can use grpc_cpp_plugin.full_path(), can we keep supporting system gRPC?

Using full_path like that is unfortunately not an option and will yield errors like:

ninja: error: 'grpc_cpp_plugin.exe', needed by 'src/arrow/flight/Flight.grpc.pb.h', missing and no known rule to make it

In any case the pkg-config name was always wrong. I've updated that here - maybe that was the problem?

Actually I think this current iteration will have the same issue of mesonbuild/meson#14905 when there is a system install of grpc. Happy to leave as is though as it won't affect CI.

Oh... I see.

WillAyd · 2025-08-27T15:27:52Z

I think the remaining issues are related to CMake and how it chooses to link the various test libraries, and what effects that has on the execution of them. I've opened the testing issue accordingly to take a look at that outside of this PR.

Current CI failures I believe are also unrelated

WillAyd requested review from assignUser, jonkeane, kou and raulcd as code owners August 7, 2025 20:19

github-actions bot added Component: C++ awaiting review Awaiting review labels Aug 7, 2025

WillAyd commented Aug 7, 2025

View reviewed changes

WillAyd added the CI: Extra Run extra CI label Aug 7, 2025

github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Aug 7, 2025

WillAyd mentioned this pull request Aug 7, 2025

[C++] 32 bit MSVC doesn't build #32111

Open

kou reviewed Aug 8, 2025

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Aug 8, 2025

WillAyd changed the title ~~Meson win support~~ GH-46797: [C++] Support Meson on Windows Aug 8, 2025

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Aug 8, 2025

WillAyd force-pushed the meson-win-support branch 2 times, most recently from c3b74be to d7c42c4 Compare August 8, 2025 14:55

WillAyd commented Aug 8, 2025

View reviewed changes

github-actions bot added awaiting changes Awaiting changes awaiting change review Awaiting change review and removed awaiting change review Awaiting change review awaiting changes Awaiting changes labels Aug 9, 2025

WillAyd force-pushed the meson-win-support branch 3 times, most recently from aca8728 to 39c6cf0 Compare August 11, 2025 19:27

WillAyd force-pushed the meson-win-support branch 7 times, most recently from 24047ce to d9d1eed Compare August 15, 2025 18:17

WillAyd force-pushed the meson-win-support branch 4 times, most recently from 59aa36c to bc7694e Compare August 15, 2025 21:54

apacheGH-46797: [C++] Support Meson on Windows

6d39114

WillAyd force-pushed the meson-win-support branch from bc7694e to 6d39114 Compare August 19, 2025 20:14

kou reviewed Aug 21, 2025

View reviewed changes

github-actions bot added awaiting changes Awaiting changes awaiting change review Awaiting change review and removed awaiting change review Awaiting change review awaiting changes Awaiting changes labels Aug 21, 2025

kou feedback

c1e3dc1

github-actions bot added awaiting changes Awaiting changes awaiting change review Awaiting change review and removed awaiting change review Awaiting change review awaiting changes Awaiting changes labels Aug 22, 2025

WillAyd mentioned this pull request Aug 27, 2025

AssertTestWithinUlp.Basics Test Fails on Windows #47442

Open

skip test on Windows, not just MSVC

0c9106f

		key: cpp-ccache-windows-${{ env.CACHE_VERSION }}-${{ hashFiles('cpp/**') }}
		restore-keys: cpp-ccache-windows-${{ env.CACHE_VERSION }}-

	bash -c "ARROW_USE_MESON=ON; ci/scripts/cpp_build.sh $(pwd) $(pwd)/build ${{ matrix.run-options \|\| '' }}"
	bash -c "ci/scripts/cpp_build.sh $(pwd) $(pwd)/build"

	ARROW_USE_MESON=ON ci/scripts/cpp_test.sh $(pwd) $(pwd)/build ${{ matrix.run-options \|\| '' }}
	ci/scripts/cpp_test.sh $(pwd) $(pwd)/build

	if host_machine.system() == 'windows' and get_option('default_library') != 'shared'
	if get_option('default_library') != 'shared'

GH-46797: [C++] Support Meson on Windows #47282

Are you sure you want to change the base?

GH-46797: [C++] Support Meson on Windows #47282

Uh oh!

Conversation

WillAyd commented Aug 7, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions bot commented Aug 7, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 8, 2025

Uh oh!

WillAyd commented Aug 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

WillAyd commented Aug 15, 2025

Uh oh!

WillAyd commented Aug 19, 2025

Uh oh!

kou commented Aug 20, 2025

Uh oh!

WillAyd commented Aug 20, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

WillAyd commented Aug 7, 2025 •

edited by github-actions bot

Loading