How does multi-ISA work?

[xiota]

Most response that most helps to elucidate how multi-ISA works is this.

The other issue appears to be some parallelism/race condition in cmake that I will ask about elsewhere.

---

What is multi-ISA? How is it supposed to work? Should DISABLE_ADVANCE_SIMD be renamed?

Notably, by adding a message commands before lines 93 and 97:

pcsx2/cmake/BuildParameters.cmake

Lines 92 to 99 in 72787d1

	if (DISABLE_ADVANCE_SIMD)
	add_compile_options("-msse" "-msse2" "-msse4.1" "-mfxsr")
	else()
	# Can't use march=native on Apple Silicon.
	if(NOT APPLE OR "${CMAKE_HOST_SYSTEM_PROCESSOR}" STREQUAL "x86_64")
	add_compile_options("-march=native")
	endif()
	endif()

 if (DISABLE_ADVANCE_SIMD)
 	message("setting individual flags")
 	add_compile_options("-msse" "-msse2" "-msse4.1" "-mfxsr")
 else()
 	# Can't use march=native on Apple Silicon.
 	if(NOT APPLE OR "${CMAKE_HOST_SYSTEM_PROCESSOR}" STREQUAL "x86_64")
 		message("setting march=native")
 		add_compile_options("-march=native")
 	endif()
 endif()

When DISABLE_ADVANCE_SIMD=ON, -march=native appears to be added three times.

-- Building for x86-64 (Multi-ISA).
setting march=native
setting individual flags
setting march=native
setting march=native

Where/How is cmake being told to make multiple passes? Where/How is it being used?

[stenzek]

It's how the SW renderer/IPU/etc have multiple codepaths for SSE4/AVX/AVX2, since they benefit the most from the wider vector width.

I see no reason to rename the variable, it wouldn't really provide any benefit, since it's not something users should be messing with.

The compile flags are not duplicated as far as I can tell.

[Ziemas]

The relevant cmake magic is here.

pcsx2/pcsx2/CMakeLists.txt

Lines 749 to 762 in 72787d1

	foreach(isa "sse4" "avx" "avx2")
	add_library(GS-${isa} STATIC ${pcsx2GSSourcesUnshared} ${pcsx2IPUSourcesUnshared} ${pcsx2SPU2SourcesUnshared})
	target_link_libraries(GS-${isa} PRIVATE PCSX2_FLAGS)
	target_compile_definitions(GS-${isa} PRIVATE MULTI_ISA_UNSHARED_COMPILATION=isa_${isa} MULTI_ISA_IS_FIRST=${is_first_isa} ${pcsx2_defs_${isa}})
	target_compile_options(GS-${isa} PRIVATE ${compile_options_${isa}})
	if (${CMAKE_VERSION} VERSION_GREATER_EQUAL 3.24)
	target_link_libraries(PCSX2 PRIVATE $<LINK_LIBRARY:WHOLE_ARCHIVE,GS-${isa}>)
	elseif(APPLE)
	message(FATAL_ERROR "MacOS builds with DISABLE_ADVANCE_SIMD=ON require CMake 3.24")
	else()
	target_link_libraries(PCSX2 PRIVATE GS-${isa})
	endif()
	set(is_first_isa "0")
	endforeach()

[xiota]

That explains the multiple passes. Why doesn't march=native cause problems? Either by allowing illegal instructions to be used or preventing needed instructions from being used when building on old machines.

[stenzek]

The CI builds enable multi-isa, and are not built with -march=native. Most files are -msse4.1, with avx/avx2 for the duplicated GS/IPU/etc files.

[xiota]

cmake appears to be executing add_compile_options("-march=native").

[stenzek]

It does not execute that line, nor add -march=native if multi-isa is enabled.

[TellowKrinkle]

if (DISABLE_ADVANCE_SIMD)
	message("setting individual flags")
	add_compile_options("-msse" "-msse2" "-msse4.1" "-mfxsr")
else()
	# Can't use march=native on Apple Silicon.
	if(NOT APPLE OR "${CMAKE_HOST_SYSTEM_PROCESSOR}" STREQUAL "x86_64")
		message("setting march=native")
		add_compile_options("-march=native")
	endif()
endif()

The march=native is clearly in the else of the if (DISABLE_ADVANCE_SIMD), if you're hitting it with that set to on, something is very broken.

[xiota]

Fresh checkout. Update submodules. Add message in front of specified add_compile_options in specified file. Run cmake (3.28.1) with following options:

-DCMAKE_BUILD_TYPE="Release"
-DCMAKE_INSTALL_PREFIX="/usr"
-DDISABLE_BUILD_DATE=ON
-DDISABLE_ADVANCE_SIMD=ON
-DCMAKE_INTERPROCEDURAL_OPTIMIZATION=ON
-DENABLE_TESTS=OFF
-Wno-dev

Also tried with unset AR NM CC CXX CFLAGS CXXFLAGS LDFLAGS MAKEFLAGS

Output still indicates the else block is somehow being executed:

-- Building for x86-64 (Multi-ISA).
setting march=native
setting individual flags
setting march=native
setting march=native

Without CC and CXX set, the system defaults to gcc/g++, and the following is also present:

  *************** UNSUPPORTED CONFIGURATION ***************

  You are not compiling PCSX2 with a supported compiler.

  It may not even build successfully.

  PCSX2 only supports the Clang and MSVC compilers.

  No support will be provided, continue at your own risk.

  *********************************************************

Build is successful, but much slower than with clang. With -DCMAKE_VERBOSE_MAKEFILE=ON added to cmake, looks like march=native is not used. But still strange that the corresponding message is printed.

[stenzek]

I still get the same output. Also,

1. CMAKE_INSTALL_PREFIX has no effect on PCSX2.
2. You shouldn't be overriding CC/CXX, you should use a toolchain file like our CI.

This sounds like it's for a third-party package, in which case, sorry, but you're on your own. We can only support the configurations we use on our CI.

[xiota]

You shouldn't be overriding CC/CXX

I said "also tried" unsetting CC/CXX. This was to rule out custom configuration issues.

CMAKE_INSTALL_PREFIX has no effect on PCSX2.

Will test that next.

We can only support the configurations we use on our CI.

That's fine. Would still like the behavior of the cmake scripts and multi-isa explained better. The variable name DISABLE_ADVANCE_SIMD makes it look like the intent is to disable AVX when it seems to do the opposite.

[stenzek]

I said "also tried" unsetting CC/CXX. This was to rule out custom configuration issues.

You're still not using a toolchain file, which is how we build PCSX2 on Linux. GCC, or the "default" compiler is not supported due to what appears to be a compiler bug.

That's fine. Would still like the behavior of the cmake scripts and multi-isa explained better.

Users don't mess with CMake, so there's no reason to explain it. If you want to contribute documentation for the CMake options, the relevant page is here: https://github.com/PCSX2/pcsx2/wiki/10-Building-on-Linux

The variable name DISABLE_ADVANCE_SIMD makes it look like the intent is to disable AVX when it seems to do the opposite.

It prevents -march=native being used, so it's disabling ISA higher than SSE4. Makes sense to me.

[xiota]

It prevents -march=native being used, so it's disabling ISA higher than SSE4. Makes sense to me.

Disabling ISA higher than SSE4 should block use of "multiple codepaths for SSE4/AVX/AVX2". But from this block, it appears to be intended to enable avx, via march=haswell or march=sandybridge.

pcsx2/tests/ctest/core/CMakeLists.txt

Lines 15 to 28 in 01842a3

	if(DISABLE_ADVANCE_SIMD)
	if(WIN32)
	set(compile_options_avx2 /arch:AVX2)
	set(compile_options_avx /arch:AVX)
	elseif(USE_GCC)
	# GCC can't inline into multi-isa functions if we use march and mtune, but can if we use feature flags
	set(compile_options_avx2 -msse4.1 -mavx -mavx2 -mbmi -mbmi2 -mfma)
	set(compile_options_avx -msse4.1 -mavx)
	set(compile_options_sse4 -msse4.1)
	else()
	set(compile_options_avx2 -march=haswell -mtune=haswell)
	set(compile_options_avx -march=sandybridge -mtune=sandybridge)
	set(compile_options_sse4 -msse4.1 -mtune=nehalem)
	endif()

You're still not using a toolchain file, which is how we build PCSX2 on Linux.

I generated a toolchain file with https://github.com/PCSX2/pcsx2/blob/master/.github/workflows/scripts/linux/generate-cmake-qt.sh. Output is the same. Commands in that else block appear to be run. cmake options are basically the same as what I used before, except it uses TRUE instead of ON and sets some options that are already set with env variables or default.

GCC, or the "default" compiler is not supported due to what appears to be a compiler bug.

The same behavior is seen regardless of whether clang or gcc is used. Problem seems to be caused by something directly related to cmake, not the compiler. The messages are printed by cmake.