Fix includes of thirdparty libs which can be unbundled on Linux by akien-mga · Pull Request #73441 · godotengine/godot

akien-mga · 2023-02-16T14:55:43Z

Changes builtin_icu and builtin_recast to match the folder names in thirdparty.

Testing on Mageia together with #73359 for an upcoming update to my official godot RPM packages for Mageia and Fedora, with the following script:

to_unbundle="embree freetype glslang graphite harfbuzz icu4c libogg libpng libtheora libvorbis libwebp mbedtls miniupnpc pcre2 recastnavigation squish wslay zlib zstd"

# Disable dlopen wrappers for Linux deps, we link them dynamically.
system_libs="use_sowrap=no "
rm -rf thirdparty/linuxbsd_headers

for lib in $to_unbundle; do
    system_libs+="builtin_"$lib"=no "
    rm -rf thirdparty/$lib
done

# Internal dep for freetype.
rm -rf thirdparty/brotli

%define _scons scons "CCFLAGS=%{?build_cflags}" "LINKFLAGS=%{?build_ldflags}" arch=%{godot_arch} $system_libs use_lto=yes use_static_cpp=no debug_symbols=yes progress=no

# Build graphical editor.
%_scons p=linuxbsd target=editor

# Build game runner.
%_scons p=linuxbsd target=template_release

~~Not 100% done compiling to make sure everything works.~~

Edit: Built a Godot 4 package for Mageia successfully.

$ ldd /usr/bin/godot
        linux-vdso.so.1 (0x00007ffe553e6000)
        libicui18n.so.72 => /lib64/libicui18n.so.72 (0x00007fbbaf000000)
        libicuuc.so.72 => /lib64/libicuuc.so.72 (0x00007fbbaee0a000)
        libharfbuzz-icu.so.0 => /lib64/libharfbuzz-icu.so.0 (0x00007fbbaf4a0000)
        libharfbuzz.so.0 => /lib64/libharfbuzz.so.0 (0x00007fbbaf389000)
        libpng16.so.16 => /lib64/libpng16.so.16 (0x00007fbbaf32c000)
        libsquish.so.0 => /lib64/libsquish.so.0 (0x00007fbbaf31e000)
        libzstd.so.1 => /lib64/libzstd.so.1 (0x00007fbbaed3a000)
        libtheoradec.so.1 => /lib64/libtheoradec.so.1 (0x00007fbbaed21000)
        libvorbis.so.0 => /lib64/libvorbis.so.0 (0x00007fbbaecf3000)
        libogg.so.0 => /lib64/libogg.so.0 (0x00007fbbaece8000)
        libwebp.so.7 => /lib64/libwebp.so.7 (0x00007fbbaec7a000)
        libmbedtls.so.14 => /lib64/libmbedtls.so.14 (0x00007fbbaec4a000)
        libmbedcrypto.so.7 => /lib64/libmbedcrypto.so.7 (0x00007fbbaebd2000)
        libmbedx509.so.1 => /lib64/libmbedx509.so.1 (0x00007fbbaebb0000)
        libwslay.so.1 => /lib64/libwslay.so.1 (0x00007fbbaf312000)
        libminiupnpc.so.17 => /lib64/libminiupnpc.so.17 (0x00007fbbaeba0000)
        libpcre2-32.so.0 => /lib64/libpcre2-32.so.0 (0x00007fbbaeb18000)
        libRecast.so.1 => /lib64/libRecast.so.1 (0x00007fbbaeaf9000)
        libembree3.so.3 => /lib64/libembree3.so.3 (0x00007fbbac600000)
        libfontconfig.so.1 => /lib64/libfontconfig.so.1 (0x00007fbbaeaab000)
        libfreetype.so.6 => /lib64/libfreetype.so.6 (0x00007fbbae9f1000)
        libasound.so.2 => /lib64/libasound.so.2 (0x00007fbbac4fc000)
        libpulse.so.0 => /lib64/libpulse.so.0 (0x00007fbbae99e000)
        libdbus-1.so.3 => /lib64/libdbus-1.so.3 (0x00007fbbae94a000)
        libspeechd.so.2 => /lib64/libspeechd.so.2 (0x00007fbbac4ef000)
        libxkbcommon.so.0 => /lib64/libxkbcommon.so.0 (0x00007fbbac4a9000)
        libudev.so.1 => /lib64/libudev.so.1 (0x00007fbbac47c000)
        libz.so.1 => /lib64/libz.so.1 (0x00007fbbac45e000)
        libXcursor.so.1 => /lib64/libXcursor.so.1 (0x00007fbbac452000)
        libXinerama.so.1 => /lib64/libXinerama.so.1 (0x00007fbbac44d000)
        libXext.so.6 => /lib64/libXext.so.6 (0x00007fbbac437000)
        libXrandr.so.2 => /lib64/libXrandr.so.2 (0x00007fbbac42a000)
        libXrender.so.1 => /lib64/libXrender.so.1 (0x00007fbbac41e000)
        libX11.so.6 => /lib64/libX11.so.6 (0x00007fbbac2df000)
        libXi.so.6 => /lib64/libXi.so.6 (0x00007fbbac2cc000)
        libglslang.so.11 => /lib64/libglslang.so.11 (0x00007fbbac000000)
        libSPIRV.so.11 => /lib64/libSPIRV.so.11 (0x00007fbbab800000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fbbab400000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fbbab724000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fbbac2aa000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fbbab22f000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fbbaf4da000)
        libicudata.so.72 => /lib64/libicudata.so.72 (0x00007fbbac2a7000)
        libglib-2.0.so.0 => /lib64/libglib-2.0.so.0 (0x00007fbbab0f9000)
        libgraphite2.so.3 => /lib64/libgraphite2.so.3 (0x00007fbbac282000)
        libgomp.so.1 => /lib64/libgomp.so.1 (0x00007fbbabfb8000)
        libsharpyuv.so.0 => /lib64/libsharpyuv.so.0 (0x00007fbbac27a000)
        libtbb.so.2 => /lib64/libtbb.so.2 (0x00007fbbab6de000)
        libxml2.so.2 => /lib64/libxml2.so.2 (0x00007fbbaaf95000)
        libbz2.so.1 => /lib64/libbz2.so.1 (0x00007fbbabfa5000)
        libbrotlidec.so.1 => /lib64/libbrotlidec.so.1 (0x00007fbbac26b000)
        libpulsecommon-16.1.so => /usr/lib64/pulseaudio/libpulsecommon-16.1.so (0x00007fbbab658000)
        libsystemd.so.0 => /lib64/libsystemd.so.0 (0x00007fbbaaecb000)
        libXfixes.so.3 => /lib64/libXfixes.so.3 (0x00007fbbabf9d000)
        libxcb.so.1 => /lib64/libxcb.so.1 (0x00007fbbab631000)
        libSPIRV-Tools-opt.so.0 => /lib64/libSPIRV-Tools-opt.so.0 (0x00007fbbaac00000)
        libSPIRV-Tools.so.0 => /lib64/libSPIRV-Tools.so.0 (0x00007fbbaaaa9000)
        libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007fbbaaa0e000)
        liblzma.so.5 => /lib64/liblzma.so.5 (0x00007fbbaa9d1000)
        libbrotlicommon.so.1 => /lib64/libbrotlicommon.so.1 (0x00007fbbaa9ae000)
        libsndfile.so.1 => /lib64/libsndfile.so.1 (0x00007fbbaa922000)
        libasyncns.so.0 => /lib64/libasyncns.so.0 (0x00007fbbab629000)
        libcap.so.2 => /lib64/libcap.so.2 (0x00007fbbab61d000)
        libgcrypt.so.20 => /lib64/libgcrypt.so.20 (0x00007fbbaa7de000)
        liblz4.so.1 => /lib64/liblz4.so.1 (0x00007fbbaa7b8000)
        libXau.so.6 => /lib64/libXau.so.6 (0x00007fbbaaec6000)
        libXdmcp.so.6 => /lib64/libXdmcp.so.6 (0x00007fbbaa7b0000)
        libvorbisenc.so.2 => /lib64/libvorbisenc.so.2 (0x00007fbbaa705000)
        libFLAC.so.12 => /lib64/libFLAC.so.12 (0x00007fbbaa6a1000)
        libspeex.so.1 => /lib64/libspeex.so.1 (0x00007fbbaa686000)
        libopus.so.0 => /lib64/libopus.so.0 (0x00007fbbaa627000)
        libmpg123.so.0 => /lib64/libmpg123.so.0 (0x00007fbbaa5cb000)
        libmp3lame.so.0 => /lib64/libmp3lame.so.0 (0x00007fbbaa555000)
        libgpg-error.so.0 => /lib64/libgpg-error.so.0 (0x00007fbbaa52e000)

For the record, this shaves off around 10% of the binary size, which isn't as huge as some might expect ;) (but it's not the primary reason to unbundle libraries, shared the burden on bugfixes - especially security fixes - among all packages is the goal).

$ ls -lh ~/Projects/godot/stable/4.0/Godot_v4.0-rc2_linux.x86_64 
-rwxr-xr-x 1 akien akien 110M Feb 14 15:13 /home/akien/Projects/godot/stable/4.0/Godot_v4.0-rc2_linux.x86_64*
$ ls -lh /usr/bin/godot
-rwxr-xr-x 1 root root 98M Feb 16 18:34 /usr/bin/godot*

Changes `builtin_icu` and `builtin_recast` to match the folder names in `thirdparty`.

akien-mga · 2023-02-16T14:57:41Z

+    if (not all(ft_linked_deps)) and any(ft_linked_deps):  # All or nothing.
+        print(
+            "These libraries should be either all builtin, or all system provided:\n"
+            "freetype, libpng, zlib, graphite, harfbuzz.\n"
+            "Please specify `builtin_<name>=no` for all of them, or none."
+        )
+        sys.exit()


This is a bit brittle, and I think the list is actually longer but with some of the components optional?
Namely fontconfig and msdfgen.

akien-mga · 2023-02-16T14:57:55Z

-    if not env["builtin_icu"]:
-        env.ParseConfig("pkg-config icu-uc --cflags --libs")
+    if not env["builtin_icu4c"]:
+        env.ParseConfig("pkg-config icu-i18n icu-uc --cflags --libs")


uspoof_* is in icu-i18n.

akien-mga · 2023-02-16T14:58:32Z


 #if BASISD_SUPPORT_KTX2_ZSTD
-#include "../zstd/zstd.h"
+#include <zstd.h>


I'll make a patch file for this once I confirm the changes work, and see what upstream thinks about it.

There's already an upstream PR for it: BinomialLLC/basis_universal#228

Fix includes of thirdparty libs which can be unbundled on Linux

e2fc0ac

Changes `builtin_icu` and `builtin_recast` to match the folder names in `thirdparty`.

akien-mga added bug platform:linuxbsd topic:buildsystem topic:thirdparty labels Feb 16, 2023

akien-mga added this to the 4.0 milestone Feb 16, 2023

akien-mga requested review from a team as code owners February 16, 2023 14:55

akien-mga commented Feb 16, 2023

View reviewed changes

akien-mga requested review from bruvzg and removed request for a team February 16, 2023 14:58

bruvzg approved these changes Feb 17, 2023

View reviewed changes

akien-mga merged commit cefe1e0 into godotengine:master Feb 17, 2023

akien-mga deleted the linux-unbundling-fixes branch February 17, 2023 09:31

akien-mga mentioned this pull request Feb 17, 2023

Made it easier to use the library with external zstdlib implementations. BinomialLLC/basis_universal#228

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix includes of thirdparty libs which can be unbundled on Linux#73441

Fix includes of thirdparty libs which can be unbundled on Linux#73441
akien-mga merged 1 commit intogodotengine:masterfrom
akien-mga:linux-unbundling-fixes

akien-mga commented Feb 16, 2023 •

edited

Loading

Uh oh!

akien-mga Feb 16, 2023

Uh oh!

akien-mga Feb 16, 2023

Uh oh!

akien-mga Feb 16, 2023

Uh oh!

akien-mga Feb 17, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

akien-mga commented Feb 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

akien-mga Feb 16, 2023

Choose a reason for hiding this comment

Uh oh!

akien-mga Feb 16, 2023

Choose a reason for hiding this comment

Uh oh!

akien-mga Feb 16, 2023

Choose a reason for hiding this comment

Uh oh!

akien-mga Feb 17, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

akien-mga commented Feb 16, 2023 •

edited

Loading