Revisiting vendored zlib option

[stackotter]

I came across #30 while investigating the prospect of swift-nio-extras using a vendored copy of zlib. It's been 6 years and Swift is starting to get used for more than just server-side software on Linux so I figured it might be worth revisiting with this new context.

---

I maintain SwiftCrossUI which depends indirectly on a vendored copy of zlib (via libpng). This vendored copy of zlib is handy because it means that apps created with SwiftCrossUI don't have a system dependency on zlib.

I realise that when #30 was discussed, swift-nio would've only been used in server-side code where system dependencies aren't all that important. In contrast, when developing GUI applications, you often want to minimise the number of system dependencies, because every system dependency makes cross-platform deployment that much harder and opens up another way for compatibility to fail.

For those reasons I request that the possibility of depending on a vendored copy of be revisited.

The reasons against #30 weren't really elaborated upon in the original discussion; are they still known and do they still hold?

If depending on a vended copy of swift-nio-extras is a no-go for server-side code, could it be included as an opt-in option via some environment variable based solution in Package.swift (given that package traits aren't available in 5.10)?

[Lukasa]

Thanks for filing this. I definitely think now is an appropriate time to revisit this question.

I'd like to begin with some fact-finding: what platforms are causing issues with zlib dependency? As far as I'm aware, all of the formally supported Swift platform ship with zlib in essentially all cases, so the only thing I can think of as being limiting is highly constrained OCI container images. It sounds to me like you're encountering this problem in some other contexts. Can you clarify which ones are causing difficulty?

[stackotter]

I believe Windows was the main reason I ended up vending zlib, but now that you've made me think it through I remembered that this package doesn't support Windows yet.

I guess in that case this issue probably isn't super important given that GUI apps generally run in 'normal' Linux distros (with zlib?), and server apps generally run in developer-controlled environments.

I originally started writing this issue because of a duplicate symbol issue, but that turned out to be a weird flaky SwiftPM issue which occurs every time that I clean build one particular application (and is most likely unrelated to nio). I decided to file the issue anyway because I generally lean towards avoiding system dependencies. I hadn't realised that zlib is pretty safe to rely on across Linux distros (although it seems like a sensible assumption in hindsight), so most of what I said in the original issue was just opinions that apply to system dependencies in general.

[Lukasa]

I believe Windows was the main reason I ended up vending zlib, but now that you've made me think it through I remembered that this package doesn't support Windows yet.

That's true, but we intend to start rectifying that soon. So it's a very valuable callout that Windows is the platform I'm not thinking of.

Vendoring zlib is reasonable, but we'll need a strategy for mangling the symbols. Otherwise this becomes a tragedy of the commons issue where including two dependencies from different parts of the ecosystem causes trouble. Alternatively, we can aim for a "blessed" implementation. @ktoso is this something the SSWG should think about dealing with?

[stackotter]

That's true, but we intend to start rectifying that soon.

Exciting!

Vendoring zlib is reasonable, but we'll need a strategy for mangling the symbols. Otherwise this becomes a tragedy of the commons issue where including two dependencies from different parts of the ecosystem causes trouble. Alternatively, we can aim for a "blessed" implementation. @ktoso is this something the SSWG should think about dealing with?

Either way I do believe that having a 'blessed' implementation would be ideal in the long run (and wouldn't be incompatible with figuring out symbol mangling).

When I was working on the swift-image-formats library (which leans on libpng and libwebp) I found many half-baked and abandoned libpng and libwebp wrapper libraries. None of them were fit to be consumed by swift-image-formats. All I needed was a set of somewhat-maintained unopinionated C library wrappers.

That's when @furbytm created the the-swift-collective GitHub organisation (which I also have maintainer permissions on). The organisation was created with the express intent of vending C library wrappers without any Swift bindings. The idea being that by not having an opinion on Swift bindings, other bindings can build on top of these central wrappers and avoid C symbol clashes when someone ends up depending on two separate bindings to the same C library.

Now that I look back at the-swift-collective's wrappers, there are a few Swift targets with a couple of random symbols, but we could easily remove those and make the libraries c-only.

the-swift-collective's zlib wrapper could provide a good starting point for a 'blessed' implementation.

tl;dr having blessed central C wrappers is useful for a wide variety of C libraries and the-swift-collective models what that could look like (including for zlib).

---

Looking forward past zlib, it'd be nice to have similar 'blessed' wrappers for all of the core 'C ecosystem' libraries such as libpng, libwebp, imgui, libjpeg-turbo, sdl. These wrappers would build the C libraries from source because if you just want to link to the system libraries you can use systemLibrary easily enough and without symbol clashes.

For convenience, these libraries could check environment variables such as LIBWEBP_USE_SYSTEM in their Package.swift files and use them to switch between a from-source target and a system library target so that people can skip building from source if they know they already have the dependency installed on their system (handy when developing).