Big rewrite of kos-ports' build system

[pcercuei]

• Ports now use a recipe.mk, that more or less contain the same as the old Makefile scripts, except that all variables are prefixed with the package's name.
• This is done so that all recipe scripts can be sourced at the same time without conflicting. This is needed so that all packages can be built in the same "make" instance without nesting instances.
• As a result, kos-ports is now fully buildable in parallel, everything is handled properly.
• The old scripts/* files are mostly gone; everything is in the top-level Makefile now.
• Lots of simplifications. SVN clone dropped, GIT clone dropped, everything is downloaded from HTTPS. This uses the fact that both gitlab and github enable downloading tarballs from a git hash.
• Downloading a tarball from a git branch is no more supported. This is because it would not allow reproductible builds.
• Custom zlib / libpng / libjpeg code moved to their own packages. This allows using the upstream cmake/autotools scripts, which will install their files properly (e.g. including the cmake scripts and the pkgconfig .pc file).
• Speaking about installing, everything is installed into a top-level sysroot/ folder. This means that regular Unix software will automatically find their dependencies through pkgconfig or cmake scripts, because the headers and libs are where they expect them to be.
• Version check is gone. Assume that the users won't downgrade from a version of kos-ports to an older version. When a package is updated to a new version, the Makefile script will automatically build the new version.
• Uninstall rule dropped. This was hacky at best.
• Support for simpler packages that don't download anything
• SHA256 hash is now stored in a variable in the recipe.mk

[pcercuei]

TODO:

- libGL: upstream CMake script needs an install() rule. Worked around right now by pointing to my repo.
- libbz2: does not install a pkg-config file, which causes freetype to not detect it
- freetype: Does not pick libbz2 (but builds fine otherwise)
- libAL: upstream CMake script needs an install() rule
- libimageload: does not use the right includes for jpeg
- libopusplay: Tries to include <opusfile/opusfile.h> instead of <opus/opusfile.h>
- libparallax: Tries to include <png/png.h> instead of <png.h>
- micropython: Needs updating the package's Makefile
- mruby: Needs updating the package's Makefile. The CI does not have "rake" either
- raylib4dc: fails because of dependencies
- SDL: Needs updating the package's Makefile, missing libGL dependency
- SDL: Should probably install a libSDL.pc file
- SDL_ttf: Needs updating the package's Makefile, missing SDL dependency
- SDL_ttf: Should ideally auto-discover SDL, instead the CFLAGS/LDFLAGS are patched in the recipe

[pcercuei]

Added a CI script, which will automatically build every package we have a recipe for.

[pcercuei]

Fixed the raylib package. I just had to use the correct git hash, lol

[pcercuei]

Fixed mruby, SDL_ttf, libjimtcl and micropython. CI is all green now :)

[QuzarDC]

Tried to go through the different line items, it's a lot but there's a lot of change too. Fortunately almost all of the change is in stuff that should have always been mostly transparent to the user and would at worst involve users removing stuff on their side to rely on standards. So most is geared towards our own maintenance:

Downloading a tarball from a git branch is no more supported. This is because it would not allow reproductible builds.

Regarding this while it's true and shouldn't be the norm, it would create a maintenance overhead for any port that does not follow a simple versioning scheme where we can point to a specific version. @gyrovorbis would be putting in weekly PRs to get the git hash for sh4zam updated, for instance.

From what I can tell though, it works just fine within the infrastructure set up here to do this:

sh4zam_DOWNLOAD_SITE =     https://github.com/gyrovorbis/sh4zam/archive/
sh4zam_DOWNLOAD_FILE =     master.tar.gz

With the exception of the rebuilding not triggering on successive makes without a clean of course.

The old scripts/* files are mostly gone; everything is in the top-level Makefile now.

Is there a good reason to keep lib.mk around and not just merge its contents into the main Makefile? If anything, perhaps just pulling it out of scripts which now just has the one file in it would be fine. Similarly it seems like most of what's in config.mk is now unused and could be dropped the FETCH/UNPACK commands would be the only ones that seem relevant. Not sure why AUTOTOOLS_HOST is even configurable but I think it just predates us having that easily from the environ.

Custom zlib / libpng / libjpeg code moved to their own packages. This allows using the upstream cmake/autotools scripts, which will install their files properly (e.g. including the cmake scripts and the pkgconfig .pc file).

It makes sense to remove the patches, but will have to look over how it's dealt with a bit more. The biggest reason to keep such things available would be legacy support, but if they're in new packages then that kind of goes against the reason to keep them at all. For zlib_getlength specifically, it might be as simple as patching zlib.h to provide it as a static inline (after some touching up) and marking is as deprecated (since it's not really a reliable thing to do anyways). For the other two it might make more sense to add the functionality to libimageload or libkmg or another hosted lib to not create more new ones and especially to not have libraries whose full contents are hosted in the kos-ports repo.

Uninstall rule dropped. This was hacky at best.

I agree with dropping it, and mixed in with that it would seem that this would fully codify the 'build-all' behavior that the whole structure of the previous implementation was pushing against but has tended to be the norm. I'm fine with that as well, though it's something that might merit further work if we were to get to 2-3x more ports. I think I would also have a preference, as a long term goal, to support dependencies in such a way that these could merely be built on-demand rather than all at once.

One thing though that I would request, despite it being much more geared towards contributors to users, would be the ability to specify the rebuilding of an individual port. make rebuild xyz or something that could clear the stamps for a single port. I didn't see any kind of option for that and it seems like it would be a pain to iterate over a new port without it. Perhaps you already had a process in mind for that or came up with one while working on the fixes for the various ports?

[pcercuei]

Tried to go through the different line items, it's a lot but there's a lot of change too. Fortunately almost all of the change is in stuff that should have always been mostly transparent to the user and would at worst involve users removing stuff on their side to rely on standards. So most is geared towards our own maintenance:

Downloading a tarball from a git branch is no more supported. This is because it would not allow reproductible builds.

Regarding this while it's true and shouldn't be the norm, it would create a maintenance overhead for any port that does not follow a simple versioning scheme where we can point to a specific version. @gyrovorbis would be putting in weekly PRs to get the git hash for sh4zam updated, for instance.

From what I can tell though, it works just fine within the infrastructure set up here to do this:

sh4zam_DOWNLOAD_SITE =     https://github.com/gyrovorbis/sh4zam/archive/
sh4zam_DOWNLOAD_FILE =     master.tar.gz

I guess that would be OK for me as long as it does not track a master branch. @gyrovorbis could make a "stable" branch that is pulled here.

With the exception of the rebuilding not triggering on successive makes without a clean of course.

The old scripts/* files are mostly gone; everything is in the top-level Makefile now.

Is there a good reason to keep lib.mk around and not just merge its contents into the main Makefile? If anything, perhaps just pulling it out of scripts which now just has the one file in it would be fine. Similarly it seems like most of what's in config.mk is now unused and could be dropped the FETCH/UNPACK commands would be the only ones that seem relevant. Not sure why AUTOTOOLS_HOST is even configurable but I think it just predates us having that easily from the environ.

The lib.mk is different as it was not included by any of the other scripts. Instead it is included by the KOSMakefile.mk files for the projects that need one.

I can move it to the top folder, no problem. I will cleanup the config.mk too.

Custom zlib / libpng / libjpeg code moved to their own packages. This allows using the upstream cmake/autotools scripts, which will install their files properly (e.g. including the cmake scripts and the pkgconfig .pc file).

It makes sense to remove the patches, but will have to look over how it's dealt with a bit more. The biggest reason to keep such things available would be legacy support, but if they're in new packages then that kind of goes against the reason to keep them at all. For zlib_getlength specifically, it might be as simple as patching zlib.h to provide it as a static inline (after some touching up) and marking is as deprecated (since it's not really a reliable thing to do anyways). For the other two it might make more sense to add the functionality to libimageload or libkmg or another hosted lib to not create more new ones and especially to not have libraries whose full contents are hosted in the kos-ports repo.

I think adding the kos-png / kos-jpeg stuff to libimageload would make perfect sense.

For zlib_getlength, honestly, I'd be in favour of dropping it and see if anybody complains :)

Uninstall rule dropped. This was hacky at best.

I agree with dropping it, and mixed in with that it would seem that this would fully codify the 'build-all' behavior that the whole structure of the previous implementation was pushing against but has tended to be the norm. I'm fine with that as well, though it's something that might merit further work if we were to get to 2-3x more ports. I think I would also have a preference, as a long term goal, to support dependencies in such a way that these could merely be built on-demand rather than all at once.

They can be built on-demand already; e.g. make SDL_ttf will build zlib, libbz2, libpng, freetype, SDL, then finally SDL_ttf (and nothing else).

One thing though that I would request, despite it being much more geared towards contributors to users, would be the ability to specify the rebuilding of an individual port. make rebuild xyz or something that could clear the stamps for a single port. I didn't see any kind of option for that and it seems like it would be a pain to iterate over a new port without it. Perhaps you already had a process in mind for that or came up with one while working on the fixes for the various ports?

I can add a clean-<pkg> rule and/or a rebuild-<pkg> rule, sure.

[pcercuei]

Updated to use the ${KOS_SYSROOT} provided by KallistiOS (in my PR there), which points to ${KOS_BASE}/sysroot/${KOS_ARCH}. Later the addons will be updated to install there too.

The Makefile was updated to use per-arch build directories, so you can use one kos-ports tree for multiple KOS platforms.
I also added support for the <pkg>_ARCHS variable in the recipe, which if set, will restrict the package to be built only for the platforms listed. I did not yet use it anywhere but we have quite a lot of packages that are Dreamcast-specific.

Finally I cleaned up setup.mk, and added clean-<pkg> and rebuild-<pkg> rules.

[QuzarDC]

For zlib_getlength, honestly, I'd be in favour of dropping it and see if anybody complains :)

A quick search on github had showed me that it's in use by https://github.com/ArtemioUrbina/240pTestSuite and https://github.com/Protofall/Crayon-Game-Framework/ at least. So it's not just obscure and unused unfortunately. Otherwise I would have agreed. At the very least it should be replaced in our examples (and that could happen prior to any of this going in).

Uninstall rule dropped. This was hacky at best.

I agree with dropping it, and mixed in with that it would seem that this would fully codify the 'build-all' behavior that the whole structure of the previous implementation was pushing against but has tended to be the norm. I'm fine with that as well, though it's something that might merit further work if we were to get to 2-3x more ports. I think I would also have a preference, as a long term goal, to support dependencies in such a way that these could merely be built on-demand rather than all at once.

They can be built on-demand already; e.g. make SDL_ttf will build zlib, libbz2, libpng, freetype, SDL, then finally SDL_ttf (and nothing else).

That one I realized, I misstated it though as I intended to mean from the end-user application side. With all kos-ports clean, running make on a kos example for png building zlib and png. Though that's probably too many steps and not really expected by anyone. And it wouldn't be changes that would be required here but rather in tooling in the main kos repo's makefile.rules/prefab or such.

One further thing that would be very useful to reduce user pain in from the update. To detect if version of kos in use has the changes in KallistiOS/KallistiOS#1458 yet, and if not to suggest either updating to v2.2.x (which should get those 1458 changes into it) from v2.2.2/pulling master again OR to switch to https://github.com/KallistiOS/kos-ports/releases/tag/v2.2.2

[pcercuei]

For zlib_getlength, honestly, I'd be in favour of dropping it and see if anybody complains :)

A quick search on github had showed me that it's in use by https://github.com/ArtemioUrbina/240pTestSuite and https://github.com/Protofall/Crayon-Game-Framework/ at least. So it's not just obscure and unused unfortunately. Otherwise I would have agreed. At the very least it should be replaced in our examples (and that could happen prior to any of this going in).

Noted.

That one I realized, I misstated it though as I intended to mean from the end-user application side. With all kos-ports clean, running make on a kos example for png building zlib and png. Though that's probably too many steps and not really expected by anyone. And it wouldn't be changes that would be required here but rather in tooling in the main kos repo's makefile.rules/prefab or such.

That should be doable pretty easily, actually. Making sure that a dependency is installed is just a matter of running ${MAKE} -C ${KOS_PORTS} <dep>, so we could have a rule in the top-level Makefile.rules that if a variable e.g. DEPENDENCIES is defined, those deps are automatically built and installed.

One further thing that would be very useful to reduce user pain in from the update. To detect if version of kos in use has the changes in KallistiOS/KallistiOS#1458 yet, and if not to suggest either updating to v2.2.x (which should get those 1458 changes into it) from v2.2.2/pulling master again OR to switch to https://github.com/KallistiOS/kos-ports/releases/tag/v2.2.2

Ok, that shouldn't be too hard to do.

[pcercuei]

pulling master again OR to switch to https://github.com/KallistiOS/kos-ports/releases/tag/v2.2.2

Did you mean v2.2.3?

[QuzarDC]

pulling master again OR to switch to https://github.com/KallistiOS/kos-ports/releases/tag/v2.2.2

Did you mean v2.2.3?

The idea would be that the kos-ports v2.2.2 tag would be what you use if you're on kos v2.2.2 or below and can't update easily/yet/immediately. The ongoing kos v2.2.x stable branch can get the changes merged into it and when v2.2.3 is cut it'll be for kos-ports master.

We could also try to have v2.2.3/v2.2.x ready with the updates (I do have KallistiOS/KallistiOS#1447 open and it seems quite a bit like a sufficient amount of bugs to merit a new release, though there's probably a dozen or so more commits that could be pulled in from the past week) and have kos ports update at the same time. That would reduce the impacted users to those already using stable and trying to update kos-ports rather than anyone building from the DCwiki instructions.

[pcercuei]

Rebased on master.

[pcercuei]

Force-pushed, I had to remove fbb92d7 because it caused miniaudio to fail to include <opusfile.h>, for some reason. I was not able to reproduce the issue locally, with the stable and LRA toolchains; so I put the blame (without proof) on the CI's CMake version being older.

[pcercuei]

Rebased + wrote a recipe file for the new lftpd package.