common/fab: no arm64 builds

[a3s7p]

Current images cannot run on widespread arm64-based cloud instances or bare metal devices like Raspberry Pi.

This should be fixed by updating common as there are already arm64 versions of Debian packages we can use.

Running the installer on arm64 is blocked by not using upstream d-i since di-live is x86(-64)-only.

However, the installer is not needed for cloud deployment or Raspberry Pi(-like) installs so this should not be a big deal in practice.

Related to & consolidates - and closes:

• Optimized builds for ARM architecture #37
• [TKLDev] Include QEMU so as to allow cross compiling of alternative architechures #265
• Port TurnKey Linux to RaspberryPi (and potentially other ARM based arch/hardware) #613

---

Update by @JedMeister:

Thanks to @heneault for the original work - sorry it's taken such a ridiculously long time to circle back to this... And thanks to @a3s7p too, for the more recent work. As noted in the conversation below, a number of existing PRs have been rebased, updated &/or refactored and some (finally) merged to provide arm64 support.

However, after some testing, consideration & review, there are a number of tweaks/adjustments/modifications/etc required before we can finalize this. The relevant repos that require updates are noted below and PRs have been opened for them. Note that they are not currently ready for merge and are WIP until further notice:

General TKL repos:

• bootstrap: support building arm64 bootstrap bootstrap#10
• buildroot: Add arm64 arch buildroot#8
• common: 18.x arm64 common#314 (by @a3s7p - will merge after testing)
  • note that some arm related code had already been merged, but issues (with amd64 builds) were discovered so those changes have been reverted for now
• buildtasks: arm64 buildtasks#92 (by @a3s7p - will merge after testing)
• TKLDev: arm64 support turnkeylinux-apps/tkldev#42

TurnKey packages with specific changes for arm support

• fab: Support for arm64 fab#31
• turnkey-chroot: support arm64 chroot on amd64 host turnkey-chroot#5
• deckdebuild: Updates for arm64 deckdebuild#10
• pool: Support arm64 pool packages pool#15

TurnKey packages to rebuild (only minor changes required)

• autoversion: arm support autoversion#10
• confconsole: Change architecture to 'any' - pkg not arch specific confconsole#89
• ebsmount: Change architecture to 'any' - pkg not arch specific ebsmount#6
• turnkey-gitwrapper: Change architecture to 'any' - pkg not arch specific turnkey-gitwrapper#6
• hubclient: Change architecture to 'any' - pkg not arch specific hubclient#2
• inithooks: Change architecture to 'any' - pkg not arch specific inithooks#60
• tkldev-detective: Change architecture to 'any' - pkg not arch specific tkldev-detective#7
• tklpatch: Change architecture to 'any' - pkg not arch specific tklpatch#5

TKLBAM related - just rebuilds - will do this last

• tklbam
• webmin-tklbam
• tklbam dependencies - inc build dependencies

[JedMeister]

As discussed offline:

arm64 versions of Debian packages

most (all?) of our packages are already being built for arm64 - inc our current (clunky) tklbam and deps

Also a few years ago a user contributed code for RPi - I started merging it - with some changes but never got it finished.

Here's the common one (rebased on current 18.x today): turnkeylinux/common#312

Here's the one for fab (rebased on master ~6 mths ago): turnkeylinux/fab#27

There is also a confconsole PR, but as per the original fab one, this one includes lots of changes irrelevant to arm (typing and other bits and pieces). So it should probably be pulled apart into a few different PRs (e.g. arm specific, typing, etc) - as the fab one was - before any of it is merged...

[a3s7p]

Thanks for your notes!

[a3s7p]

@JedMeister

It is possible to build core as .img(.xz) with no installer for arm64 target with changes from these PRs:

1. added support for arm64 - specifically raspberry pi 4 common#312 - no modifications yet but see below
2. arm64 fab#28
3. accept FAB_ARCH value if set bootstrap#9

Some notes/hacks:

• The TKL bookworm deb repo doesn't actually seem to have any packages with Architecture: arm64, this looks wrong?

• Maybe the packages which only have interpreted code e.g. turnkey-version turnkey-ssl webmin-tklbam turnkey-sysinfo hubdns should have Architecture: all instead of Architecture: any set in debian/control? According to the docs:

Specifying only any indicates that the source package isn't dependent on any particular architecture and should compile fine on any one. The produced binary package(s) will be specific to whatever the current build architecture is.

Specifying only all indicates that the source package will only build architecture-independent packages.

As far as I understand, having it set to all instead of any will result in the same binary package being compatible with any arch which would be better than the current situation of two separate but identical binary packages for amd64 and arm64 from the same source package. Since those packages are not compiled, having architecture-specific versions of them built seems unnecessary.

• I guess the best way to be able to install the arm64 packages is to have a local repo with packages cleanly built from source targeting arm64 and add it to the chroot sources.list in common/conf/bootstrap_apt. Rebuilding the packages seemed tedious (or maybe I don't have the tkldev-fu to make it easy) so I mirrored the official repo and modified the existing amd64 packages to pretend to be arm64. It worked with a few extra steps but I don't really recommend this.

• The current PRs are RPi-specific and there's a lot of stuff there that cloud doesn't really need. Ideally we would clean this up to have a "base" arm64 layer (conf/overlay/etc), then either RPi or cloud addons (or other platforms?) on top. I'm happy to take care of that after we can test the current state of things works.

Prerequisites on the host TKLDev:

1. packages

To build arm64 bootstrap:

# apt -y install binfmt-support qemu-user-static

To build product.img:

# apt -y install dosfstools kpartx

2. build an arm64 bootstrap while it's not TKL-hosted:

# cd /turnkey/fab/bootstraps
# mv bookworm bookworm-amd64
# git clone git@github.com:a3s7p/bootstrap.git
# cd bootstrap
# git checkout arm64
# export FAB_ARCH=arm64
# make install

Once the bootstrap tarball is TKL-hosted like the amd64 one, there should be some trivial changes:

• tkldev-setup will need to be updated to download it
• fab will need to be tweaked to support per-arch bootstraps

3. set up the local arm64 repo or fix the official one to provide arm64 packages
   3a) if local repo, rsync or bind-mount it to bootstrap subdirectory and add it to sources.list in conf/bootstrap_apt

4. export FAB_ARCH=arm64 && cd core && make clean && make, which should then hopefully work and produce a build/product.img.xz

Thoughts?

[JedMeister]

@a3s7p - great work 😁

It is possible to build core as .img(.xz) with no installer for arm64 target with changes from these [3] PRs

I've now merged those 3 PRs.

I did a quick test, but nothing of significance, so we'll need to watch out for any regressions! Otherwise all good....

The TKL bookworm deb repo doesn't actually seem to have any packages with Architecture: arm64, this looks wrong?

Hmm, you are right. TBH, I'm not 100% sure why because I'm sure that I recall building them?! I double checked Bullseye too and nothing there either?! Perhaps I just did it locally on a RPi TKLDev? Anyway that will need to be resolved.

Maybe the packages which only have interpreted code e.g. turnkey-version turnkey-ssl webmin-tklbam turnkey-sysinfo hubdns should have Architecture: all instead of Architecture: any set in debian/control?

👍 that's the way! FWIW I adjusted the control files for those you've noted and rebuilt them locally.

I was planning to push them to the bookworm-testing repo for a little more testing (to avoid the unlikely risk of breaking someone's existing system) and make dev easier for you. But unfortunately, for some reason it didn't work and I ended up with an empty bookworm-testing repo 😢

I doubt it's related to the control file changes, but I don't think that anything else has changed since I last pushed to it? So until I have time to have a closer look and debug that, I'm not able to push forward and add them to the TKL apt repo.

I guess the best way to be able to install the arm64 packages is to have a local repo with packages cleanly built from source targeting arm64 and add it to the chroot sources.list in common/conf/bootstrap_apt. Rebuilding the packages seemed tedious (or maybe I don't have the tkldev-fu to make it easy) so I mirrored the official repo and modified the existing amd64 packages to pretend to be arm64. It worked with a few extra steps but I don't really recommend this.

Reasonable workaround. Although once set up, pool makes (re)building packages relatively trivial - albeit a bit slow.

The current PRs are RPi-specific and there's a lot of stuff there that cloud doesn't really need. Ideally we would clean this up to have a "base" arm64 layer (conf/overlay/etc), then either RPi or cloud addons (or other platforms?) on top. I'm happy to take care of that after we can test the current state of things works.

Sounds fair. Perhaps any of the RPi (or other platform) specific parts could be moved to buildtasks? bt-... script/s and/or patch/es? Then we could just have the core components in fab/etc. Like you say though, that's for tidying up once it all works...

Prerequisites on the host TKLDev [...]

Sounds great man.

Thanks for reminding me of the usage of binfmt & qemu too. I knew it was possible, but didn't recall the details. IIRC when I played with this last I used an RPi4 host. But being able to build different arch images and packages all on a single host is the go. 😁

[a3s7p]

@JedMeister thanks!

👍 that's the way! FWIW I adjusted the control files for those you've noted and rebuilt them locally.

Great. Not sure if those are all the packages we'd need to adjust as GH code search is not giving me accurate results for some reason. Will check locally in cloned repos.

I doubt it's related to the control file changes, but I don't think that anything else has changed since I last pushed to it?

Could be that something just bit-rotted away... anyway this is not blocking anything right now, just needs to be figured out by release time.

Reasonable workaround. Although once set up, pool makes (re)building packages relatively trivial - albeit a bit slow.

Thanks, will take a look.

Sounds fair. Perhaps any of the RPi (or other platform) specific parts could be moved to buildtasks? bt-... script/s and/or patch/es? Then we could just have the core components in fab/etc. Like you say though, that's for tidying up once it all works...

That's a good point. I'd actually prefer for things to be centralized ideally (especially when we use Debian live-build at some point) but given the status quo it makes sense to have the RPi stuff as a new buildtasks script and the cloud stuff as an extension of bt-ec2 (if needed).

I'll look into that.

[JedMeister]

Sounds good mate and unless I missed something I'm not (yet) blocking further work, so I'll leave it with you for now...

[a3s7p]

@JedMeister it occurs to me that the original “issue” of not having any arm64 builds is fixed now that the PRs have been tested and merged. The linked issues should be resolved too.

What do you think about closing this and having new issue(s) tracking cloud builds etc? I can take care of that.

[JedMeister]

Apologies it has taken me so long to explicitly respond @a3s7p .

TL;DR sounds great!

Are there new issues yet? (sorry I could look myself, but in the middle of things ATM). If not could you please open new ones as you see fit and once I'm sure that we have those, I'll close this one.