building in a qemu vm sandbox

[mahaase]

feature request:

it would be nice, if it is possible to select e.g. aarch64 ARCH for sandbox to build "native" aarch64 on x86_64 system by using qemu-user-static.

currently we built a sandbox with that binary. the performance with qemu5 is impressive!
so we build aarch64 artifacts (on x86_64 jenkins) for aarch64 targets (on which we also use bob natively! nvidia agx)

[jkloetzke]

Just out of curiosity: where does qemu come from in your current solution? Do you use a regular distro image?

[mahaase]

qemu5 itself is installed on system.

just copying e.g. the /usr/bin/qemu-aarch64-static into the sandbox for a aarch64 system (bob sandbox).

[daxcore]

Do you use a regular distro image?

missed to answer: i used a rootfs created by debootstrap in this case for aarch64.

[mahaase]

guys... maybe someone has an idea. i used bob aarch64 sandbox for some weeks. I could build e.g. devel::make for aarch64 a.s.o.
i hadn't copyied the qemu-aarch64-static into the usr/bin/ location of the sandbox. it worked still out of the box.
(i also used build.sh shell and i got a aarch64 system, checked by uname -a)
now i have a new pc, and i don't get this up again. i always have to copy the static binary, that the sandbox can be executed. no idea how this worked before. somehow the system picked the correct binary automatically...

[jkloetzke]

Works for me too here automatically. Actually I have no idea how it works without having the static executable in the sandbox. 🤷‍♂️

[mahaase]

could you check, if u have /lib/ld-linux-aarch64.so.1 in hostsystem?

it is possible to register architecutres in binfmt.
(https://wiki.debian.org/QemuUserEmulation)

there are 2 kinds of issues:

• can't execute at all --> binfmt register
• just qemu-aarch64-static: Could not open '/lib/ld-linux-aarch64.so.1': No such file or directory --> no idea

has your executable no dependency to ld?

what's with QEMU_LD_PREFIX = /etc/qemu-binfmt/aarch64
for me there is no /etc/qemu-binfmt/aarch64. correct path would be /usr/aarch64-linux-gnu.
maybe i have to set QEMU_LD_PREFIX or to fix the path... hmm

[jkloetzke]

No, /lib/ld-linux-aarch64.so.1 is not on my host. That is supposed to be in the sandbox I guess. qemu-aarch64-static is, as the name suggests, statically linked so that it is executable in any environment.

[mahaase]

which sandbox do u use. i realized my /bin/bash in my current sandbox isn't static linked. that should be the problem. (i used my aarch64Sandbox branch)

[jkloetzke]

I don't have an aarch64-sandbox right now but I remember that it just worked. It's not required that bash is linked statically. The shared libraries and the runtime loader are taken from the sandbox.

[mahaase]

wtf. i analyzed the problem. it still doesn't work for ubuntu bionic. no idea why this ever worked.

If you are using stretch or earlier, it also needs to be accessible from inside the chroot jail. From Debian buster and later, this isn't necessary because Linux will use the qemu binary from the host system instead of from the chroot. This means that you can use the dynamically linked qemu on buster or later but not stretch or earlier because the host libraries will not be accessible from inside the chroot.
https://wiki.debian.org/QemuUserEmulation

i also tested it with a plain new ubuntu18.04. no way. in a plain new ubuntu 20.04 it works out of the box.
ubuntu 18.04 is the basesystem of all our linux machines. would be nice, if i could "upgrade" this feature, but didn't work by upgrading some packages, like binfmt-support or qemu-user-static. maybe it's a kernel thing...

UPDATE: haha. if i get it up in WSL2 20.04, than it also works automatically in 18.04, too. not really usable, but ... interessting.
so maybe it is a problem of WSL2. The best solution would be, to mount the static binary into the sandbox. this will work.

[mahaase]

UPDATE: i am still stuck at:

/bob/4541d59ec630011bbeb34d770898a6a38bf95f84/workspace/build/./prev-gcc/xg++ -B/bob/4541d59ec630011bbeb34d770898a6a38bf95f84/workspace/build/./prev-gcc/ -B/usr/aarch64-bob-linux-gnu/bin/ -nostdinc++ -B/bob/4541d59ec630011bbeb34d770898a6a38bf95f84/workspace/build/prev-aarch64-bob-linux-gnu/libstdc++-v3/src/.libs -B/bob/4541d59ec630011bbeb34d770898a6a38bf95f84/workspace/build/prev-aarch64-bob-linux-gnu/libstdc++-v3/libsupc++/.libs  -I/bob/4541d59ec630011bbeb34d770898a6a38bf95f84/workspace/build/prev-aarch64-bob-linux-gnu/libstdc++-v3/include/aarch64-bob-linux-gnu  -I/bob/4541d59ec630011bbeb34d770898a6a38bf95f84/workspace/build/prev-aarch64-bob-linux-gnu/libstdc++-v3/include  -I/bob/2d39d4f66ef24b73262ee12a66f76b50a6c87a34/workspace/libstdc++-v3/libsupc++ -L/bob/4541d59ec630011bbeb34d770898a6a38bf95f84/workspace/build/prev-aarch64-bob-linux-gnu/libstdc++-v3/src/.libs -L/bob/4541d59ec630011bbeb34d770898a6a38bf95f84/workspace/build/prev-aarch64-bob-linux-gnu/libstdc++-v3/libsupc++/.libs  -I/bob/2d39d4f66ef24b73262ee12a66f76b50a6c87a34/workspace/libcpp -I. -I/bob/2d39d4f66ef24b73262ee12a66f76b50a6c87a34/workspace/libcpp/../include -I/bob/2d39d4f66ef24b73262ee12a66f76b50a6c87a34/workspace/libcpp/include -I/bob/fd4521c02437dbbfd9f82b6ba7310f081c69599d/workspace/usr/include -I/bob/e86c2d58866161fb8ca651b5acc96cd5baa78493/workspace/usr/include -I/bob/f14ff0d2b87d0bd8bb1ef5c0ae712bb2a7ae333f/workspace/usr/include -I/bob/2db0fb385ac59e7d087a5f08d0c65a63aa323c0e/workspace/usr/include -g -O2 -gtoggle -W -Wall -Wno-narrowing -Wwrite-strings -Wmissing-format-attribute -pedantic -Wno-long-long  -fno-exceptions -fno-rtti -I/bob/2d39d4f66ef24b73262ee12a66f76b50a6c87a34/workspace/libcpp -I. -I/bob/2d39d4f66ef24b73262ee12a66f76b50a6c87a34/workspace/libcpp/../include -I/bob/2d39d4f66ef24b73262ee12a66f76b50a6c87a34/workspace/libcpp/include -I/bob/fd4521c02437dbbfd9f82b6ba7310f081c69599d/workspace/usr/include -I/bob/e86c2d58866161fb8ca651b5acc96cd5baa78493/workspace/usr/include -I/bob/f14ff0d2b87d0bd8bb1ef5c0ae712bb2a7ae333f/workspace/usr/include -I/bob/2db0fb385ac59e7d087a5f08d0c65a63aa323c0e/workspace/usr/include -fPIC -c -o mkdeps.o -MT mkdeps.o -MMD -MP -MF .deps/mkdeps.Tpo /bob/2d39d4f66ef24b73262ee12a66f76b50a6c87a34/workspace/libcpp/mkdeps.c
In file included from /bob/2d39d4f66ef24b73262ee12a66f76b50a6c87a34/workspace/libcpp/system.h:32:0,
                 from /bob/2d39d4f66ef24b73262ee12a66f76b50a6c87a34/workspace/libcpp/charset.c:21:
/bob/4541d59ec630011bbeb34d770898a6a38bf95f84/workspace/build/prev-gcc/include/stdint.h:9:26: fatal error: stdint.h: No such file or directory
compilation terminated.

i found a solution to activate the qemu stuff, like said in last post, i installed wsl2 ubuntu18 and ubuntu 20. than i activate sudo /etc/init.d/binfmt-support start in ubuntu20. the result is, that qemu also works in ubuntu18 wsl. stange, but better than nothing :)