What
While proving the #865 host-wide x86 QEMU lock on beast (container breenix-x86), every docker/qemu/run-x86-boot-tests.sh 1 / run-x86-prod-profile-boot-test.sh invocation that ran past roughly a minute of wall-clock got silently SIGKILLed -- the whole process tree (the bash script AND its qemu-system-x86_64 child) vanished from ps with no ERR-trap output, no FAIL line, no facts line, mid-write to its own serial capture. This happened 4 times across 2 independent gate scripts, at 4 different points in each boot's own sequence ([init] Breenix init starting (PID 1), and three separate times right after a [RECLAIM_DRAIN:...] line deep in the userspace test cohort) -- ruling out any single code-level marker or line as the trigger.
Evidence
sudo journalctl -k --since "<window>" on the beast HOST (not the container) shows an ACTIVE, ongoing hardware fault on physical disk sdh at the same times as every one of the 4 deaths, e.g.:
Sep 06 06:48:51 beast kernel: device offline error, dev sdh, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Sep 06 06:48:51 beast kernel: Buffer I/O error on dev sdh, logical block 0, async page read
Sep 06 06:48:51 beast kernel: ldm_validate_partition_table(): Disk read failed.
...
Sep 06 06:49:18 beast kernel: sd 4:0:3790:0: Power-on or device reset occurred
and, ~10 minutes earlier:
Sep 06 06:38:56 beast kernel: sd 4:0:3785:0: attempting task abort!scmd(0x00000000ac59a87e), outstanding for 30317 ms & timeout 30000 ms
A 5-minute journalctl window at the time of investigation carried 88 lines mentioning sdh. This is a real, ongoing physical-disk problem on the beast host, not a container- or Breenix-scoped issue -- it can affect any Incus container's workload on this host, not only breenix-x86.
Impact
Any beast-hosted gate/build whose runtime crosses a bad I/O window is at risk of being silently killed mid-run with zero diagnostic output (a hard SIGKILL bypasses every set -e/ERR-trap/EXIT-trap in the killed script, so nothing it would normally print on failure gets a chance to run). This is a false-negative risk for every x86 gate on this host, independent of any Breenix code change.
Ask
Beast host storage health needs operator attention (likely a failing/failed drive in the array backing sdh -- Power-on or device reset occurred and task abort cycles are typical pre-failure SMART-adjacent symptoms, not a transient blip). Not something a Breenix PR can fix; filing so it's tracked and the beast-hosted gate battery's next red isn't mis-attributed to code.
Not a Breenix code defect
The #865 host-wide x86 QEMU lock itself worked correctly across 2 independent concurrent-invocation trials run on this same host before this disk issue's proof was pinned down: a second invocation correctly blocked on QEMU HOST LOCK: waiting for /root/.cache/breenix/x86-qemu.lock for 90s and again for 420+s (9+ distinct 30s-interval wait messages) while a first invocation held the lock, and correctly acquired and booted once the first released it. This issue is scoped to the host's storage hardware, not the lock or gate logic.
What
While proving the #865 host-wide x86 QEMU lock on beast (container
breenix-x86), everydocker/qemu/run-x86-boot-tests.sh 1/run-x86-prod-profile-boot-test.shinvocation that ran past roughly a minute of wall-clock got silently SIGKILLed -- the whole process tree (the bash script AND its qemu-system-x86_64 child) vanished frompswith no ERR-trap output, no FAIL line, no facts line, mid-write to its own serial capture. This happened 4 times across 2 independent gate scripts, at 4 different points in each boot's own sequence ([init] Breenix init starting (PID 1), and three separate times right after a[RECLAIM_DRAIN:...]line deep in the userspace test cohort) -- ruling out any single code-level marker or line as the trigger.Evidence
sudo journalctl -k --since "<window>"on the beast HOST (not the container) shows an ACTIVE, ongoing hardware fault on physical disksdhat the same times as every one of the 4 deaths, e.g.:and, ~10 minutes earlier:
A 5-minute journalctl window at the time of investigation carried 88 lines mentioning
sdh. This is a real, ongoing physical-disk problem on the beast host, not a container- or Breenix-scoped issue -- it can affect any Incus container's workload on this host, not onlybreenix-x86.Impact
Any beast-hosted gate/build whose runtime crosses a bad I/O window is at risk of being silently killed mid-run with zero diagnostic output (a hard SIGKILL bypasses every
set -e/ERR-trap/EXIT-trap in the killed script, so nothing it would normally print on failure gets a chance to run). This is a false-negative risk for every x86 gate on this host, independent of any Breenix code change.Ask
Beast host storage health needs operator attention (likely a failing/failed drive in the array backing
sdh--Power-on or device reset occurredandtask abortcycles are typical pre-failure SMART-adjacent symptoms, not a transient blip). Not something a Breenix PR can fix; filing so it's tracked and the beast-hosted gate battery's next red isn't mis-attributed to code.Not a Breenix code defect
The #865 host-wide x86 QEMU lock itself worked correctly across 2 independent concurrent-invocation trials run on this same host before this disk issue's proof was pinned down: a second invocation correctly blocked on
QEMU HOST LOCK: waiting for /root/.cache/breenix/x86-qemu.lockfor 90s and again for 420+s (9+ distinct 30s-interval wait messages) while a first invocation held the lock, and correctly acquired and booted once the first released it. This issue is scoped to the host's storage hardware, not the lock or gate logic.