aarch64: unlocked serial writers can byte-interleave on SMP boots, corrupting single-shot markers

[ryanbreen]

ring_span_self_check::report() (kernel/src/tracing/providers/irq.rs, #[cfg(feature = "boot_tests")]) writes its [RING_SPAN:...] marker via the lock-free raw_serial_* writers, by design, since it runs inside trace_timer_tick and taking the standard logger lock there risks the documented interrupt-vs-logger-lock deadlock (a timer interrupt firing on the same CPU that already holds the logger lock).

That lock-freedom has an observed cost on a real multi-core (-smp 4) aarch64 boot: another CPU's own unlocked, multi-byte serial write (an early-boot [TEST:...:START] self-test marker in the cases caught so far) can be in flight on the SAME shared UART at the same instant, and the two writers' bytes interleave. Caught live during the failure-trace-capture PR-2 fix round (2026-09-05):

[TEST:filesystem:vi[rtio_RINGblk_writ_SPAN:cpu=0:spae_rn_ms=ead_verify:START]

([TEST:filesystem:virtio_blk_write_verify:START] interleaved byte-for-byte with [RING_SPAN:cpu=0:span_ms=...])

[TEST:memory:stack_dept[RING_SPAN:cpu=0:sph:an_mSTAs=RT]

([TEST:memory:stack_depth:START] interleaved with the same marker)

Measured rate: 2 of 6 ./docker/qemu/run-aarch64-boot-test-strict.sh boots on this branch's UNMODIFIED base commit (TICK_SAMPLE=20, before any fix-round edit) hit this and scored "Ring-span self-check marker missing" -- confirmed pre-existing, not introduced by the fix round. Reproduced again at 2/3 and 3/8 on other sample boots in the same round.

This corrupts the OUTPUT TEXT only -- the computed span_ms/ticks_total/tick_events values held in kernel memory before the print are correct; only their serialization to the shared UART races. Two observed failure shapes:

1. The whole [RING_SPAN:...] line is garbled beyond regex matching -> gate reports "marker missing" (a false FAIL -- safe direction, a good boot gets rejected).
2. In principle the corruption could land inside a numeric field's own digits rather than around it, which would misreport a NUMBER rather than blank the line -- not observed for RING_SPAN in the boots this round captured (0 of the corrupted samples above landed inside a digit run), but the same byte-interleaving mechanism causes it, so it is not ruled out by this round's sample either.

This is not unique to ring_span_self_check: tests/serial_line_atomicity_structure.rs's UNLOCKED_MULTI_BYTE_WRITE_ANCHORS census already tracks multiple other unlocked multi-byte writers in the tree (dump_on_panic, hold_pinned_wake_for_home, etc.) as an accepted, load-bearing trade-off for code that must not take the logger's lock. ring_span_self_check::report() is itself one of those anchors. The interleaving hazard is a consequence of THAT accepted trade-off applying to more than one writer on a real SMP boot, not a defect unique to this one call site.

Why not fixed in the PR-2 fix round that found this: a real fix needs mutual exclusion between report() and whichever other unlocked or locked serial writer is active during its window, which is either (a) a new lock-free arbitration primitive that other writers would also have to adopt (a change with a much wider blast radius than this PR's own scope -- kernel/src/tracing/providers/irq.rs plus two gate scripts), or (b) acquiring the standard logger lock, which is unsafe here for the documented same-CPU-reentrancy reason above (a timer interrupt on the same CPU that already holds the logger lock deadlocks against it). Neither is a same-day, single-file fix.

claim-lint:ok: scope comparison is this PR's own diff (irq.rs + 2 gate scripts) versus a change touching the shared serial-write path; not a counted claim.

Suggested directions (not evaluated in depth):

• A single global lock-free "UART reservation" token (CAS-based, bounded spin, IRQs stay masked on the acquiring CPU only) that each unlocked writer checks before its own multi-byte write, so writers serialize against each other without any of them blocking -- untested, a direction, not a proposed fix.
• Or: accept "marker missing" as a real, if annoying, failure mode of the affected self-checks (it fails safe -- 5 of 5 corrupted boots this round observed rejected an otherwise-passing boot; 0 of 5 accepted a bad one) and document the false-FAIL rate observed per check instead of trying to remove the race.

Filed per the "any failure you find is your problem" policy: this was newly discovered while investigating a different finding on this same gate (aarch64 strict-gate RING_SPAN flakiness), confirmed pre-existing on the branch's base commit (not caused by the fix), and is out of proportionate scope for that same-day fix round.

[ryanbreen]

3 more occurrences today, on this Mac (native HVF QEMU, -smp 4, cortex-a72),
across 60 boots of ./docker/qemu/run-aarch64-boot-test-strict.sh run for the
failure-capture PR-3 review round. Each one reds the gate on
Ring-span self-check marker missing, and in each the marker IS on the wire
with another writer's bytes through it. Serials preserved in-repo under
docs/planning/green-program/failure-capture/serials/pr3-gate/.

At the pre-fix head 254f1396 (20 boots, 18/20):

[RING_SPAN:cpu=0:span_ms=1490[TE:writSTes=:filesy550stem:virt:dropped=io_0blk_uni:ticks_total=nitialize3978d_read:S:tickTART]

and, in the other boot of that run, a splice that took the token itself:

:dropped=[virtio-blk] Ver0:ticks_total=3983:tick_events=ifi62]ed: read_sector checks BLOCK_DEVICE.is_none

At the branch head (40 boots, 19/20 twice), 1 occurrence:

[TEST:scheduler:workqueue_operational:P[ASS]RING_SPAN:cpu=0:spa

The rate is what is new here, not the mechanism: 3 of 60 on this host today
against the 0 of 40 the same gate produced on 2026-09-05. The second example
above is worth noting on its own -- the splice landed inside [RING_SPAN
itself, so a grep for the token finds nothing at all and the failure reads as
"the marker was never emitted" rather than "the marker was corrupted". Any
scorer that distinguishes those two cases has to look for the fields, not the
token.

[ryanbreen]

Merged at 2be24e3 (PR #874, ruling R188).

What moved: ring_span_self_check's print site came out of trace_timer_tick
entirely. The tick now only computes the measurement and publishes it through
seven atomic stores (six Relaxed value stores + a Release ready flag); a
new thread-context boot test, ring_span_report, waits for that publication
and prints the marker through the locked serial_println! writer instead, so
it now serializes against the other [TEST:...] lines on the shared UART
rather than racing them. report()'s six raw_serial_str calls drop out of
tests/serial_line_atomicity_structure.rs's UNLOCKED_MULTI_BYTE_WRITE_ANCHORS
census (70 -> 69 anchors). 20-boot strict-gate battery at the merged head:
0 RING_SPAN reds (20/20 well-formed markers); oracle scored RED on main's
unmodified code against the same committed serial. Details, mutations and the
oracle in docs/planning/green-program/failure-capture/847-RING-SPAN-THREAD-PRINT-2026-09-06.md.

What this does NOT touch, per this issue's own scope: the general
byte-interleaving hazard across the OTHER 69 unlocked multi-byte serial
writers the census tracks (dump_on_panic, hold_pinned_wake_for_home, etc.)
is unchanged -- this PR removes the one writer that didn't need to be in that
accepted trade-off, not the trade-off itself. Leaving this open since that's
what the issue title and body describe.