Oracle-bearing chains are not reproducible: two replays of IDENTICAL code graded 1 vs 3 DRC and 8 vs 14 connectivity issues on the same board

[drandyhaas]

Found while trying to A/B the #678 pour-promise weld. It is not a bug in that feature; it is a property of the replay chain that invalidates single-run A/B on any board whose chain contains plane repair or an oracle step — which is most of the interesting ones.

The measurement

Two replays of orangecrab's recorded 15-command chain (runs_set3/orangecrab/redo_commands.sh), pruned to the file-dependency chain, same input board, same flags, same commit 2e15780d, same machine, minutes apart:

arm	commit	real DRC	connectivity issues	routed nets
oc_base2	2e15780d	1	8	154
oc_fix2	2e15780d	3	14	152

The final boards differ by ~1976 of 5872 segments. Nothing about the code differed between them — I had labelled the directories "base" and "fix" and only caught it by reading redo_stress_test.py's own first line, Code under test: ... (commit <sha>), which prints the version for exactly this reason.

Why it matters

I read that pair as "the change regresses the board", gated a feature on it, and wrote the number into a commit message, CLAUDE.md and a code comment before the check caught it. A pair like this is convincing: it has a table, a direction and a plausible mechanism. Any future A/B judged on one replay pair of an oracle chain is a coin flip with a plausible story attached.

Concretely, the corpus A/B that eventually decided the #678 knob moved one board by one net. That signal is an order of magnitude smaller than this noise floor, so on a single chain replay it would have been undetectable in either direction.

Cause, almost certainly known

This repo has already diagnosed and fixed one instance of this class: route_disconnected_planes's ±1-segment wobble was kicad-cli's own run-to-run nondeterminism — its threaded fill/ratsnest jitters the reported unconnected-item anchor coordinates between identical invocations, and oracle_reconnect's decisions keyed on the raw coordinates (same-position test, Zone|Zone split, retry-cap keys, seeds) flipped with them. Fixed in c62ca21 by _snap_zone_anchor canonicalising Zone anchors to the lex-min cell of the traced fill island, plus processing links in canonical sorted order. After that fix: 6 oracle runs across different-UUID boards identical, 3 full repair chains 0 geometric diffs.

The spread measured here is far larger than ±1 segment, so either that fix does not cover this path, or a new raw-coordinate consumer has appeared since — which is precisely the "if a new wobble appears, suspect a new raw-coordinate consumer" case that finding anticipated.

Suggested first steps

1. Isolate the stage. Replay the chain with the oracle/plane-repair steps stubbed out and confirm the earlier steps are bit-reproducible (a single route.py step demonstrably is: in a separate three-arm test the control and a no-op arm produced byte-identical copper — 6097 segments and 625 vias compared field by field).
2. Rerun the oracle alone, twice, on the same file — the documented way to confirm kicad-cli report jitter — and diff the reported anchors, not just the outputs.
3. Audit for new raw-coordinate consumers of kicad-cli-reported positions added since c62ca21; canonicalise any to island/cluster identity rather than branching on raw equality.

Meanwhile

CLAUDE.md now records the rule: grade a plane/oracle chain change by a corpus A/B, never by one replay pair. That is a workaround, not a fix — corpus A/B costs ~$1 and 40 minutes, and it should not be the only way to answer a question about one board.

Refs #490 (repair under refill jitter), #659, #678.

[drandyhaas]

Root cause found, and measured: it is kicad-cli's unconnected-items report, reaching copper through the #648 source union

Step 1 – isolate the stage. orangecrab's oracle-bearing route step (redo_commands.sh line 46, step3_planes_final → step4_reconnect), two runs per arm, same commit, same input, same machine:

arm	run 1 vs run 2
KICAD_ORACLE_UNION=1 (today's default)	4242 vs 4243 segments, 7 differing
KICAD_ORACLE_UNION=0	4139 vs 4139 segments, byte-identical (615 vias both)

So the route step is deterministic apart from the union, and the union is the whole of the divergence. The default link source (exact_unconnected, pcbnew's ZONE_FILLER) is bit-reproducible: 5 separate processes on the same file gave the identical 49-link set (sha 38f014f3…). The c62ca21 finding holds; what changed is that #648 added kicad-cli back as a second opinion.

Step 2 – rerun kicad-cli alone on one unchanged file (kicad_unconnected, 5 runs): 78 items every time, 4 of 5 reports differ from the first. Not just ±1 cell — anchors 1.1 mm apart for the same gap (P1.35V (181.1,101.1)↔(183.2,102.7) vs ↔(184.3,102.7)), the reported item kind flipping pad↔track at one coordinate (160.15,104.15), and a zone|zone item at (135.01,92.02) present in some runs and absent in others.

Step 3 – the new raw-coordinate consumer is the union block in oracle_reconnect (kicad_oracle.py, _cli648). It branches on the raw coordinates three times — coincident-anchor exact equality, the 2.0 mm "already covered by an exact link" threshold (which a 1.1 mm wobble crosses), and a 3-decimal dedup key — and then hands surviving kicad-cli anchors to the weld as its endpoints. _snap_zone_anchor never sees them: it only snaps kind == 'zone', and the snap loop is guarded by not _links_from_exact, which is false once the exact source seeded the list.

Can kicad-cli be made deterministic? No setting does it (tested)

The only lever KiCad offers is the advanced-config key MaximumThreads (kicad_advanced, reachable for kicad-cli via KICAD_CONFIG_HOME); the thread pool reads nothing else. Verified it took effect (kicad-cli wrote its 10.0/*.json into the isolated config dir; one DRC went 15.7 s → 23.5 s wall for the same 15.5 s CPU) — and the report still differed in 4 of 5 runs. Per the KiCad source the residual order-dependence is not threading: the DRC provider reports edge.GetSourceNode()->Parent()/Pos() per ratsnest edge; ratsnest nodes sit in a std::multiset<…, CN_PTR_CMP> whose comparator orders by (x, y) only, so coincident anchors (a track end and a pad at one point — exactly the pad↔track flip above) fall back to insertion order, which comes from the cluster BFS over an unordered_map; equal-weight MST edges have no geometric tie-break either. That is an upstream bug (a stable secondary key in CN_PTR_CMP + a sorted cluster pass would fix it); no existing KiCad issue matches.

I also tried canonicalizing each kicad-cli link to the pair of copper components it joins (true nearest approach, board-derived) before the union decides. It helps but cannot close it, because the demand set itself varies: raw 74/75/75/74/75 links (5 distinct sets) → canonical 70/71/71/70/71 (3 distinct sets); on the step above it took 7 differing segments to 4. Patch kept aside, not landed.

Where that leaves the fix

The only complete fix is to keep kicad-cli out of the default copper path — KICAD_ORACLE_UNION defaulting to 0 (measured byte-identical; on this one board also 13 vs 14 DRC and 11 vs 12 connectivity issues in its favour). That is a routing default change and per the repo rule needs a corpus A/B before it flips; one board is not that evidence. Until then the CLAUDE.md rule stands: grade oracle-bearing chains by corpus A/B, never one replay pair — and KICAD_ORACLE_UNION=0 is the way to get a reproducible single-board replay today.

Refs #648 (the union), #490 (the deterministic exact source), c62ca21 (the zone-anchor snap this does not reach).

[pentolope]

Cross-evidence from the neighbouring channel: the exact source itself is not universally reproducible — pcbnew's ZONE_FILLER races under load on some board states, which touches step 1's premise that exact_unconnected is bit-reproducible.

We hit it through the other consumer (plane_fragility, not the union): three replays of a recorded route.py argv, inputs digested and unchanged, run 2 of 3 differing by 3 of 1275 segments — one 45° corner, one grid cell. The first divergence in the log came before any net routed: Plane fragility field: 115803 vs 115792 cells, and over 13 fragility-on replays the field count correlated 1:1 with the copper. The input was the flip: ZONE_FILLER returns one of two different polygons for one pour — 319 vs 485 vertices, areas 0.013 mm² apart. Different copper, not different vertex ordering.

Measured at 30 fills per cell, fresh process per fill, six busy cores alongside, two machines (Ubuntu 24.04, KiCad 10.0.5 distro package, 8 cores):

board state	flips
staged mid-layout snapshot	4–5 / 30 on both machines
earlier placed snapshot, same design	4–5 / 30
the finished routed board	0 / 30 — and 0 / 98 total across idle, loaded, concurrent and from-empty arms

So a 5-run sample on one board can honestly read "bit-reproducible" (ours did, repeatedly, until we probed the right board state under load) while a mid-layout snapshot of the same design flips at ~15%. Relevance here: on a fill-susceptible board the demand set itself would wobble through the exact source even at KICAD_ORACLE_UNION=0 — same conclusion as the canonicalization paragraph above reached for the cli source, arriving via the other door.

A row for the "no setting does it (tested)" table: affinity pinning does not close the filler race either. Through an os.sched_setaffinity-pinned in-process fill it still flipped 5/30, and under taskset -c 0 kicad-cli pcb drc --refill-zones --save-board 11/30 — one core serializes the threads but preemption still interleaves them. Load matters far more than parallelism: idle machines sat stable for tens of runs, which is presumably why both these bugs stay hidden.

Self-contained repro bundle (the two flipping boards, the finished-board negative control, stock-KiCad probes with no repo code in the fill path, both attractor polygons, both machines' numbers): https://github.com/pentolope/KiCadRoutingTools/releases/download/zonefill-race-evidence/kicad-zonefiller-nondeterminism.zip — also the basis of a report being filed against KiCad itself. #831's per-run geometry disclosure is what let us pin this channel quickly; the workaround on our side is KICAD_PLANE_FRAGILITY_COST=0 for the fragility consumer, which is the same shape as the conclusion here — keep the non-reproducible oracle out of the copper path until upstream is fixed.

Refs #831, #888 (the affinity measurements are written up in its body).

[pentolope]

The issue I identified with the ZONE_FILLER in my comment above exists in KiCad 10.0.5 and was fixed in KiCad 10.0.6.