Track potential intermittent/flaky CI test failures (NOT confirmed bugs — code/CI/environment triage)

[hjmjohnson]

Important

These are NOT confirmed bugs. They are recurring intermittent CI failures
surfaced by a 30-day automated review, offered as a starting point for
triage. Each candidate may turn out to be a code issue (e.g. a real race),
a test issue (fragile assertion, timing assumption), or a CI/environment
issue (runner resource pressure, download flakiness, dashboard scripting).
They are listed so the failure modes can be tracked and investigated, not as
assertions of defects.

The environment for the failures is often the root cause. For example, timeouts under VALGRIND, or similar.

Summary

A 30-day review (2026-05-25 → 06-25) of ITK CI failures across GitHub Actions
(ITK.Pixi, ITK.Arm64) and CDash (Azure DevOps + Kitware nightlies;
~18,920 failed-test rows, 676 distinct test names) found that most red CI is
CI-environment/infrastructure, with a small set of tests that fail
intermittently across unrelated commits and multiple platforms. This issue
tracks both so fixes/mitigations can be prioritized.

Methodology & how to read the tables

Failures were aggregated by test name × platform × commit/day. The signal that
separates a likely-flaky test from a one-off breakage:

• Cross-commit (fails on several unrelated source revisions) and/or
• Cross-platform + multi-day (fails on Linux and Windows and macOS over
  many distinct days).

A test failing many times on a single day / single build is almost always a
transient breakage that was then fixed — those are excluded below.

A. CI-environment / infrastructure failure modes (dominate red runs)

These are not test bugs; they prevent runs from producing a meaningful result.

Mode	Observed	Likely layer
ExternalData download failures (CID/tarball fetch)	~39% of red ITK.Pixi runs	CI/network/cache
Dashboard exit 255 "warnings-as-fatal" (0 real test failures)	~74% of red ITK.Arm64 runs	CI scripting (ci_completed_successfully)
Runner setup / apt-disk / tool-setup (exit 127)	several ITK.Pixi runs	CI/runner
One Windows nightly config red for many days (Windows11-VS22x64-RelWithDebInfo-Favorite-Remotes)	~32 tests × 15 days	single machine/config

B. Tests that fail intermittently across commits/platforms (candidates only)

Candidate	Evidence	Working hypothesis (unconfirmed)
ParallelSparseFieldLevelSetRobustness.SweepRepeat / .ConcurrentMultiPipeline	132× / 102× over 18 days; Linux+Win+macOS incl. TSan	Most-substantiated. Test is a self-described deadlock reproducer; points at gang-scheduling assumptions in ParallelSparseFieldLevelSetImageFilter under pool/TBB backends. Investigation in progress.
ShapeLabelMapFixture (5 3D_* cases)	187× / 30 days. Two modes: 3 non-Direction cases Windows-MSVC-only (105×); 2 _Direction cases cross-platform (Win+macOS+Linux)	"Resulting value" baselines stored at only 4–5 decimals skewed the fixed 1e-4 window. PR #6521 explores both fixes: first the tolerance-plan change (relative tolerance, loosen-only), now baseline regeneration at full double precision + tightened tolerances. CI to confirm which holds on MSVC.
itkMeshFileReadWriteTest01	4 unrelated commits; ubuntu+windows	Mesh round-trip FP-formatting/precision — likely test tolerance
PythonLazyImportTime	115× / 18 days; Win+Linux Python	Asserts a wall-clock threshold — fragile on shared/loaded hosts (test design)
MaskedAssignFixture.SetGetPrint	276× / 18 days; concentrated in valgrind/coverage	Instrumentation slowdown/timeout — environment interaction
itkComposeBigVectorImageFilterTest	92× / 18 days; mostly coverage builds	Resource/time under coverage instrumentation

Explicitly excluded (single-day transients — NOT flakes)

ComputeImageSpectralDensityTest (06-12), GeneralizedEigenDecomposition.IdentityPencilMatchesStandard
(06-19), itkImageFileWriterStreamingTest1_2/_3 (06-14), and a 120-test mass
red on one Windows nightly (06-25) — each a single bad build/commit that was
fixed, not recurring flakiness.

Suggested next steps

• Treat Section A as infrastructure work (ExternalData fetch retry/caching;
  reconsider warnings-as-fatal on ARM) — likely the biggest reliability payoff.
• Treat Section B as per-test investigations; confirm/deny each before any
  code change. Re-running a failed job and checking whether a different test
  fails is a quick flake/real discriminator.

Happy to break any individual candidate into its own focused issue once it's
confirmed.

[hjmjohnson]

Update — ParallelSparseFieldLevelSetRobustness.* is now root-caused and has a fix in #6519.

This candidate turned out to be a real deadlock in ParallelSparseFieldLevelSetImageFilter (not a test/environment flake): the apply-update phase blocks each work unit on its boundary neighbors (SignalNeighborsAndWait/WaitForNeighbor), which requires gang scheduling that ParallelizeArray does not guarantee — so a work-pool/TBB backend can leave a neighbor queued behind a blocked unit and deadlock. The runWithDeadline watchdog then aborts, surfacing as the intermittent CI failures recorded here.

Fix in #6519 dispatches only that one phase on a dedicated thread per work unit. The race is rare (380 oversubscribed local runs produced 0 hangs), so the mechanism was confirmed with a deterministic minimal reproducer (barrier on a bounded executor deadlocks; dedicated-thread gang completes); the filter's baseline output is bit-identical with the fix.

The other candidates in this issue remain open for triage.

[blowekamp]

There are two ParallelSparseFieldLevelSetRobustness tests failing on the nightly valgrind build due to time out.

I am not sure if there is an underlying problem with the code and the time out is due to a dead lock or if it just runs too slow.

[hjmjohnson]

Correction to my earlier comment on ParallelSparseFieldLevelSetRobustness.*. A decisive follow-up experiment did not confirm a filter deadlock, so #6519 has been closed (not merged).

Summary (full detail in #6519): the deadlock mechanism is provable in the abstract (a barrier on a bounded executor deadlocks; a dedicated-thread gang doesn't), but the filter itself completed in 770 robustness runs plus forced count > P probes on 32³ and 128³ images (up to 64 work units vs 16 TBB threads) — zero deadlocks. Its neighbor sync uses counting semaphores and skips empty boundaries, so it does not require gang scheduling in practice.

Most likely the intermittent failures here are runWithDeadline(300 s) watchdog timeouts under CI load, not deadlocks. Recommended direction: treat this candidate as a test-robustness / CI-load item (widen/relax the watchdog, shrink the work-unit sweep), not a filter bug. Re-classifying it accordingly under this issue.

[hjmjohnson]

Failure-mode breakdown for the Section B candidates (last 30 days, from the CDash details field — which distinguishes timeout/abort/assertion; the status field alone records all three as Failed).

Candidate	Fails	details mode	Class
itkComposeBigVectorImageFilterTest	33	33× Completed (Timeout)	CTest timeout
ParallelSparseFieldLevelSetRobustness	131	108× Subprocess aborted (300 s watchdog), 22× Failed, 1× SEGFAULT	deadline/watchdog (timeout-natured)
ShapeLabelMapFixture	187	187× Completed (Failed)	assertion (FP/ordering)
MaskedAssignFixture	99	99× Completed (Failed)	assertion
PythonLazyImportTime	56	56× Completed (Failed)	assertion (timing-threshold)
itkMeshFileReadWriteTest01	14	14× Completed (Failed)	comparison

Timeout count: 1 strict CTest timeout (itkComposeBigVectorImageFilterTest); 2 if the 300 s watchdog self-aborts on ParallelSparseFieldLevelSetRobustness are counted as timeout-natured. The other 4 are genuine assertion/comparison failures, not timeouts.

Corrections to my earlier hypotheses above: MaskedAssignFixture and PythonLazyImportTime are assertion failures (Completed (Failed)), not instrumentation/timing timeouts as I'd guessed.

Suggested triage split:

• Timeout / load → itkComposeBigVectorImageFilterTest (raise its CTest TIMEOUT) and ParallelSparseFieldLevelSetRobustness (watchdog-under-load; widen/relax the 300 s watchdog or shrink the work-unit sweep — see BUG: Fix ParallelSparseFieldLevelSet deadlock under work-pool backends #6519 analysis: not a confirmed deadlock).
• Assertion / nondeterminism (investigate the test/code) → ShapeLabelMapFixture (highest volume, 187), MaskedAssignFixture (99), PythonLazyImportTime (56, fragile wall-clock assertion), itkMeshFileReadWriteTest01 (14).

[hjmjohnson]

Catalog of the assertion/threshold types behind the non-timeout candidates (extracted from the test sources, so triage can see what each one checks and why it flaps).

Candidate	Threshold type	Value(s)	Why it can flap
ShapeLabelMapFixture	Absolute FP tolerance on computed geometry (EXPECT_NEAR / ITK_EXPECT_VECTOR_NEAR)	1e-4 on ~25 "resulting value" asserts: Perimeter, Centroid, EquivalentSphericalPerimeter/Radius, EquivalentEllipsoidDiameter, FeretDiameter, Flatness, Elongation, OrientedBoundingBoxOrigin (file also uses 1e-10 ×15, 1e-99/exact ×11)	Perimeter/moment/OBB estimation differs across platform & compiler in the 4th–5th significant digit; 1e-4 absolute on values like Perimeter≈28.39 is ~3.5e-6 relative — tight. Top-volume candidate (187).
PythonLazyImportTime	Wall-clock ceiling	MAX_CUMULATIVE_IMPORT_MS = 5000.0 (fails if import itk cumulative time > 5 s)	Slow/loaded CI runner pushes import past 5 s. Load-sensitive timing assertion.
PythonLazyImportTime	Module-count ceiling (not timing)	MAX_SYS_MODULES_AFTER_IMPORT_ITK = 750	Eager-loading regression — deterministic, not load-related.
itkComposeBigVectorImageFilterTest	CTest timeout + memory lock	no explicit TIMEOUT (CTest default) + RESOURCE_LOCK MEMORY_SIZE; test builds 100-channel VectorImage<unsigned char,3> of size³	Memory-bound; under contention it exceeds the default timeout → Completed (Timeout) (this is the 1 true CTest-timeout candidate).
itkMeshFileReadWriteTest01	Round-trip value comparison (no explicit numeric tolerance in source)	exact / format-sensitive	VTK ASCII FP formatting/precision on write→read round-trip.
MaskedAssignFixture	(remote-module test — not in the core ITK tree on this checkout; from the *-Favorite-Remotes nightly)	n/a here	Threshold not extractable from core tree.

Threshold-type summary

• Absolute FP tolerance too tight for cross-platform geometry → ShapeLabelMapFixture (1e-4, ~25 asserts). Most impactful fix: switch the "resulting value" asserts to a relative tolerance, or loosen to ~1e-3.
• Wall-clock ceiling → PythonLazyImportTime (5000 ms) — already commented as "intentionally loose… tightened over time"; raise or make non-blocking on loaded runners.
• CTest default timeout under memory pressure → itkComposeBigVectorImageFilterTest — set an explicit generous TIMEOUT.
• Exact round-trip comparison → itkMeshFileReadWriteTest01 — verify the writer/reader precision; may need a tolerance.

[hjmjohnson]

Draft fix for the top assertion candidate (ShapeLabelMapFixture, 187 fails) in #6521: converts the "resulting value" computed-geometry asserts from 1e-4 absolute to a magnitude-relative tolerance (max(1e-4, 1e-3·|expected|)). Loosen-only; verified/exact asserts unchanged.

[blowekamp]

Correction to my earlier comment on ParallelSparseFieldLevelSetRobustness.*. A decisive follow-up experiment did not confirm a filter deadlock, so #6519 has been closed (not merged).

Summary (full detail in #6519): the deadlock mechanism is provable in the abstract (a barrier on a bounded executor deadlocks; a dedicated-thread gang doesn't), but the filter itself completed in 770 robustness runs plus forced count > P probes on 32³ and 128³ images (up to 64 work units vs 16 TBB threads) — zero deadlocks. Its neighbor sync uses counting semaphores and skips empty boundaries, so it does not require gang scheduling in practice.

Most likely the intermittent failures here are runWithDeadline(300 s) watchdog timeouts under CI load, not deadlocks. Recommended direction: treat this candidate as a test-robustness / CI-load item (widen/relax the watchdog, shrink the work-unit sweep), not a filter bug. Re-classifying it accordingly under this issue.

@hjmjohnson This is partially in accurate. As I posted earlier the tests are failing in the valgrind builds. Running them locally, they do not time out locally. They do produce valgrid defects. There are real issues with the test and code.