0014 — GCCP data-plane facts (SPEC-75, stacked on #17) - #19
Draft
rducom wants to merge 15 commits into
Draft
Conversation
…C-CONTRACT.md) Stacked sur feat/seeded-rebuild (#17). 12 incréments, un commit/patch chacun : epochs noncés + truncated-au-resize + close consumed|preserve (cbt), incarnation/-ESTALE + registre d'outcomes CANCELED + get_bdevs enrichi + SB étendu + verify intégrée + enveloppes + epoch-auto + pause×rebuild (raid/nvmf). Conformité : mêmes scénarios que SimDataPlane (spdk-csi W1).
…es (SPEC-75 contract)
0014.1 (RPC-CONTRACT §5): bdev_cbt_epoch_open gains an optional opaque nonce
(CP-generated, <=31 chars) stored on the epoch and echoed in get_bdevs — kills
the epoch-id ABA across maintenance rounds; a generation takeover re-stamps it.
0014.2 (§5, T-D6): a base-bdev RESIZE marks every live epoch truncated (the
live bitmap does not cover the growth zone — the delta is a lie). Never
cleared: survives takeovers, dies with the epoch. Routes the CP to D14 full.
0014.3 (§4): bdev_cbt_epoch_close gains mode preserve|consumed. preserve
(default) keeps the H1 restore-unconsumed-delta discipline; consumed discards
the frozen delta under caller certification and REQUIRES a non-empty
rebuild_token (-EPERM otherwise; registry validation lands with 0014.5).
Unknown mode is -EINVAL, never a silent preserve.
0014.6 (cbt part): get_bdevs driver_specific.cbt.epochs[] = {epoch_id, nonce,
state, generation, truncated} — the EpochObservation source, no dedicated poll.
Validated: -fsyntax-only clean (vendored headers + config stub); standalone
suite 27/27; conformance mirror = SimDataPlane scenarios (spdk-csi W1).
…ome registry (CANCELED contract)
- patch 0014: bdev_raid_create requires incarnation; delete/add/remove/
seeded-rebuild accept expected_incarnation => -ESTALE on mismatch (never
best-effort). Unclaimed SB-reassembled raids refuse any expected_incarnation.
- patch 0015: process-wide rebuild outcome registry (survives the raid,
15 min lazy TTL), tokens per attempt (CP-deterministic or auto), full
rebuilds feed the same registry, bdev_raid_get_rebuild_outcomes {token?},
CANCELED contract (destroy/member-removal with rebuild in flight => no
process write after the RPC returns).
- cbt: epoch_close(mode=consumed) now validates the token against the
registry (succeeded+verified) — unusable by design until 0014.8 (verify).
Contract: spdk-csi/docs/RPC-CONTRACT.md §1/§2/§4 (frozen W0 2026-07-13).
…ation in get_bdevs - patch 0016: SB minor 0->1, per-member content_generation/view_epoch carved from reserved bytes. RecordDivergence: survivors gen++ in the SAME SB transaction as the ejection; rebuild-completion/skip_rebuild promotion adopts the survivors' max in the same transaction as the CONFIGURED flip. Extended reassembly restores both for ALL slots (a lagging generation is the durable staleness proof the cold-recovery pass reads). - patch 0017: get_bdevs base_bdevs_list[] gains state (derived precedence failed > write_only > configured > configuring > absent), since (unix seconds of last state flip — CP anti-flap input), generation facts, and the wrapping cbt's live-epoch facts (epoch_nonce/epoch_state/truncated) via the new vbdev_cbt_query_latest_epoch() cross-module query. - cbt: vbdev_cbt_query.h — minimal raid-facing query surface (no internals). Contract: spdk-csi/docs/RPC-CONTRACT.md §3/§8 (frozen W0 2026-07-13).
…IVERGENT on double mismatch) After the copy completes and BEFORE the process concludes (SB CONFIGURED flip and registry seal both gated): sample 64 uniform-stride windows across the copied extent (seed ranges for seeded rebuilds, whole raid otherwise), compare source leg (arbiter) vs target under quiesce_range. First mismatch: re-copy from the arbiter + re-verify (counts in registry bytes, logged as healed). Second mismatch under the same lock: outcome DIVERGENT (-EILSEQ) — no host write can explain it, the copy process is broken. Success: verified=true, which is exactly what bdev_cbt_epoch_close(mode=consumed) demands (§4). Scope: raid1 plain-data (no DIF); anything else completes unverified. Honest coverage (T-C12): probabilistic process-bug detector (~64 MiB sampled); the exhaustive detector is bdev_raid_verify_ranges (0014.12, §10b). Cancellation mid-verify unwinds like the copy loop (CANCELED seal, no writes after delete returns). Contract: spdk-csi/docs/RPC-CONTRACT.md §10a (frozen W0 2026-07-13).
…, verify_ranges (0014 COMPLETE) - patch 0019 (0014.10): the raid opens an auto epoch on every surviving member's cbt at ejection (vbdev_cbt_auto_epoch_open — never an implicit takeover, -EEXIST when an OPEN round already bounds it); the auto nonce is reported via get_bdevs, which is how the CP adopts the round. Without this, the unplanned-loss path degrades to D14 full rebuilds. - patch 0020 (0014.11, DEC-11): explicit audited force-resume — the fence path resumes whatever barrier stands before preempt-and-abort; the broken barrier's own tokened resume reports it, the snapshot saga replays cleanly. - patch 0021 (0014.9): process-wide envelopes — per-class caps x(nominal, maintenance), regime switch, rebuild concurrency bound. Rebuild cap frozen at process alloc; seeded beyond the bound = -EAGAIN; auto-attach admitted loudly. Default 0 = unlimited: a failed set is a CP incident. - patch 0022 (0014.12): bdev_raid_verify_ranges — exhaustive divergence detector, 1 MiB LBA-locked chunks, verify-envelope paced, reports (<=128 ranges + exact count + truncated flag) and never repairs; registry mirrors progress under the token. All 12 increments of the 0014 GCCP data-plane facts are now implemented. Contract: spdk-csi/docs/RPC-CONTRACT.md (frozen W0 2026-07-13).
When the oldest epoch is stuck OPEN (opened but never frozen/consumed) and wedges new epoch_open with -ENOSPC, emit a specific diagnostic naming the stale_backend_id and pointing at the CP node-id-vs-observer-lvol-id key mismatch (§4.5). Additive log only; no behaviour change.
…e regen The series was format-mixed (format-patch mboxes vs raw diffs), which broke git am and made the documented regen round-trip unable to reproduce the current bytes. patches.sh regen now emits one raw git diff per commit, with the commit subject (NNNN-name) as the filename — a regen of an untouched series is byte-stable. git apply (Dockerfile + check/apply) is the only consumer and is unchanged. patches.sh check: 24/24 green on a pristine pinned clone.
…ability
- 0022 (amended): bdev_raid_verify_ranges becomes bi-mode. raid1 keeps the
copy-compare walk; raid5f gets a SYNDROME mode — XOR of all k+1 members'
strips must be zero per stripe (rotating parity position irrelevant; no
raid5f internals, generic geometry only). All members readable or -EAGAIN
('repair first' — m=1 leaves nothing to check against); ranges must be
full-stripe aligned; per-stripe report, report-only; envelope pacing
charged on the k+1 strips actually read; LBA lock / outcome registry /
incarnation reused. Result gains a 'mode' field.
- 0023 (new): degraded-service counters in get_bdevs driver_specific.raid —
reconstruct_reads_absent / reconstruct_reads_error (0006 fallback) /
degraded_write_stripes (F-e violation gauge) / last_degraded_ts. Relaxed
atomics on the I/O paths; process-rebuild reconstructs deliberately not
counted (repair work, not degraded host service).
- 0024 (new): spdk_dd propagates bdev I/O completion errors — upstream
ignores 'success' in all three bdev callbacks, so dd exited 0 while every
write was rejected (found by vec-smoke); a silently failed READ is worse
(stale buffer written out as data).
- RPC-CONTRACT: new 'Erasure coding — raid5f' section (syndrome contract,
-EAGAIN semantics, verify-at-seal is EXPLICIT: 0018 completes raid5f
processes UNVERIFIED by design, INV-EC-5 seals via syndrome verify).
- Dockerfile: honest module comments (no raid6/raid10 anywhere in SPDK;
raid5f = XOR single parity, not Reed-Solomon) + image label loses the
fictional RDMA claim (built without --with-rdma, SF-11/SF-2).
- scripts/vec-smoke.sh: single-node pre-flight on file-backed AIO (2+1,
strip 64K): full-stripe readback byte-identical; 4K write rejected by
write_unit discipline; pristine syndrome 0-divergent; parity-strip
corruption detected per stripe; rebuild_ranges repairs; degraded refuses
-EAGAIN; 0023 counters visible. ALL GREEN 9/9. Oracle note: spdk_dd exit
code pre-0024 was not trustworthy — readback + bdev error message are.
Validated: patches.sh check 24/24 on pristine clone; docker build (builder
stage) green; vec-smoke ALL GREEN in the built image.
…SPEC-75G G1) raid5f only accepts full-stripe writes (EC-3) and remote initiators are MDTS-bounded (AM-5), so the bench writer must run INSIDE the target pod. Patch 0024 makes spdk_dd's exit code honest, which the bench oracles rely on.
…mitive
A clean nexus teardown never erases the legs' raid superblocks (upstream
raid_bdev_delete frees resources only), so republishing a volume on a
DIFFERENT node mints a new nexus incarnation whose bdev_raid_create is
refused on every leg ('Superblock of a different raid bdev found', one leg
racing examine into a resurrected stale raid) -> CreatePrimaryNexus loops
-32602 forever and the volume is unattachable anywhere but its last node.
Caught live on turing 2026-08-03 by QA DataIntegrity_SurvivesPodCrash
(force-killed pod rescheduled cp-3 -> cp-2; publish failed 10:49:30->10:51:01
until test teardown).
The RPC is the missing primitive the control plane composes with its own
authorization (the create request's backend list) and the D15b divergence
gate: write-open refuses claimed legs, a bdev without a valid SB (signature
+ CRC via the standard loader) is an idempotent no-op that never writes, a
valid SB gets its signature block zeroed (parse fails, examine ignores,
create accepts). Peer-audited via the 0011 hook.
…n3-pip pkgdep.sh -d died on a pip-internal TypeError (make_requirement_preparer missing 'allow_editables') on today's :43 pull; the 2026-07-16 green build's digest is pinned for both builder and debug stages. Bump deliberately.
…pip-compile pip-tools 7.6.0 (released since the 2026-07-16 green build) calls pip 26.2 internals with a signature pip no longer has (make_requirement_preparer / 'allow_editables' TypeError), killing pkgdep.sh -d. PIP_CONSTRAINT pins pip-tools==7.5.3 + setuptools==83.0.0 for every pip invocation in pkgdep, venv --upgrade-deps included. Complements the fedora:43 digest pin: base AND PyPI drift are now both deliberate.
…other way round The 7.5.3 pin proved the pair diagnosis wrong: pip-tools 7.5.3 + pip 26.2 still dies (pip 26.2 ADDED the required 'allow_editables' kwarg pip-tools does not pass). Pin the full 2026-07-16 green triple.
…wns re-admission
A live raid1 whose remote member flapped (target pod rolling) saw the examine
path re-add the returning base and start its own rebuild, racing the
controller's D7 write_only attach ('raid ... is in process' refusals) while
the member's fabric connects were failing — and the collision corrupted the
heap: spdk_tgt SIGSEGV (exit 139), caught live on turing 2026-08-03 during
TargetRoll_Under_Active_Write. Under GCCP the control plane is the ONE brain
for member re-admission (write_only + seeded rebuild, generation-fenced);
the data plane must not run a second one.
Claimed raids (incarnation set, fork 0014) now refuse the examine re-add with
a NOTICE; unclaimed raids keep the upstream behavior (boot-time reassembly is
what the superblock is for).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Contrat gelé :
spdk-csi/docs/RPC-CONTRACT.md(2026-07-13). Plan des 12 incréments :docs/0014-GCCP-FACTS.md. Un commit/patch par item ; la suite de conformité SimDataPlane (spdk-csi W1) et ce fork partagent les mêmes scénarios. Le labelbuild-imagessera posé quand les premiers incréments C seront là.