Test/concurrent make test

.context/TASKS.md

@@ -47,9 +47,9 @@ the name-based policy work.
 - [ ] Graduate generic internal helpers to spike-sdk-go (nonce/crypto, Shamir verify, permission (de)serialize, validation, trust/spiffeid, URL builders, `GCMNonceSize`, `Id()`, canonical permission set) → ideas/research-sdk-extraction.md #source:jira.xml #added:2026-07-14
 
 ### Phase 3: Testing `#priority:medium`
-- [ ] Make `make test` concurrent again (currently serialized by env setup) → ideas/research-cli-testing.md #source:jira.xml #added:2026-07-14
+- [x] Make `make test` concurrent again → ideas/research-cli-testing.md #source:jira.xml #added:2026-07-14 #done:2026-07-17 (removed -p 1 once the data-dir isolation landed; nothing else shared state across packages — no fixed ports, no t.Parallel, env vars are per-process. Full -race suite: 29.7s serialized to 2.9s concurrent, roughly 10x; two consecutive concurrent runs clean)
 - [x] Move the sqlite state tests off the real ~/.spike/data/spike.db: make test deleted the live dev environment database mid-run (bit us three times this week). #added:2026-07-16 #done:2026-07-16 (fs.NexusDataFolder is sync.Once-memoized, so per-test t.Setenv cannot work; instead each affected package sets SPIKE_NEXUS_DATA_DIR to a per-run temp dir in TestMain before the first resolution — state/base, state/persist, and backend/sqlite/persist — verified: a full package run leaves ~/.spike untouched)
-- [ ] Add integration tests: root key cached/recovered/not-re-initialized; secret & policy CRUD; Pilot denies when Nexus uninitialized / warns when unreachable → ideas/research-cli-testing.md #source:jira.xml #added:2026-07-14
+- [ ] Add integration tests: root key cached/recovered/not-re-initialized; secret & policy CRUD; Pilot denies when Nexus uninitialized / warns when unreachable → ideas/research-cli-testing.md #source:jira.xml #added:2026-07-14 #in-progress (2026-07-17: Slice A shipped — specs/integration-tests.md; app/nexus/internal/state/integration covers the root-key lifecycle, the not-re-initialized-twice invariant, CRUD, and an in-process shard-restore round trip inside the normal suite. Remaining: Slice B, the Pilot uninitialized/unreachable behaviors, gated on the spec open questions)
 - [ ] Raise CLI command coverage to 60%+ via unit + HTTP-mock tests; fix `t.Skip()`ed tests; DI-refactor `sendShardsToKeepers` → ideas/research-cli-testing.md #source:jira.xml #added:2026-07-14
 - [x] `start.sh` should exercise recovery/restore and encryption/decryption #source:jira.xml #added:2026-07-14 #done:2026-07-16 (encryption/decryption checks live in start.sh since the policy-validation rework; recovery/restore is exercised by make drill-recovery, kept as a separate second-terminal script deliberately so the crash simulation never runs inside the normal startup path)
 - [x] Scripted live recovery/restore drill: once `make start` completes cleanly, run `spike operator recover`, kill Nexus and the Keepers, restart Nexus alone, feed the shards back via `spike operator restore` (scriptable via stdin since fix/operator-restore), and verify a pre-crash secret reads back. Rationale: the 2026-07-16 code review found no live breakage (shard-index fidelity intact end to end; guards use exact SPIFFE role matching, unaffected by the policy-name migration), so only a drill can prove the Phase 1 "recovery/restore is broken" claim stale and close both tasks. Needs the recover/restore role entries (spire-server-entry-recover-register.sh / -restore-register.sh), which make start does not register by default. #added:2026-07-16 #done:2026-07-16 (implemented as hack/bare-metal/drill/recovery-drill.sh behind make drill-recovery; the drill first exposed the Nexus boot-order deadlock, then passed end to end once it was fixed)

app/nexus/internal/state/integration/doc.go

@@ -0,0 +1,12 @@
+//    \\ SPIKE: Secure your secrets with SPIFFE. — https://spike.ist/
+//  \\\\\ Copyright 2024-present SPIKE contributors.
+// \\\\\\\ SPDX-License-Identifier: Apache-2.0
+
+// Package integration exercises the SPIKE Nexus state layer end to end
+// against the real sqlite persistence stack: the root-key lifecycle,
+// secret and policy operations, and the operator shard-restore round
+// trip (an in-process mirror of the live recovery drill under
+// hack/bare-metal/drill). The package contains no production code; it
+// exists so the seams between state, persist, and recovery stay covered
+// by the normal test suite. See specs/integration-tests.md, Slice A.
+package integration

app/nexus/internal/state/integration/lifecycle_test.go

@@ -0,0 +1,234 @@
+//    \\ SPIKE: Secure your secrets with SPIFFE. — https://spike.ist/
+//  \\\\\ Copyright 2024-present SPIKE contributors.
+// \\\\\\\ SPDX-License-Identifier: Apache-2.0
+
+package integration
+
+import (
+	"fmt"
+	"os"
+	"testing"
+
+	"github.com/spiffe/spike-sdk-go/api/entity/data"
+	"github.com/spiffe/spike-sdk-go/config/env"
+	"github.com/spiffe/spike-sdk-go/crypto"
+	"github.com/spiffe/spike-sdk-go/security/mem"
+
+	"github.com/spiffe/spike/app/nexus/internal/initialization/recovery"
+	state "github.com/spiffe/spike/app/nexus/internal/state/base"
+)
+
+// TestMain isolates the package run: the sqlite backend writes into a
+// per-run temporary directory (fs.NexusDataFolder memoizes its result,
+// so the override must precede the first resolution), and the backend
+// store type is pinned to sqlite explicitly since the lifecycle under
+// test only exists for persistent backends.
+func TestMain(m *testing.M) {
+	dir, mkErr := os.MkdirTemp("", "spike-state-integration-test-*")
+	if mkErr != nil {
+		fmt.Fprintln(os.Stderr,
+			"failed to create a temporary data directory:", mkErr)
+		os.Exit(1)
+	}
+
+	for key, value := range map[string]string{
+		env.NexusDataDir:      dir,
+		env.NexusBackendStore: "sqlite",
+	} {
+		if setErr := os.Setenv(key, value); setErr != nil {
+			_ = os.RemoveAll(dir)
+			fmt.Fprintln(os.Stderr, "failed to set "+key+":", setErr)
+			os.Exit(1)
+		}
+	}
+
+	code := m.Run()
+
+	_ = os.RemoveAll(dir)
+	os.Exit(code)
+}
+
+const (
+	secretPath = "integration/db/creds"
+	policyName = "integration-workload-can-read"
+)
+
+// TestStateLifecycle walks the state layer through its whole life:
+// initialization, secret and policy writes, a duplicate initialization
+// (which must not recompute or corrupt anything), the export of
+// operator recovery shards, a simulated root-key loss, a shard-based
+// restore, and finally proof that the pre-crash data is readable again.
+// The stages depend on each other and run in order.
+func TestStateLifecycle(t *testing.T) {
+	rootKey := &[crypto.AES256KeySize]byte{}
+	for i := range rootKey {
+		rootKey[i] = byte(i + 1)
+	}
+
+	// Stage 1: initialize and verify the root key is cached.
+	state.Initialize(rootKey)
+	if state.RootKeyZero() {
+		t.Fatal("root key is not cached after Initialize")
+		return
+	}
+
+	// Stage 2: write a secret and a policy through the real stack.
+	secretValues := map[string]string{
+		"username": "spike",
+		"password": "integration-v1",
+	}
+	if upsertErr := state.UpsertSecret(secretPath, secretValues); upsertErr != nil {
+		t.Fatalf("failed to upsert the secret: %v", upsertErr)
+		return
+	}
+
+	got, getErr := state.GetSecret(secretPath, 0)
+	if getErr != nil {
+		t.Fatalf("failed to read the secret back: %v", getErr)
+		return
+	}
+	if got["password"] != secretValues["password"] {
+		t.Fatalf("secret round trip mismatch: got %q", got["password"])
+		return
+	}
+
+	if _, policyErr := state.UpsertPolicy(data.Policy{
+		Name:            policyName,
+		SPIFFEIDPattern: `^spiffe://spike\.ist/workload/.*$`,
+		PathPattern:     `^integration/.*$`,
+		Permissions:     []data.PolicyPermission{"read"},
+	}); policyErr != nil {
+		t.Fatalf("failed to upsert the policy: %v", policyErr)
+		return
+	}
+
+	policy, policyGetErr := state.GetPolicy(policyName)
+	if policyGetErr != nil {
+		t.Fatalf("failed to read the policy back: %v", policyGetErr)
+		return
+	}
+	if policy.PathPattern != `^integration/.*$` {
+		t.Fatalf("policy round trip mismatch: got %q", policy.PathPattern)
+		return
+	}
+
+	// Stage 3: a duplicate initialization must not recompute root key
+	// material or recreate the backend. The observable invariant: the
+	// secret written before the duplicate call stays readable, which
+	// proves the backend (and the cipher derived from the original
+	// key) survived intact.
+	state.Initialize(rootKey)
+	if state.RootKeyZero() {
+		t.Fatal("root key lost after a duplicate Initialize")
+		return
+	}
+	if _, rereadErr := state.GetSecret(secretPath, 0); rereadErr != nil {
+		t.Fatalf("secret unreadable after duplicate Initialize: %v", rereadErr)
+		return
+	}
+
+	// Stage 4: export recovery shards while healthy, as the operator
+	// recover flow does, and keep only a threshold-sized subset to
+	// prove reconstruction does not need every share.
+	shardMap := recovery.NewPilotRecoveryShards()
+	threshold := env.ShamirThresholdVal()
+	if len(shardMap) < threshold {
+		t.Fatalf("expected at least %d shards, got %d",
+			threshold, len(shardMap))
+		return
+	}
+
+	shards := make([]crypto.ShamirShard, 0, threshold)
+	for idx, value := range shardMap {
+		if len(shards) == threshold {
+			break
+		}
+		shards = append(shards, crypto.ShamirShard{
+			ID:    uint64(idx),
+			Value: value,
+		})
+	}
+
+	// Stage 5: simulate the crash by zeroing the cached root key, the
+	// in-process equivalent of losing Nexus and every Keeper.
+	state.LockRootKey()
+	mem.ClearRawBytes(state.RootKeyNoLock())
+	state.UnlockRootKey()
+	if !state.RootKeyZero() {
+		t.Fatal("root key still cached after the simulated crash")
+		return
+	}
+
+	// Stage 6: restore from the shard subset. The restore path
+	// initializes the state first and only then reaches for a SPIFFE
+	// source to hydrate the Keepers; that step fails via log.FatalErr,
+	// which the panic mode converts into a recoverable panic. The panic
+	// is therefore expected here, and it fires after the part under
+	// test has completed.
+	//
+	// A malformed workload API address makes the source creation fail
+	// at validation time. With no SPIFFE_ENDPOINT_SOCKET at all,
+	// go-spiffe would instead dial the default socket with an
+	// unbounded context and hang the test forever (the missing
+	// SVID-acquisition timeout is tracked as its own task).
+	t.Setenv("SPIFFE_ENDPOINT_SOCKET", "bogus://fail-fast")
+	t.Setenv("SPIKE_STACK_TRACES_ON_LOG_FATAL", "true")
+	func() {
+		defer func() {
+			if r := recover(); r == nil {
+				t.Error("expected a panic at the SPIFFE-source boundary" +
+					" after the state restore")
+			}
+		}()
+		recovery.RestoreBackingStoreFromPilotShards(shards)
+	}()
+
+	// Stage 7: the restore must have recomputed the original root key
+	// and the pre-crash data must be readable again.
+	if state.RootKeyZero() {
+		t.Fatal("root key not restored from shards")
+		return
+	}
+
+	state.LockRootKey()
+	restoredMatches := *state.RootKeyNoLock() == *rootKey
+	state.UnlockRootKey()
+	if !restoredMatches {
+		t.Fatal("restored root key differs from the original")
+		return
+	}
+
+	restored, restoredErr := state.GetSecret(secretPath, 0)
+	if restoredErr != nil {
+		t.Fatalf("secret unreadable after the restore: %v", restoredErr)
+		return
+	}
+	if restored["password"] != secretValues["password"] {
+		t.Fatalf("secret mismatch after the restore: got %q",
+			restored["password"])
+		return
+	}
+
+	if _, policyRereadErr := state.GetPolicy(policyName); policyRereadErr != nil {
+		t.Fatalf("policy unreadable after the restore: %v", policyRereadErr)
+		return
+	}
+
+	// Stage 8: deletion and undeletion survive the restored state.
+	if delErr := state.DeleteSecret(secretPath, []int{1}); delErr != nil {
+		t.Fatalf("failed to delete the secret: %v", delErr)
+		return
+	}
+	if _, deletedErr := state.GetSecret(secretPath, 1); deletedErr == nil {
+		t.Error("expected an error reading a deleted secret version")
+	}
+
+	if undelErr := state.UndeleteSecret(secretPath, []int{1}); undelErr != nil {
+		t.Fatalf("failed to undelete the secret: %v", undelErr)
+		return
+	}
+	if _, revivedErr := state.GetSecret(secretPath, 1); revivedErr != nil {
+		t.Fatalf("secret unreadable after undelete: %v", revivedErr)
+		return
+	}
+}

makefiles/Test.mk

@@ -25,14 +25,15 @@ test/cover:
 # Usage: make test
 # Executes all tests in the project with verbose output and race detection
 # Does not generate coverage reports (use test/cover for that)
-# Flags: -v (verbose), -race (race detection), -buildvcs (include VCS info),
-# -p 1 (sequential execution to avoid race conditions)
-# NOTE: Sequential execution is temporary workaround for concurrent environment
-# variable/database access
-# FIXME: Remove -p 1 flag once issue with concurrent test isolation is resolved
+# Flags: -v (verbose), -race (race detection), -buildvcs (include VCS info)
+# Packages run concurrently (Go's default -p). This is safe because the
+# sqlite-backed test packages isolate their data directories per run via
+# TestMain (SPIKE_NEXUS_DATA_DIR points at a temporary directory), no
+# test binds a fixed network port, and environment variables are
+# per-process, so package-level parallelism cannot leak state.
 .PHONY: test
 test:
-	go test -v -race -buildvcs -p 1 ./...
+	go test -v -race -buildvcs ./...
 
 # Comprehensive code quality audit
 # Usage: make audit

specs/integration-tests.md

@@ -0,0 +1,75 @@
+# Spec: Integration Test Suite
+
+## Status
+
+Proposed (2026-07-17). Backs the TASKS.md Phase 3 item "Add integration
+tests" and draws on `ideas/research-cli-testing.md`.
+
+## Problem
+
+The behaviors that broke this month — the Nexus boot-order deadlock,
+the policy get-by-name regression, the restore flow — were all caught
+by hand or by the live drill, not by tests. The suite exercises units
+well but nothing verifies the seams: root-key lifecycle against a real
+backing store, CRUD through the real persistence stack, and the
+Pilot's behavior when Nexus is uninitialized or unreachable.
+
+## Approach: three slices, in increasing coupling order
+
+### Slice A: state-layer integration (no SPIRE, runs in `make test`)
+
+A new test-only package exercising the real state + sqlite persistence
+stack in-process, using the per-run temp-dir isolation that already
+exists (`SPIKE_NEXUS_DATA_DIR` via `TestMain`):
+
+- Root key: `Initialize` caches it; a second initialization does not
+  regenerate or re-encrypt (the "not re-initialized twice" invariant);
+  `RestoreBackingStoreFromPilotShards` recovers the same key from
+  threshold shards and the data written before the "crash" reads back
+  (an in-process mirror of the live drill).
+- Secret CRUD: put/get/delete/undelete/list through state + sqlite,
+  including versioning metadata.
+- Policy CRUD: create/get/delete/list by name through state + sqlite,
+  including the pattern-regex compilation invariants.
+
+These run as part of the normal suite; the concurrent gate keeps them
+cheap.
+
+### Slice B: live integration (`//go:build integration`, opt-in)
+
+Build-tagged tests gated on `SPIKE_INTEGRATION_TEST=1`, assuming a
+healthy `make start` environment, in the spirit of the recovery drill:
+
+- Pilot denies operations when Nexus is uninitialized.
+- Pilot warns (does not hang or panic) when Nexus is unreachable.
+- A CLI-level smoke pass: secret put/get/delete, policy create/get by
+  name, cipher round trip; asserts on stdout now that data goes there.
+
+Run manually or in a dedicated CI job:
+`SPIKE_INTEGRATION_TEST=1 go test -tags=integration ./...`.
+
+### Slice C: HTTP-mock helpers for CLI coverage (separate task)
+
+Mocking the SDK's mTLS transport feeds the "CLI coverage to 60%+"
+task, not this one; it is out of scope here and tracked separately.
+
+## Open questions (decide before Slice B)
+
+1. Should Slice B live in this repository now, or wait until there is
+   a CI runner with a SPIRE environment? A tagged suite nobody runs
+   rots quietly, which is how the original "CI integration test is
+   broken" task was born.
+2. Does Slice B subsume the recovery drill, or stay complementary?
+   Recommendation: complementary. The drill kills real processes; the
+   tagged tests only observe a healthy environment.
+
+## Acceptance Criteria
+
+- [ ] Slice A package passes in the normal `make test` run, isolated
+      from `~/.spike`, leaving no artifacts.
+- [ ] The "not re-initialized twice" invariant has an explicit test.
+- [ ] The shard-restore round trip has an in-process test mirroring
+      the live drill's semantics.
+- [ ] Secret and policy CRUD paths are covered end to end at the state
+      layer, including deletion and undeletion.
+- [ ] Slice B lands only after the open questions are answered.