Skip to content

Commit ea494f1

Browse files
gedaiuclaude
andauthored
Services refuse to boot without their required config, and cluster-agent routes split one per file (#2082)
Every service now fails fast at startup when a required environment variable or database parameter is missing, instead of booting half-configured. `requiredEnv` and new enforce helpers in `libs/shared` do the checking, and `pg-pool` refuses a connection with missing parameters. Request body limits and the event drain timeout now come from shared constants, so the Floor, lore-api, the event-router and cluster-agent can no longer disagree about them. cluster-agent's 235-line `routes/cluster.ts` is split into one file per endpoint under `routes/cluster/` (agents, jobs, pods, pod listing, per-task tokens, restart), with the shared dependencies in `cluster-route-deps.ts`. The heartbeat loop and registrant were tidied along the way. The branch also adds the `collaborative-feature-planning` spec. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 1a12135 commit ea494f1

62 files changed

Lines changed: 1123 additions & 602 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

adrs/ADR-024-ubiquitous-language-execution-model.md

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -114,10 +114,10 @@ and lose the update; no `resourceVersion` ever crosses the wire.
114114
describes the read surface, which callers do still reach over HTTP.)*
115115
([validated by reports created:false for code 409, so a redelivered claim is idempotent](apps/cluster-agent/src/outbound/kube-agent-api.test.ts#L26), [`kube-agent-api.test.ts:19`](apps/cluster-agent/src/outbound/kube-agent-api.test.ts#L19), [`kube-agent-api.test.ts:33`](apps/cluster-agent/src/outbound/kube-agent-api.test.ts#L33), [`kube-agent-api.test.ts:39`](apps/cluster-agent/src/outbound/kube-agent-api.test.ts#L39), [`kube-agent-api.test.ts:47`](apps/cluster-agent/src/outbound/kube-agent-api.test.ts#L47))
116116
- A missing CR is an ordinary answer — `found:false` at 200, not a 404 that
117-
would be indistinguishable from the route itself being absent. ([validated by answers 200 with found:false for a missing CR, not 404](apps/cluster-agent/src/transport/routes/cluster.test.ts#L73), [`cluster.test.ts:131`](apps/cluster-agent/src/transport/routes/cluster.test.ts#L131))
117+
would be indistinguishable from the route itself being absent. ([validated by answers 200 with found:false for a missing CR, not 404](apps/cluster-agent/src/transport/routes/cluster/cluster-routes.test.ts#L73), [`cluster-routes.test.ts:131`](apps/cluster-agent/src/transport/routes/cluster/cluster-routes.test.ts#L131))
118118
- The list serves ONE apiserver page per call and the caller drives `continue`.
119119
A one-shot list is not a convenience: 180 accumulated CRs at ~1.4MB of status
120-
each blew Node's heap and crash-looped the Floor on 2026-07-24. ([validated by passes the caller's continue token straight through, one page per call](apps/cluster-agent/src/transport/routes/cluster.test.ts#L84), [`cluster.test.ts:97`](apps/cluster-agent/src/transport/routes/cluster.test.ts#L97), [`cluster.test.ts:228`](apps/cluster-agent/src/transport/routes/cluster.test.ts#L228), [`cluster.test.ts:218`](apps/cluster-agent/src/transport/routes/cluster.test.ts#L218))
120+
each blew Node's heap and crash-looped the Floor on 2026-07-24. ([validated by passes the caller's continue token straight through, one page per call](apps/cluster-agent/src/transport/routes/cluster/cluster-routes.test.ts#L84), [`cluster-routes.test.ts:97`](apps/cluster-agent/src/transport/routes/cluster/cluster-routes.test.ts#L97), [`cluster-routes.test.ts:228`](apps/cluster-agent/src/transport/routes/cluster/cluster-routes.test.ts#L228), [`cluster-routes.test.ts:218`](apps/cluster-agent/src/transport/routes/cluster/cluster-routes.test.ts#L218))
121121
- The paging the route requires is walked by the CLIENT, not pushed onto every
122122
caller: `listByLabel` follows `continue` to the end and returns the whole
123123
match. A truncated list is worse than a failed one — it answers, and the
@@ -148,7 +148,7 @@ and lose the update; no `resourceVersion` ever crosses the wire.
148148
- `DELETE /api/cluster/per-task-tokens/{taskId}` reclaims a terminal task's
149149
Secret key and catalog clones — the one per-task-token operation that stays
150150
a route, since a settled task's cleanup runs from the Floor, not the cluster
151-
that provisioned. ([validated by reclaims a task's token and catalog clones](apps/cluster-agent/src/transport/routes/cluster.test.ts#L207))
151+
that provisioned. ([validated by reclaims a task's token and catalog clones](apps/cluster-agent/src/transport/routes/cluster/cluster-routes.test.ts#L207))
152152
- One call also means one OUTCOME: a provision whose recipe pair fails to land
153153
takes back everything it had already provisioned — the Secret key AND any
154154
catalog object that landed before the failure — before it throws. `cleanup`
@@ -165,9 +165,9 @@ and lose the update; no `resourceVersion` ever crosses the wire.
165165
from being amputated by a merge split across the network.
166166
([validated by writes the station before the agent definition that points at it](apps/cluster-agent/src/outbound/paired-writes.test.ts#L27), [`paired-writes.test.ts:35`](apps/cluster-agent/src/outbound/paired-writes.test.ts#L35), [`paired-writes.test.ts:54`](apps/cluster-agent/src/outbound/paired-writes.test.ts#L54))
167167
- The log tail is clamped by the AGENT, because the Floor's clamp no longer
168-
protects this process's heap. ([validated by clamps the tail server-side rather than trusting the caller](apps/cluster-agent/src/transport/routes/cluster.test.ts#L121), [`cluster.test.ts:187`](apps/cluster-agent/src/transport/routes/cluster.test.ts#L187), [`cluster.test.ts:197`](apps/cluster-agent/src/transport/routes/cluster.test.ts#L197), [`cluster.test.ts:177`](apps/cluster-agent/src/transport/routes/cluster.test.ts#L177))
168+
protects this process's heap. ([validated by clamps the tail server-side rather than trusting the caller](apps/cluster-agent/src/transport/routes/cluster/cluster-routes.test.ts#L121), [`cluster-routes.test.ts:187`](apps/cluster-agent/src/transport/routes/cluster/cluster-routes.test.ts#L187), [`cluster-routes.test.ts:197`](apps/cluster-agent/src/transport/routes/cluster/cluster-routes.test.ts#L197), [`cluster-routes.test.ts:177`](apps/cluster-agent/src/transport/routes/cluster/cluster-routes.test.ts#L177))
169169
- Every route requires the same bearer token every other service-to-service
170-
call presents. ([validated by refuses every route without a bearer token](apps/cluster-agent/src/transport/routes/cluster.test.ts#L143), [validated by refuses to restart without a bearer token](apps/cluster-agent/src/transport/routes/cluster.test.ts#L266))
170+
call presents. ([validated by refuses every route without a bearer token](apps/cluster-agent/src/transport/routes/cluster/cluster-routes.test.ts#L143), [validated by refuses to restart without a bearer token](apps/cluster-agent/src/transport/routes/cluster/cluster-routes.test.ts#L266))
171171
- A CR the controller has not stamped yet reads as Pending rather than absent —
172172
the distinction a watcher acts on. ([validated by a CR the controller has not stamped yet maps to Pending, not absence](libs/shared/src/outbound/cluster/agent-node-status.test.ts#L6), [`agent-node-status.test.ts:12`](libs/shared/src/outbound/cluster/agent-node-status.test.ts#L12))
173173
- An empty minted token is refused where the cause is legible, rather than
@@ -189,7 +189,7 @@ scoped settling a run to the one cluster this Floor can reach; it now settles
189189
from the event's own report, which carries the full status. The distinction
190190
above still governs the READ surface — the reconcile pass, the reaper's status
191191
probe, the pod-log reads — where a caller genuinely has to ask.*
192-
([validated by answers 200 with found:false for a missing CR, not 404](apps/cluster-agent/src/transport/routes/cluster.test.ts#L73), [`k8s-errors.test.ts:28`](apps/cluster-agent/src/lib/k8s-errors.test.ts#L31), [`k8s-errors.test.ts:44`](apps/cluster-agent/src/lib/k8s-errors.test.ts#L47), [`k8s-errors.test.ts:56`](apps/cluster-agent/src/lib/k8s-errors.test.ts#L59), [`kubernetes.test.ts:21`](apps/floor/src/events/handlers/kubernetes.test.ts#L21), [`kubernetes.test.ts:49`](apps/floor/src/events/handlers/kubernetes.test.ts#L49), [`kubernetes.test.ts:55`](apps/floor/src/events/handlers/kubernetes.test.ts#L55))
192+
([validated by answers 200 with found:false for a missing CR, not 404](apps/cluster-agent/src/transport/routes/cluster/cluster-routes.test.ts#L73), [`k8s-errors.test.ts:28`](apps/cluster-agent/src/lib/k8s-errors.test.ts#L31), [`k8s-errors.test.ts:44`](apps/cluster-agent/src/lib/k8s-errors.test.ts#L47), [`k8s-errors.test.ts:56`](apps/cluster-agent/src/lib/k8s-errors.test.ts#L59), [`kubernetes.test.ts:21`](apps/floor/src/events/handlers/kubernetes.test.ts#L21), [`kubernetes.test.ts:49`](apps/floor/src/events/handlers/kubernetes.test.ts#L49), [`kubernetes.test.ts:55`](apps/floor/src/events/handlers/kubernetes.test.ts#L55))
193193

194194
The reconcile pass keeps paging, and its seam narrowed with the cut: it now
195195
depends on one page-fetch method rather than a slice of a Kubernetes client, so
@@ -200,7 +200,7 @@ The Role this service carries also closes two gaps the Floor had been silently
200200
living with: it never held `delete` on `agents` or `agents/status`, yet issued
201201
both at sites that swallowed the failure — which is why the CR prune could
202202
never actually shrink the pile it was written to shrink.
203-
([validated by deletes a CR — the verb the Floor's RBAC never granted](apps/cluster-agent/src/transport/routes/cluster.test.ts#L108))
203+
([validated by deletes a CR — the verb the Floor's RBAC never granted](apps/cluster-agent/src/transport/routes/cluster/cluster-routes.test.ts#L108))
204204

205205
Hierarchy: **Factory ⊃ Floor(s) ⊃ AssemblyLines ⊃ Stations ⊃ Agents** — the design
206206
side; its runtime shadow is **AssemblyRun ⊃ StationRuns ⊃ Agents**.

apps/cluster-agent/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,7 @@ probe the apiserver.
8383
| `LORE_AGENTS_NAMESPACE` | Namespace it acts on (default `ai-agents`) |
8484
| `LORE_AGENT_SECRETS_NAME` | Secret holding per-task token keys (default `agent-secrets`) |
8585
| `GITHUB_APP_ID` / `GITHUB_APP_PRIVATE_KEY` / `GITHUB_APP_INSTALLATION_ID` | The GitHub App triple used to mint per-task installation tokens |
86-
| `LORE_CLUSTER_AGENT_IDENTITY_SECRET` / `_NAMESPACE` / `_KEY` | Where the registered `{id, token}` persists; local runs fall back to `LORE_CLUSTER_AGENT_IDENTITY_FILE` |
86+
| `LORE_CLUSTER_AGENT_IDENTITY_SECRET` / `_NAMESPACE` / `_KEY` | Required. The Kubernetes Secret the registered `{id, token}` persists in (`_KEY` defaults to `identity.json`); `npm start` points it at minikube |
8787

8888
The first three have no defaults and no off switch: the process exits naming
8989
whichever are missing.
Lines changed: 5 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,18 @@
11
/** The cluster surface this agent exposes: what the Kubernetes clients in `outbound` provide and what the `transport` routes call — a contract owned by neither side. */
22

3-
import type {
4-
Agent as AgentCr,
5-
AgentDefinition,
6-
Station,
7-
} from "@re-cinq/agent-contracts";
3+
import type { Agent as AgentCr } from "@re-cinq/agent-contracts";
84
import type {
95
AgentPodInfo,
106
PodSummary,
117
RunningPodInfo,
128
} from "@re-cinq/lore-shared";
9+
import type { Page, PageRequest } from "@re-cinq/lore-shared/lib/paginate.js";
10+
import type { CrdPair } from "@re-cinq/lore-shared/project/agents/agent-crd.js";
1311

1412
export interface ClusterDeps {
1513
agents: {
1614
get(name: string): Promise<AgentCr | null>;
17-
list(opts: {
18-
labelSelector?: string;
19-
limit: number;
20-
continue?: string;
21-
}): Promise<{ items: AgentCr[]; continueToken?: string }>;
15+
list(request: PageRequest): Promise<Page<AgentCr>>;
2216
remove(name: string): Promise<void>;
2317
};
2418
pods: {
@@ -32,10 +26,7 @@ export interface ClusterDeps {
3226
cleanup(taskId: string): Promise<void>;
3327
};
3428
catalog: {
35-
applyPair(pair: {
36-
agentDefinition: AgentDefinition;
37-
station: Station;
38-
}): Promise<void>;
29+
applyPair(pair: CrdPair): Promise<void>;
3930
deletePair(name: string): Promise<void>;
4031
};
4132
}

apps/cluster-agent/src/domain/identity.ts

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,9 @@ export interface IdentityStore {
1010
save(identity: ClusterAgentIdentity): Promise<void>;
1111
}
1212

13-
/** Where the identity persists — decided at boot so a half-configured Secret store refuses to start instead of idling behind a green /healthz. */
14-
export type IdentityStoreConfig =
15-
| { kind: "file"; path: string }
16-
| { kind: "secret"; name: string; namespace: string; key: string };
13+
/** The Secret the identity persists in — decided at boot so a half-configured store refuses to start instead of idling behind a green /healthz. */
14+
export interface IdentityStoreConfig {
15+
name: string;
16+
namespace: string;
17+
key: string;
18+
}

apps/cluster-agent/src/events/claim/heartbeat-loop.test.ts

Lines changed: 2 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ import {
33
heartbeatIntervalMs,
44
heartbeatOnce,
55
runHeartbeatLoop,
6+
type HeartbeatOutcome,
67
} from "./heartbeat-loop.js";
78

89
const IDENTITY = { id: "agent-1", token: "lca_tok" };
@@ -110,11 +111,7 @@ describe("heartbeatOnce", () => {
110111

111112
describe("runHeartbeatLoop", () => {
112113
it("beats at the fixed interval and re-registers on an unauthorized beat", async () => {
113-
const outcomes: Array<"ok" | "unauthorized" | "error"> = [
114-
"ok",
115-
"unauthorized",
116-
"error",
117-
];
114+
const outcomes: HeartbeatOutcome[] = ["ok", "unauthorized", "error"];
118115
const sleeps: number[] = [];
119116
let reRegistered = 0;
120117
let ticks = 0;

apps/cluster-agent/src/events/claim/heartbeat-loop.ts

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -19,10 +19,13 @@ export interface HeartbeatDeps {
1919
log?: (line: string) => void;
2020
}
2121

22-
/** One beat. Returns "ok" | "unauthorized" | "error"; never throws. */
22+
/** What one beat concluded: the cluster answered, refused the credential, or could not be reached. */
23+
export type HeartbeatOutcome = "ok" | "unauthorized" | "error";
24+
25+
/** One beat; never throws. */
2326
export async function heartbeatOnce(
2427
deps: HeartbeatDeps,
25-
): Promise<"ok" | "unauthorized" | "error"> {
28+
): Promise<HeartbeatOutcome> {
2629
const { id, token } = deps.identity();
2730
const log = heartbeatLog(deps);
2831

@@ -79,9 +82,11 @@ export async function runHeartbeatLoop(deps: HeartbeatLoopDeps): Promise<void> {
7982
await runPollLoop<"ok" | "unauthorized" | "error">({
8083
tick: deps.beat,
8184
onOutcome: async (outcome) => {
82-
if (outcome === "unauthorized") {
83-
await deps.reRegister();
85+
if (outcome !== "unauthorized") {
86+
return;
8487
}
88+
89+
await deps.reRegister();
8590
},
8691
delayFor: () => deps.intervalMs,
8792
sleep: deps.sleep,

apps/cluster-agent/src/events/claim/registrant.ts

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,13 @@ export interface RegistrantOpts {
4242
running: () => boolean;
4343
}
4444

45+
/** What registering takes: the triple to register with, where to persist the identity, and how to hand the minted token to the run pods. Named because both `establishIdentity` and the single-flight re-registration take exactly this. */
46+
export interface RegistrationOpts {
47+
config: RegistrationConfig;
48+
store: IdentityStore;
49+
publishTelemetryCredential: (id: ClusterAgentIdentity) => Promise<void>;
50+
}
51+
4552
// What both side loops need: where to talk, who this cluster is, and whether the process is still up. One shape because they are started together and stopped together.
4653
interface SideLoopOpts {
4754
env: NodeJS.ProcessEnv;
@@ -77,10 +84,8 @@ export async function runRegistrant(opts: RegistrantOpts): Promise<void> {
7784
});
7885
}
7986

80-
/** Registers, and hands back the identity as a GETTER rather than a value: a 401 rotates it mid-run, and every loop must read the current one rather than the one it captured at startup. */
81-
async function establishIdentity(
82-
opts: Pick<RegistrantOpts, "config" | "store" | "publishTelemetryCredential">,
83-
): Promise<{
87+
/** Hands back the identity as a GETTER: a 401 rotates it mid-run, and every loop must read the current one, not the one it captured at startup. */
88+
async function establishIdentity(opts: RegistrationOpts): Promise<{
8489
identity: () => ClusterAgentIdentity;
8590
reRegister: () => Promise<ClusterAgentIdentity | null>;
8691
}> {
@@ -100,7 +105,7 @@ async function establishIdentity(
100105

101106
// The first registration, with the line that says this cluster is now claiming. Announced here rather than by the caller because the id only exists once registration has succeeded.
102107
async function registerAndAnnounce(
103-
opts: Pick<RegistrantOpts, "config" | "store" | "publishTelemetryCredential">,
108+
opts: RegistrationOpts,
104109
): Promise<ClusterAgentIdentity> {
105110
const identity = await registerWithBackoff({
106111
config: opts.config,

apps/cluster-agent/src/events/claim/start-claim-loop.ts

Lines changed: 8 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -10,10 +10,7 @@ import { writeAgentEventsAuth } from "./agent-events-secret.js";
1010
import { stopLatch } from "../../lib/stop-latch.js";
1111
import { runRegistrant, type RegistrantOpts } from "./registrant.js";
1212
import { enforceCatalogProfile } from "../../work/catalog/catalog-sync-loop.js";
13-
import {
14-
FileIdentityStore,
15-
identityStoreConfig,
16-
} from "../../outbound/identity-store.js";
13+
import { identityStoreConfig } from "../../outbound/identity-store.js";
1714
import type {
1815
ClusterAgentIdentity,
1916
IdentityStore,
@@ -90,13 +87,15 @@ function rotationSlot(): {
9087
};
9188
}
9289

93-
/** Detached on purpose: `startClaimLoop` returns a handle immediately so a caller can stop the agent before it has finished registering. The catch is unreachable by design — register and claim never throw — but a defect here must surface as a log rather than an unhandled rejection that kills the process. */
90+
/** Detached so a caller can stop the agent before it registers; register and claim never throw, so the catch only turns a defect into a log instead of a process-killing unhandled rejection. */
9491
function launchRegistrant(
9592
storeConfig: IdentityStoreConfig,
9693
opts: Omit<RegistrantOpts, "store">,
9794
): void {
98-
void buildIdentityStore(storeConfig)
99-
.then((store) => runRegistrant({ ...opts, store }))
95+
void Promise.resolve(storeConfig)
96+
.then((config) =>
97+
runRegistrant({ ...opts, store: buildIdentityStore(config) }),
98+
)
10099
.catch((err) => {
101100
console.error(
102101
"[cluster-agent] claim loop crashed — this agent will not register or claim until restarted:",
@@ -138,14 +137,8 @@ function publishCredential(
138137
};
139138
}
140139

141-
/** In a cluster the identity persists through the Kubernetes Secret API — the chart mounts the container read-only, so a file write would EROFS and strand the identity. File store only for local runs. */
142-
async function buildIdentityStore(
143-
config: IdentityStoreConfig,
144-
): Promise<IdentityStore> {
145-
if (config.kind === "file") {
146-
return new FileIdentityStore(config.path);
147-
}
148-
140+
/** Through the Kubernetes Secret API, never a file: the chart mounts the container read-only, so a file write would EROFS and strand the identity. */
141+
function buildIdentityStore(config: IdentityStoreConfig): IdentityStore {
149142
return new KubeIdentityStore(
150143
kubeIdentitySecretsApi(config.namespace),
151144
config.name,

0 commit comments

Comments
 (0)