Eryph is designed to run as cooperating components (controller, compute API, identity, network, genepool, host agents). In eryph-zero these components are composed into a single process and talk over an in-memory bus. The earlier standalone runtimes still exist but were no longer kept in sync once eryph-zero was completed, so today a lot of shared knowledge between components is short-cut through in-process sharing (a shared container, shared singletons, a shared state database, an in-memory bus) that would not survive a real split across processes or hosts.
This proposal describes a concept to prepare eryph for component splitting — running the components as separate services again — with the focus on the biggest missing piece: how configuration is distributed to components. This is not about failover clustering or an HA controller; it is about making the component boundaries real again.
The preferred model is: the controller owns the cluster configuration and distributes it via messages to components that have registered as alive. A model where every component needs a full local copy of the configuration (offline sync) is explicitly not wanted. Networking is the counter-example that already works this way and is largely cluster-ready — but it still needs a service-auth PKI.
Goal and non-goals
Goal
- A controller-driven configuration distribution concept (push over messages, not local copies).
- A way for components to register and signal liveness.
- A way for components to authenticate when they register and receive configuration.
Non-goals (for now)
- eryph does not build HA itself — no automatic failure detection, leader election, fencing or state replication. HA is an operational concern delegated to the deployment platform (Kubernetes for the control plane, Windows failover clustering for Windows hosts).
Deployment & scaling model. The state database is a shared DB used by both the endpoint services and the controller. In cluster deployments this is MariaDB, which already exists as a state-store provider (eryph-zero uses SQLite; the provider is pluggable). The scaling model is:
- Endpoint services (API): scale horizontally, active-active — stateless app servers sharing the DB and consuming the API queue as competing consumers. This is the primary scaling focus. The compute API reading the DB directly is the intended endpoint pattern (like nova-api over a cell DB), not something to refactor away — in cluster mode it simply runs against the shared MariaDB rather than the local SQLite file.
- Hosts (agents): scale horizontally, as today — they go through the controller over the bus and never touch the DB.
- Controller: cold standby, not active-active — normally one active instance, with a standby that takes over on restart so the cluster stays operational for users. eryph does not implement active-active controller coordination; the concurrency-safety already in the code (per-resource locks, competing-consumer idempotency) exists so a brief failover overlap is safe, while the failover itself is orchestrated operationally.
Component registration
Components should become first-class, registered entities instead of being discovered statically (today discovery is a hardcoded endpoint map plus a MachineName-based agent locator, and liveness is only inferred from the last inventory timestamp).
- A component registers with the controller on startup over the bus (reusing the existing per-recipient queues and routing), advertising its type, a stable component id, its version and its capabilities.
- A periodic heartbeat keeps the registration alive; the controller marks components stale/dead past a TTL.
- The registration becomes the controller's service catalog: it replaces the static endpoint map and the
MachineName agent locator, and it is the basis for routing commands to the right component instance.
- This generalizes the mechanism networking already anticipates (
IClusterTopologyProvider is documented in code as the seam where "host enrollment" will later read registered hosts from the state DB).
This is conceptually the same idea OpenStack uses (services report their state to a central registry; the conductor owns the database and agents talk to it via RPC).
Configuration distribution
The controller holds the authoritative copy of each cluster configuration domain — a named, versioned configuration namespace such as network-providers or projects — as a record with a monotonic version. A component never reads cluster config from the database; it receives the config over the bus and applies it through a domain-specific realizer: a piece of code that translates a config snapshot into actual system state (network rules, switch config, etc.) idempotently, so re-applying the same version is always safe. (eryph already uses this pattern for networking.) The controller tracks, per component, which version of each domain has actually been applied — that record is what makes the system self-correcting.
The lifecycle is a simple loop:
Component Controller
│ register (+ versions I already have) │
│────────────────────────────────────────▶│ authenticate, store registration
│ │ pick the domains this component may see
│ config snapshot {domain, version, data} │
│◀─────────────────────────────────────────│
│ apply via realizer (idempotent) │
│ applied {domain, version, ok/fail} │
│────────────────────────────────────────▶│ record applied version
│ │
│ ── periodic heartbeat (+ applied vers) ▶│ detect drift (applied < authoritative) → re-push
A component is drifted on a domain when its recorded applied version is behind the controller's authoritative version (for example, it was briefly down and missed a change notification).
- Initial sync on registration: the controller sends a snapshot of every domain the component is entitled to, scoped by its service role (e.g. host agent vs. endpoint service — the role determines which config domains a component may receive).
- Incremental update: a config change bumps the domain's version (under a per-domain lock) and the controller notifies / pushes to the affected live components.
- Acknowledgement: the component applies the config and reports the applied version; the controller records it per component.
- Heartbeat & drift: each heartbeat echoes the component's applied versions; if they lag the authoritative version (e.g. a change notification was missed), the controller re-pushes. So missed messages heal on the next heartbeat.
- Restart: a component re-registers with the versions it still has, and the controller sends only the newer/missing domains (a full snapshot if it has none). Because realizers are idempotent, re-applying is always safe.
- Apply failure: a failed apply is retried with backoff a bounded number of times, then the component is flagged degraded for that domain — never an infinite re-push loop.
Delivery, retries and idempotent late/duplicate handling reuse the existing operation/saga machinery rather than a parallel mechanism. The net guarantee: every live, in-scope component converges to the authoritative version, even across missed messages and restarts, and a late acknowledgement can never regress the recorded state.
How the flow is authenticated. The config flow uses only the message bus, and authentication happens at the transport — not via per-message signatures. A component proves its identity once, by presenting its certificate when it connects to the message broker (mutual TLS); the broker then stamps a verified sender identity on every message that component publishes. So the controller scopes config by the verified sender, never by anything the message merely claims about itself. (A separate, also certificate-secured, HTTP path exists for enrollment and for the few APIs that need a bearer token — but it carries no config and is not part of this flow.)
Configuration model
Not all configuration is the same. We distinguish:
- Cluster-wide configuration (network providers, projects/policies, genepool sources): the controller is the authority and distributes it. Some of this is file-authoritative today — the file is the source of truth (e.g. network providers are read from
p_networks.yml). Making the controller the authority instead is an authority inversion: the controller-owned record becomes canonical and the file becomes a compatibility mirror. It has to be done carefully and behavior-preserving (keep writing the file as a mirror until the controller-owned record is trusted), not a quick seeder swap.
- Host-local / agent configuration — an explicit exception: the per-agent configuration expresses what the agent can access and how (datastores, switches, host capabilities). Even though central management would be convenient, each agent keeps its own configuration locally and authoritatively. The controller never overwrites it; the agent only advertises derived capabilities upward at registration. This preserves the property that an agent can still run from its local configuration when the controller is unreachable.
- Identity's own clients — another exception: identity is the identity service, so it owns its clients/identities. The
system-client master credential is seeded and owned locally by identity, identically in standalone and cluster, and is never distributed or owned by the controller. Per-component identities are provisioned through identity itself (its client service) at enrollment. A future identity split needs identity's own persistent client store — that is identity owning its data, not the controller taking over.
- Secret-bearing configuration: secrets already travel inside messages today (config drives are serialized into commands, with a redaction flag). Any secret-bearing configuration that reaches a remote component must build on that existing mechanism.
Service authentication & PKI
Components must authenticate when they register and receive configuration. The proposal is a layered trust model with a single root:
- A new cluster CA issues a per-component certificate (leaf). This is what is missing today — the certificate code can only create self-signed certs, not a CA.
- The certificate is the bootstrap credential the component presents to the identity service, which already supports client-credentials and a certificate-based client assertion. Identity issues a role-scoped token — so identity stays the authorization authority (Keystone-style), the certificate is the transport identity.
- The same certificate is used for mutual TLS on the message broker, on service-to-service HTTP, and (eventually) on the OVN southbound connection.
- Enrollment / bootstrap: a remote component is enrolled with a one-time enrollment token (kubeadm-join style), pins the CA out of band, generates a key locally, and receives its leaf + CA chain. In eryph-zero there is no enrollment step — a startup handler creates the CA and issues the local components' certificates in-process, so nothing changes for the single-process user.
- Rotation/revocation: short-lived leaves with automatic renewal; the registration record acts as the revocation authority (a deregistered component is no longer renewed and loses its broker user).
Networking
Networking is the template: the controller already owns the OVN northbound database and the realizer, and the host agent already runs the OVN chassis against the southbound database. The remaining gaps are (1) the northbound DB + realizer still run in the controller process (splitting the network module onto its own host is real work) and (2) OVN connections are still plaintext — the component PKI provides the certificate OVS expects, so securing OVN is then blocked only on the OVN library exposing TLS parameters.
Decided design choices
- Broker authorization: a single message-broker vhost with per-component users, scoped by role-derived permissions (e.g. a host agent may write to the controller queue but only read its own queue). Per-role vhosts are kept in reserve for a future multi-tenant-broker scenario.
- OVN remote chassis: no separate milestone — folded into the host-agent split, with OVN-over-TLS as a hard entry gate (and a trusted-segment interim if the OVN TLS work lags).
- Controller restart / standby takeover: the split runtime uses a durable saga/timeout store so in-transit messages are not lost, but in-flight operations are not resumed when the standby takes over (they fail cleanly and are re-run). Resuming operations would edge toward HA, which is out of scope. eryph-zero keeps its in-memory profile unchanged.
Migration approach
The work can be staged so that eryph-zero stays single-process and fully working at every step, by introducing the abstractions first (registry, config distribution, CA) with single-host implementations that behave exactly as today, then:
- Wire the cluster runtime profile (selection, not new infrastructure — the selected components support both single-process and split): the existing shared MariaDB state-store provider, a durable Rebus saga/timeout store via the existing selectors, and a state-DB-backed distributed lock (rides MariaDB) instead of the file lock. eryph-zero keeps its SQLite + in-memory + file-lock profile unchanged.
- Make modules talk only via the registry/config messages even in-process.
- Invert authority for the one file-authoritative cluster domain — network providers — keeping the file as a mirror. (Identity is not inverted; it keeps owning its clients.)
- Introduce per-component identity + the cluster CA in-process.
- Switch the split runtime to a durable bus/transport profile (RabbitMQ + mTLS).
- Split the host agent (it never touches the database), gated on OVN TLS.
- Scale out the endpoint services horizontally against the shared DB, and generalize to the remaining components (network module split, genepool secrets, identity persistence).
Open considerations
- The shared MariaDB state-store backend already exists as a provider, so cluster mode is a deployment/config choice, not a DB-replacement project. The only ongoing requirement is keeping the MariaDB migrations in sync as new state tables are added (component registration, config records, enrollment tokens). The compute API's direct DB reads are the intended endpoint pattern, not a refactor target.
- The genepool currently holds per-genepool API keys in-process; cluster mode must scope/distribute them through the secret-handling path.
- The controller's cold-standby model keeps cluster mode simple. Because only one controller is active at a time, the would-be active-active hazards are naturally satisfied and need no eryph-built coordination: the cluster-global scheduled jobs (inventory/cleanup, per-instance timers), the in-process OVN northbound realizer, and the id generator (hardcoded id
0) all assume a single active writer, which holds. What cluster mode does need is a state-DB-backed distributed lock instead of the host-local file lock (the Medallion distributed-lock abstraction is already in place and can be backed by the MariaDB DB) so the existing per-resource locks stay valid across the brief failover overlap when an old active and a starting standby may briefly coexist. The per-resource locks, competing-consumer idempotency and timestamp-guarded status make that overlap safe without leader election.
Eryph is designed to run as cooperating components (controller, compute API, identity, network, genepool, host agents). In eryph-zero these components are composed into a single process and talk over an in-memory bus. The earlier standalone runtimes still exist but were no longer kept in sync once eryph-zero was completed, so today a lot of shared knowledge between components is short-cut through in-process sharing (a shared container, shared singletons, a shared state database, an in-memory bus) that would not survive a real split across processes or hosts.
This proposal describes a concept to prepare eryph for component splitting — running the components as separate services again — with the focus on the biggest missing piece: how configuration is distributed to components. This is not about failover clustering or an HA controller; it is about making the component boundaries real again.
The preferred model is: the controller owns the cluster configuration and distributes it via messages to components that have registered as alive. A model where every component needs a full local copy of the configuration (offline sync) is explicitly not wanted. Networking is the counter-example that already works this way and is largely cluster-ready — but it still needs a service-auth PKI.
Goal and non-goals
Goal
Non-goals (for now)
Deployment & scaling model. The state database is a shared DB used by both the endpoint services and the controller. In cluster deployments this is MariaDB, which already exists as a state-store provider (eryph-zero uses SQLite; the provider is pluggable). The scaling model is:
Component registration
Components should become first-class, registered entities instead of being discovered statically (today discovery is a hardcoded endpoint map plus a
MachineName-based agent locator, and liveness is only inferred from the last inventory timestamp).MachineNameagent locator, and it is the basis for routing commands to the right component instance.IClusterTopologyProvideris documented in code as the seam where "host enrollment" will later read registered hosts from the state DB).This is conceptually the same idea OpenStack uses (services report their state to a central registry; the conductor owns the database and agents talk to it via RPC).
Configuration distribution
The controller holds the authoritative copy of each cluster configuration domain — a named, versioned configuration namespace such as
network-providersorprojects— as a record with a monotonic version. A component never reads cluster config from the database; it receives the config over the bus and applies it through a domain-specific realizer: a piece of code that translates a config snapshot into actual system state (network rules, switch config, etc.) idempotently, so re-applying the same version is always safe. (eryph already uses this pattern for networking.) The controller tracks, per component, which version of each domain has actually been applied — that record is what makes the system self-correcting.The lifecycle is a simple loop:
A component is drifted on a domain when its recorded applied version is behind the controller's authoritative version (for example, it was briefly down and missed a change notification).
Delivery, retries and idempotent late/duplicate handling reuse the existing operation/saga machinery rather than a parallel mechanism. The net guarantee: every live, in-scope component converges to the authoritative version, even across missed messages and restarts, and a late acknowledgement can never regress the recorded state.
How the flow is authenticated. The config flow uses only the message bus, and authentication happens at the transport — not via per-message signatures. A component proves its identity once, by presenting its certificate when it connects to the message broker (mutual TLS); the broker then stamps a verified sender identity on every message that component publishes. So the controller scopes config by the verified sender, never by anything the message merely claims about itself. (A separate, also certificate-secured, HTTP path exists for enrollment and for the few APIs that need a bearer token — but it carries no config and is not part of this flow.)
Configuration model
Not all configuration is the same. We distinguish:
p_networks.yml). Making the controller the authority instead is an authority inversion: the controller-owned record becomes canonical and the file becomes a compatibility mirror. It has to be done carefully and behavior-preserving (keep writing the file as a mirror until the controller-owned record is trusted), not a quick seeder swap.system-clientmaster credential is seeded and owned locally by identity, identically in standalone and cluster, and is never distributed or owned by the controller. Per-component identities are provisioned through identity itself (its client service) at enrollment. A future identity split needs identity's own persistent client store — that is identity owning its data, not the controller taking over.Service authentication & PKI
Components must authenticate when they register and receive configuration. The proposal is a layered trust model with a single root:
Networking
Networking is the template: the controller already owns the OVN northbound database and the realizer, and the host agent already runs the OVN chassis against the southbound database. The remaining gaps are (1) the northbound DB + realizer still run in the controller process (splitting the network module onto its own host is real work) and (2) OVN connections are still plaintext — the component PKI provides the certificate OVS expects, so securing OVN is then blocked only on the OVN library exposing TLS parameters.
Decided design choices
Migration approach
The work can be staged so that eryph-zero stays single-process and fully working at every step, by introducing the abstractions first (registry, config distribution, CA) with single-host implementations that behave exactly as today, then:
Open considerations
0) all assume a single active writer, which holds. What cluster mode does need is a state-DB-backed distributed lock instead of the host-local file lock (the Medallion distributed-lock abstraction is already in place and can be backed by the MariaDB DB) so the existing per-resource locks stay valid across the brief failover overlap when an old active and a starting standby may briefly coexist. The per-resource locks, competing-consumer idempotency and timestamp-guarded status make that overlap safe without leader election.