diff --git a/public/talos/v1.14/configure-your-talos-cluster/system-configuration/discovery.mdx b/public/talos/v1.14/configure-your-talos-cluster/system-configuration/discovery.mdx index 8447d71f..937bae43 100644 --- a/public/talos/v1.14/configure-your-talos-cluster/system-configuration/discovery.mdx +++ b/public/talos/v1.14/configure-your-talos-cluster/system-configuration/discovery.mdx @@ -19,8 +19,8 @@ When discovery is enabled, this information is shared and kept up to date across Discovery works through a **registry**, a backend that nodes publish their connection information to and read peer information from. Talos supports two registry types: -- **Service registry**: Nodes publish to and read from an external discovery service. This is enabled by default and does not depend on Kubernetes or etcd, so it continues to work even when Kubernetes is unavailable. -- **Kubernetes registry**: Nodes publish discovery data as annotations on Kubernetes `Node` resources. This is disabled by default. +- **Service registry**: Nodes publish to and read from an external discovery service. This is configured with a `DiscoveryServiceConfig` document, enabled by default, and does not depend on Kubernetes or etcd, so it continues to work even when Kubernetes is unavailable. +- **Kubernetes registry**: Nodes publish discovery data as annotations on Kubernetes `Node` resources. This is deprecated and disabled by default. The Kubernetes registry is deprecated. Starting with Kubernetes 1.32, the `AuthorizeNodeWithSelectors` feature gate restricts `Node` resource read access in a way that prevents the Kubernetes registry from functioning correctly. Disabling the feature gate is not recommended as it removes other important security protections. @@ -34,22 +34,30 @@ To see a live demo of cluster discovery, see the video below: ## Registries -By default, Talos uses the `service` registry. The `kubernetes` registry is disabled by default. Peers are aggregated from all enabled registries. +By default, Talos uses the `service` registry. The `kubernetes` registry is deprecated and disabled by default. Peers are aggregated from all enabled registries. -To disable a registry, set `disabled: true` in the cluster configuration. For example, to disable the `service` registry: +The service registry is enabled by the presence of a `DiscoveryServiceConfig` document. A freshly generated machine configuration includes one named `default` that points at the public discovery service at `https://discovery.talos.dev/`. + +To disable the discovery service, remove all `DiscoveryServiceConfig` documents from the machine configuration. With no service registry and no Kubernetes registry, [member discovery is effectively disabled](#what-changes-when-discovery-is-disabled). + + +The `DiscoveryServiceConfig` document and the legacy `.cluster.discovery` configuration block are mutually exclusive. A machine configuration must not contain both. + + +### Kubernetes registry + +The Kubernetes registry has no `DiscoveryServiceConfig` equivalent and should not be used in new clusters. It can only be enabled through the deprecated `.cluster.discovery` configuration block, where it is disabled by default: ```yaml cluster: discovery: enabled: true registries: - service: - disabled: true + kubernetes: + disabled: false ``` -[Disabling all registries](#what-changes-when-discovery-is-disabled) effectively disables member discovery entirely. - -### Kubernetes registry +Because the legacy `.cluster.discovery` block and `DiscoveryServiceConfig` documents are mutually exclusive, a configuration that uses this block must also configure the service registry there (under `registries.service`) rather than with a `DiscoveryServiceConfig` document. The Kubernetes registry stores discovery data as annotations on Kubernetes `Node` resources: @@ -90,9 +98,32 @@ The discovery service is aware of the client version, cluster ID, number of affi Nodes must be able to reach the discovery service on TCP port 443. For organisations that require it, the discovery service may be self-hosted under a commercial license and [downloaded from GitHub](https://github.com/siderolabs/discovery-service). +The service registry is configured with a `DiscoveryServiceConfig` document. To use a self-hosted discovery service, append the following document to the machine configuration with the `endpoint` set to your own instance: + +```yaml +apiVersion: v1alpha1 +kind: DiscoveryServiceConfig +name: self-hosted +endpoint: https://discovery.example.com/ +``` + +The `name` must be unique, and the `endpoint` must use the `https://` or `grpc://` scheme. For a highly available setup, append additional `DiscoveryServiceConfig` documents, each with a unique `name` and `endpoint`. Nodes publish to and read from every configured discovery service, so the cluster continues to discover peers as long as at least one of them is reachable: + +```yaml +apiVersion: v1alpha1 +kind: DiscoveryServiceConfig +name: discovery-1 +endpoint: https://discovery.example.com/ +--- +apiVersion: v1alpha1 +kind: DiscoveryServiceConfig +name: discovery-2 +endpoint: https://discovery-backup.example.com/ +``` + ## What changes when discovery is disabled -Talos can operate with discovery disabled, but this affects several features and behaviours: +Discovery is disabled by removing all `DiscoveryServiceConfig` documents from the machine configuration (and not configuring the deprecated Kubernetes registry). Talos can operate with discovery disabled, but this affects several features and behaviours: - [KubeSpan](../../networking/kubespan) and KubePrism require discovery and do not function correctly without it. - Initial cluster bootstrap and recovery may take longer, as peer and control plane endpoints are not available from discovery. diff --git a/public/talos/v1.14/networking/kubespan.mdx b/public/talos/v1.14/networking/kubespan.mdx index a4d0e99a..271d1e6a 100644 --- a/public/talos/v1.14/networking/kubespan.mdx +++ b/public/talos/v1.14/networking/kubespan.mdx @@ -93,15 +93,13 @@ For advanced eBPF features without these constraints, use Cilium's built-in Wire To enable KubeSpan for a new cluster, we can use the `--with-kubespan` flag in `talosctl gen config`. This will enable peer discovery and KubeSpan. +The `DiscoveryServiceConfig` document enables the discovery service used for cluster member discovery. The deprecated Kubernetes registry is problematic with KubeSpan, when the control plane endpoint is routable itself via KubeSpan, and is left unconfigured. + ```yaml -cluster: - discovery: - enabled: true - # Configure registries used for cluster member discovery. - registries: - kubernetes: # Kubernetes registry is problematic with KubeSpan, if the control plane endpoint is routable itself via KubeSpan. - disabled: true - service: {} +apiVersion: v1alpha1 +kind: DiscoveryServiceConfig +name: default +endpoint: https://discovery.talos.dev/ --- apiVersion: v1alpha1 kind: KubeSpanConfig @@ -113,12 +111,13 @@ enabled: true # Enable the KubeSpan feature. ### Enabling for an existing cluster -In order to enable KubeSpan on an existing cluster, add `KubeSpanConfig` and `discovery` settings in the machine config for each machine in the cluster (`discovery` is enabled by default): +In order to enable KubeSpan on an existing cluster, add `KubeSpanConfig` and `DiscoveryServiceConfig` documents to the machine config for each machine in the cluster (the discovery service is enabled by default): ```yaml -cluster: - discovery: - enabled: true +apiVersion: v1alpha1 +kind: DiscoveryServiceConfig +name: default +endpoint: https://discovery.talos.dev/ --- apiVersion: v1alpha1 kind: KubeSpanConfig diff --git a/public/talos/v1.14/reference/configuration/network/kubespanconfig.mdx b/public/talos/v1.14/reference/configuration/network/kubespanconfig.mdx index 528ef309..7313c7aa 100644 --- a/public/talos/v1.14/reference/configuration/network/kubespanconfig.mdx +++ b/public/talos/v1.14/reference/configuration/network/kubespanconfig.mdx @@ -52,7 +52,7 @@ filters: `enabled` bool - Enable the KubeSpan feature.
Cluster discovery should be enabled with cluster.discovery.enabled for KubeSpan to be enabled. + Enable the KubeSpan feature.
The discovery service should be enabled with a DiscoveryServiceConfig document for KubeSpan to be enabled.