Cluster creation fails with certificate signed by unknown authority

[aossama]

Hello All,

I am setting up an on-prem Omni instance in an air-gapped (disconnected) environment. Omni is hosted on Talos based K8s cluster which has been provisioned manually. The setup uses self-signed CA for all the components (image factory and omni).

Below is the snippet used to deploy Omni:

... output omitted ...
      containers:
        - name: omni
          args:
            - --account-id=XXX
            - --advertised-api-url=https://omni.example.com/
            - --advertised-kubernetes-proxy-url=https://kube.omni.example.com/
            - --auth-auth0-enabled=false
            - --auth-saml-enabled=true
            - --auth-saml-url=https://auth.example.com/realms/omni/protocol/saml/descriptor
            - --bind-addr=0.0.0.0:8080
            - --cert=/etc/omni/ssl/omni-ingress-tls/tls.crt
            - --event-sink-port=8091
            - --image-factory-address=https://factory.example.com
            - --k8s-proxy-bind-addr=0.0.0.0:8095
            - --key=/etc/omni/ssl/omni-ingress-tls/tls.key
            - --kubernetes-registry=r.example.com/siderolabs/kubelet
            - --log-server-port=8092
            - --machine-api-advertised-url=https://api.omni.example.com/
            - --machine-api-bind-addr=0.0.0.0:8090
            - --machine-api-cert=/etc/omni/ssl/api-omni-ingress-tls/tls.crt
            - --machine-api-key=/etc/omni/ssl/api-omni-ingress-tls/tls.key
            - --name=onprem-omni
            - --private-key-source=file:///etc/omni/omni.asc
            - '--registry-mirror="factory.talos.dev=r.example.com,docker.io=r.example.com,gcr.io=r.example.com,ghcr.io=r.example.com,registry.k8s.io=r.example.com,quay.io=r.example.com,k8s.gcr.io=r.example.com"'
            - --siderolink-api-advertised-url=https://api.omni.example.com/
            - --siderolink-use-grpc-tunnel=false
            - --siderolink-wireguard-advertised-addr=wireguard.omni.example.com:50180
            - --siderolink-wireguard-bind-addr=0.0.0.0:50180
            - --talos-installer-registry=r.example.com/siderolabs/installer
          image: ghcr.io/siderolabs/omni:v1.2.1
          imagePullPolicy: IfNotPresent
          ports:
            - name: omni
              protocol: TCP
              containerPort: 8080
            - name: siderolink
              protocol: TCP
              containerPort: 8090
            - name: event-sink
              protocol: TCP
              containerPort: 8091
            - name: log-server
              protocol: TCP
              containerPort: 8092
            - name: k8s-proxy
              protocol: TCP
              containerPort: 8095
            - name: wireguard
              containerPort: 50180
              protocol: UDP
          volumeMounts:
            - mountPath: /etc/ssl/certs/user-ca-bundle.crt
              name: user-ca-bundle
              subPath: tls.crt
              readOnly: true
            - mountPath: /etc/omni/ssl/omni-ingress-tls
              name: omni-ingress-tls
              readOnly: true
            - mountPath: /etc/omni/ssl/api-omni-ingress-tls
              name: api-omni-ingress-tls
              readOnly: true
            - name: omni-certificates
              mountPath: /etc/omni/omni.asc
              subPath: omni.asc
              readOnly: true
      restartPolicy: Always
      volumes:
        - name: omni-certificates
          secret:
            secretName: omni-certificates
        - name: api-omni-ingress-tls
          secret:
            secretName: api-omni-ingress-tls
        - name: omni-ingress-tls
          secret:
            secretName: omni-ingress-tls
        - name: user-ca-bundle
          secret:
            secretName: ca-bundle

Omni starts properly, and the integration with the image-factory works perfectly fine. Omni is able to communicate with image-factory and retrieve all the required data.

Now comes the point where I need to use an image instead of the generated ISO. In addition, I don't want to manage my own custom secureboot certificates. So I rely on the disk image provided by factory.talos.dev, where I am using nocloud-amd64-secureboot.raw.xz for spinning up the virtual machines.

Below is the sequence followed;

1. Download nocloud-amd64-secureboot.raw.xz from factory.talos.dev
2. Create a VM based on the raw disk
3. Boot the machine in maintenance mode
4. Download Machine Join Config from Omni UI
5. Append TrustedRootsConfig, which has all the CAs used to trust omni endpoints. Basically doing: cat trusted-roots-config.yaml >> machine-config.yaml
6. Apply the machine-config.yaml, using talosctl apply-config -n 192.168.122.191 --insecure --file machine-config.yaml

At this point the machine establish wireguard link properly and the machine joins Omni, and I can see it active and ready to be used.

The problem occurs when creating a new cluster. The process gets stuck, and the controllers start emitting error like;

{"level":"error","ts":1760631990.3361323,"caller":"qruntime/qruntime.go:295","msg":"reconcile failed","component":"omni_runtime","controller":"ClusterBootstrapStatusController","namespace":"default","type":"ClusterStatuses.omni.sidero.dev","id":"talos-a4f0","job":"reconcile","error":"error bootstrapping cluster 'talos-a4f0': rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority\"","interval":72.720895685,"busy":0.062465278}
{"level":"error","ts":1760631828.0076385,"caller":"qruntime/qruntime.go:295","msg":"reconcile failed","component":"omni_runtime","controller":"ControlPlaneStatusController","namespace":"default","type":"MachineSets.omni.sidero.dev","id":"talos-a4f0-control-planes","job":"reconcile","error":"Etcd status check failed: rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority\"","interval":88.074408353,"busy":0.090233003}
{"level":"error","ts":1760631863.5084,"caller":"qruntime/qruntime.go:295","msg":"reconcile failed","component":"omni_runtime","controller":"MachineSetEtcdAuditController","namespace":"default","type":"MachineSets.omni.sidero.dev","id":"talos-a4f0-control-planes","job":"reconcile","error":"error listing etcd members for cluster \"talos-a4f0\": rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority\"","interval":45.568168676,"busy":0.00198196}

Things I've tried during troubleshooting;

1. Using cluster's talosconfig to communicate with machine's apid result in same error

   $ talosctl --talosconfig talos-a4f0-talosconfig.yaml -n 192.168.122.191 get members
   rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
   

2. Tried connecting to the apid endpoint directly from omni's pod to perform troubleshooting over the established wireguard link,

   root@omni-76cd545b65-k4pq5:/# curl -k -v https://[fdae:41e4:649b:9303:ffa6:d54c:39cf:f67c]:50000
   ... output omitted ...
   * TLSv1.3 (OUT), TLS handshake, Finished (20):
   * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / X25519MLKEM768 / ED25519
   * ALPN: server accepted h2
   * Server certificate:
   *  subject: CN=maintenance-service.talos.dev
   *  start date: Oct 16 07:55:33 2025 GMT
   *  expire date: Oct 17 07:55:33 2025 GMT
   *  issuer: 
   *  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
   *   Certificate level 0: Public key type ED25519 (256/128 Bits/secBits), signed using ED25519
   * Connected to fdae:41e4:649b:9303:ffa6:d54c:39cf:f67c (fdae:41e4:649b:9303:ffa6:d54c:39cf:f67c) port 50000
   * using HTTP/2
   * [HTTP/2] [1] OPENED stream for https://[fdae:41e4:649b:9303:ffa6:d54c:39cf:f67c]:50000/
   * [HTTP/2] [1] [:method: GET]
   

3. Tried using talosctl directly from within omni's pod and ended with the same result;

   root@omni-76cd545b65-k4pq5:/# ./talosctl --talosconfig talos-a4f0-talosconfig.yaml -n [fdae:41e4:649b:9303:ffa6:d54c:39cf:f67c]:50000 get members
   rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
   

4. Using the insecure flag works

   root@omni-76cd545b65-k4pq5:/# ./talosctl --talosconfig talos-a4f0-talosconfig.yaml -n [fdae:41e4:649b:9303:ffa6:d54c:39cf:f67c]:50000 get securitystate -insecure
   WARNING: : server version 1.11.1 is older than client version 1.11.3
   NODE   NAMESPACE   TYPE            ID              VERSION   SECUREBOOT   UKISIGNINGKEYFINGERPRINT   PCRSIGNINGKEYFINGERPRINT                                                                          SELINUXSTATE
          runtime     SecurityState   securitystate   1         true                                    9C:42:05:91:48:E1:57:A0:30:F5:ED:C5:1B:D4:96:7A:2A:3B:1B:C6:4C:DD:48:94:1A:3E:CE:3C:3F:DC:03:2F   enabled, permissive
   
   

I cannot figure out what happens wrong in the process, and I might be doing something wrong. So any help is really appreciated.

[steverfrancis]

Hi - can you let us know what customer or trialing company you are with so we can get you the right support?

[aossama]

Hi, actually this is a home lab not with a customer or a company.

[steverfrancis]

Ah, it's rare to see airgapped at home. :-)
@rothgar can you see if our docs are missing something?

[rothgar]

It looks like you get the machine to join Omni via a join config and trusted certificate chain but when you create a cluster Omni is removing the TrustedRootsConfig patch that was applied with the Omni connection config.

I would try downloading/booting an image with talos.config.inline set with the custom CA you need instead of applying it as a patch to a vanilla Talos image. You can see how to do it here. https://docs.siderolabs.com/talos/v1.11/reference/kernel#talos-config-early-and-talos-config-inline

Let us know if that works.

[aossama]

@rothgar Thank you so much for pointing me to the right direction. Indeed loading the CA during VM boot made it work.

However, I didn't try explicity using talos.config.inline, instead I modified the VM's user-data source and put the machine-config of Omni and the TrustedRootsConfig. This made Talos boot up and read the user-data and automatically join the machine to Omni.

I am trying to avoid building or customizing anything that could lead to requiring manual intervention.

Repeating my thanks to you and the amazing team at SideroLabs.