Cannot get cluster up and running after power outage.

[reitermarkus]

Hi,

after a recent power outage, I cannot get my cluster to work.

The cluster consists of 3 control plane nodes and 3 worker nodes. The control plane nodes were shut down gracefully with talosctl shutdown, the worker nodes were not.

Once power was back, I tried starting all nodes again, but somehow the nodes cannot connect to each other now. I then shut down all nodes again and then only started the control plane nodes for now.

etcd seems to be working for the control plane nodes:

$ talosctl etcd status
NODE        MEMBER             DB SIZE   IN USE           LEADER             RAFT INDEX   RAFT TERM   RAFT APPLIED INDEX   LEARNER   ERRORS
10.0.0.45   262004e315e77d11   296 MB    67 MB (22.79%)   262004e315e77d11   250179372    7965        250179372            false     
10.0.0.44   75b609d075a092ee   296 MB    67 MB (22.65%)   262004e315e77d11   250179372    7965        250179372            false     
10.0.0.41   2640a1220881c48f   294 MB    67 MB (22.68%)   262004e315e77d11   250179372    7965        250179372            false     

kubectl cannot connect to the cluster:

$ kubectl get nodes
E0712 21:37:55.896974    9058 memcache.go:265] couldn't get current server API group list: Get "https://10.0.0.40:6443/api?timeout=32s": dial tcp 10.0.0.40:6443: connect: connection refused
E0712 21:37:55.897328    9058 memcache.go:265] couldn't get current server API group list: Get "https://10.0.0.40:6443/api?timeout=32s": dial tcp 10.0.0.40:6443: connect: connection refused
E0712 21:37:55.898418    9058 memcache.go:265] couldn't get current server API group list: Get "https://10.0.0.40:6443/api?timeout=32s": dial tcp 10.0.0.40:6443: connect: connection refused
E0712 21:37:55.899546    9058 memcache.go:265] couldn't get current server API group list: Get "https://10.0.0.40:6443/api?timeout=32s": dial tcp 10.0.0.40:6443: connect: connection refused
E0712 21:37:55.900630    9058 memcache.go:265] couldn't get current server API group list: Get "https://10.0.0.40:6443/api?timeout=32s": dial tcp 10.0.0.40:6443: connect: connection refused
The connection to the server 10.0.0.40:6443 was refused - did you specify the right host or port?

The VIP seems to be working, i.e. ping 10.0.0.40 succeeds.

The control plane nodes show similar errors:

$ talosctl dmesg -n 10.0.0.44 -f
10.0.0.44: user: warning: [2023-07-12T19:38:40.508806429Z]: [talos] controller failed {"component": "controller-runtime", "controller": "k8s.NodeLabelsApplyController", "error": "1 error(s) occurred:\x5cn\x5cterror getting node: Get \x5c"https://localhost:6443/api/v1/nodes/talos-net1?timeout=30s\x5c": dial tcp [::1]:6443: connect: connection refused"}
10.0.0.44: user: warning: [2023-07-12T19:38:48.392209429Z]: [talos] controller failed {"component": "controller-runtime", "controller": "k8s.KubeletStaticPodController", "error": "error refreshing pod status: error fetching pod status: Get \x5c"https://127.0.0.1:10250/pods/?timeout=30s\x5c": remote error: tls: internal error"}
10.0.0.44: user: warning: [2023-07-12T19:39:03.711738429Z]: [talos] controller failed {"component": "controller-runtime", "controller": "k8s.KubeletStaticPodController", "error": "error refreshing pod status: error fetching pod status: Get \x5c"https://127.0.0.1:10250/pods/?timeout=30s\x5c": remote error: tls: internal error"}
10.0.0.44: user: warning: [2023-07-12T19:39:19.324283429Z]: [talos] controller failed {"component": "controller-runtime", "controller": "k8s.KubeletStaticPodController", "error": "error refreshing pod status: error fetching pod status: Get \x5c"https://127.0.0.1:10250/pods/?timeout=30s\x5c": remote error: tls: internal error"}
10.0.0.44: user: warning: [2023-07-12T19:39:29.106327429Z]: [talos] controller failed {"component": "controller-runtime", "controller": "k8s.NodeLabelsApplyController", "error": "1 error(s) occurred:\x5cn\x5cterror getting node: Get \x5c"https://localhost:6443/api/v1/nodes/talos-net1?timeout=30s\x5c": dial tcp [::1]:6443: connect: connection refused"}
10.0.0.44: user: warning: [2023-07-12T19:39:29.820869429Z]: [talos] kubernetes endpoint watch error {"component": "controller-runtime", "controller": "k8s.EndpointController", "error": "failed to list *v1.Endpoints: Get \x5c"https://10.0.0.40:6443/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0\x5c": dial tcp 10.0.0.40:6443: connect: connection refused"}
10.0.0.44: user: warning: [2023-07-12T19:39:34.729343429Z]: [talos] controller failed {"component": "controller-runtime", "controller": "k8s.KubeletStaticPodController", "error": "error refreshing pod status: error fetching pod status: Get \x5c"https://127.0.0.1:10250/pods/?timeout=30s\x5c": remote error: tls: internal error"}
10.0.0.44: user: warning: [2023-07-12T19:39:50.090978429Z]: [talos] controller failed {"component": "controller-runtime", "controller": "k8s.KubeletStaticPodController", "error": "error refreshing pod status: error fetching pod status: Get \x5c"https://127.0.0.1:10250/pods/?timeout=30s\x5c": remote error: tls: internal error"}
10.0.0.44: user: warning: [2023-07-12T19:39:56.113291429Z]: [talos] controller failed {"component": "controller-runtime", "controller": "k8s.ManifestApplyController", "error": "error creating mapping for object /v1/Namespace/kubelet-serving-cert-approver: Get \x5c"https://localhost:6443/api?timeout=32s\x5c": dial tcp [::1]:6443: connect: connection refused"}
10.0.0.44: user: warning: [2023-07-12T19:40:05.750056429Z]: [talos] controller failed {"component": "controller-runtime", "controller": "k8s.KubeletStaticPodController", "error": "error refreshing pod status: error fetching pod status: Get \x5c"https://127.0.0.1:10250/pods/?timeout=30s\x5c": remote error: tls: internal error"}

apid is running:

$ talosctl service apid status
NODE     10.0.0.44
ID       apid
STATE    Running
HEALTH   OK
EVENTS   [Running]: Health check successful (14m2s ago)
         [Running]: Started task apid (PID 1928) for container apid (14m2s ago)
         [Preparing]: Creating service runner (14m3s ago)
         [Preparing]: Running pre state (14m3s ago)
         [Waiting]: Waiting for service "containerd" to be "up" (14m3s ago)
         [Waiting]: Waiting for service "containerd" to be "up", api certificates (14m4s ago)
NODE     10.0.0.45
ID       apid
STATE    Running
HEALTH   OK
EVENTS   [Running]: Health check successful (14m1s ago)
         [Running]: Started task apid (PID 1928) for container apid (14m2s ago)
         [Preparing]: Creating service runner (14m2s ago)
         [Preparing]: Running pre state (14m2s ago)
         [Waiting]: Waiting for service "containerd" to be "up" (14m2s ago)
         [Waiting]: Waiting for service "containerd" to be "up", api certificates (14m3s ago)
NODE     10.0.0.41
ID       apid
STATE    Running
HEALTH   OK
EVENTS   [Running]: Health check successful (14m2s ago)
         [Running]: Started task apid (PID 2499) for container apid (14m2s ago)
         [Preparing]: Creating service runner (14m3s ago)
         [Preparing]: Running pre state (14m3s ago)
         [Waiting]: Waiting for service "containerd" to be "up", api certificates (14m4s ago)

[smira]

There are many resources available for you to help:

• talosctl -n <controlplane-ip> health
• https://www.talos.dev/v1.4/advanced/troubleshooting-control-plane/
• talosctl -n <ip1,ip2,ip3> support grabs logs from all components into a single .zip package