Best practice for true zero-downtime Kubernetes upgrades: Should we use --upgrade-kubelet false and manual kubelet upgrades?

[dedene]

Hi Talos team,

I'm reaching out to find some ideas on the best approach for achieving true zero-downtime Kubernetes upgrades with Talos when doing minor k8s upgrades.

We always experience a short downtime during a Kubernetes upgrade (i.e. from 1.30 to 1.31). Despite having Pod Disruption Budgets configured, all containers across nodes are restarted simultaneously, breaking our high availability. This appears to be normal and caused by the kubelet changing container spec hash values during the upgrade process?

I noticed the talosctl upgrade-k8s command has an --upgrade-kubelet flag (default: true) but I did not find much documentation on it. The Talos documentation also describes a manual process for upgrading the kubelet on individual nodes.

Would the following approach provide zero-downtime upgrades?

1. Run talosctl upgrade-k8s --upgrade-kubelet false to upgrade the control plane and other components
2. Manually upgrade the kubelet on each node one by one using the documented process
3. Wait until all containers are restarted before proceeding to the next node

Any guidance on this would be greatly appreciated, as maybe I am missing something.

Thanks a lot for your help!
Peter

[smira]

I have never seen containers being restarted on kubelet upgrade, I don't think it's expected.

I don't think kubelet can even modify pod spec (vs. status).

[dbackeus]

Upgrading to 1.31 does cause mass container restarts because of a change in the container spec hashing algorithm: kubernetes/kubernetes#129385

[dedene]

Hmm 🤔 I came across "All containers are restarted after upgrade, because the container spec hash value is changed." in https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/ and when inspecting our pods they all had an event Container <containername> definition changed, will be restarted so that's why I thought, maybe this is normal behavior.

Thanks for your quick answer, I'll do some more research and try to find where this is coming from or if I can reproduce this consistently.

[smira]

Talos is not kubeadm-based, so it doesn't directly apply to Talos.

[apfuzz]

@dedene I've made the same observations as you, specifically with regard to upgrading from 1.30 to 1.31 (but apparently not from 1.31 to 1.32). I will be testing the --upgrade-kubelet false option to see how that behaves then follow this procedure to upgrade kubelet manually after draining each node. I wasn't able to find any documentation that speaks to that flag otherwise.

[apfuzz]

I upgraded my test cluster using talosctl upgrade-k8s -n <control_plane_node_ip> --to $K8S_VERSION --upgrade-kubelet=false, which worked as expected.

automatically detected the lowest Kubernetes version 1.32.9
updating "kube-apiserver" to version "1.32.10"
updating "kube-controller-manager" to version "1.32.10"
updating "kube-scheduler" to version "1.32.10"
updating kube-proxy to version "1.32.10"
skipped updating kubelet

After that command finished, I drained each node and went to patch it using the patch command here. In my case, I have a multi-document machine config so the patch doesn't work.

JSON6902 patches are not supported for multi-document machine configuration

I modified the machine config manually with talosctl edit mc and that worked just fine (no reboot). After that, kubectl get node reported the updated version for each node.

I'll be using this process for upgrading my production cluster so I don't have to wait for a maintenance window. Would be interested to know if the container restarts going from 1.30.x to 1.31.x was some sort of anomaly as doing it all in one go (and no container restarts) would be preferred.

[dbackeus]

We encountered the dreaded "all containers restart" issue while running talosctl upgrade-k8s to upgrade from Kubernetes 1.30 to 1.31 our cluster yesterday.

The reason was due to kubelet changing how container specs are hashed which caused it to detect container changes even where there were none, see issue at: kubernetes/kubernetes#129385

The issue was closed as not planned since the Kubernetes documentation clearly states that kubelet upgrades are not expected to be graceful and should be executed on drained nodes only.

The talosctl upgrade-k8s documentation currently states:

This will automatically update the components needed to upgrade Kubernetes safely. Upgrading Kubernetes is non-disruptive to the cluster workloads.

But this is not necessarily true. According to ChatGPT the same hashing change issue has historically happened in Kubernetes versions 1.8 and 1.14 as well. 1.14 was released all the way back when Talos was in alpha stages, so presumably Talos deployments haven't had to deal with a non graceful kubelet upgrade path until version 1.31.

So while it is quite rare for kubelet upgrades to cause issues, it can happen. The documentation should be factual about this.