Howto guide for Talos secret rotation

[james-callahan]

The machine config has several secrets in it: for each secret, there should be a documented procedure for rotating it (preferably without cluster downtime):

• cluster.secret
• secret.bootstraptoken
• secret.secretboxencryptionsecret
• trustdinfo.token
• certs.etcd.key (and .crt would need to be updated to match)
• certs.k8s.key (and .crt would need to be updated to match)
• certs.k8saggregator.key (and .crt would need to be updated to match)
• certs.k8sserviceaccount.key (also at least for our deployment, any change here would need to be synchronised with AWS federation)
• certs.os.key

An example proceduce might be as simple as:

1. Generate a new token
2. Set the token in your control plane's machineconfig/userdata
3. At this point, nodes using the old value will be unable to join the cluster
4. Set the token in your config for new nodes

Or perhaps some way of generating a second secret; installing it as an alternative; rolling all nodes; and then removing the old secret.

[smira]

You can rotate any of that in a non-graceful way.

Talos and Kubernetes API CAs support graceful rotation today.

[james-callahan]

You can rotate any of that in a non-graceful way.

What do you mean by non-graceful?

If that implies a cluster outage then that seems to be a non-starter;
e.g. I can imagine a non-graceful rotation of the etcd CA would result in nodes being unable to form consensus for some brief period

Talos and Kubernetes API CAs support graceful rotation today.

That's good! From my list above that would take care of certs.os.key and certs.k8s.key right?

[smira]

If that implies a cluster outage then that seems to be a non-starter;
e.g. I can imagine a non-graceful rotation of the etcd CA would result in nodes being unable to form consensus for some brief period

Yes, that's the state of things.

Kubernetes and Talos CAs are the only two secrets which might need to be rotated e.g. if talosconfig or kubeconfig leaks out, it's hard to imagine same for the etcd CA.

[james-callahan]

Kubernetes and Talos CAs are the only two secrets which might need to be rotated e.g. if talosconfig or kubeconfig leaks out, it's hard to imagine same for the etcd CA.

Sure etcd is a bit rare-er; but I assume the secret exists for a reason?

I would consider certs.k8sserviceaccount.key to be quite critical for our use case: it's used to establish trust between our cluster and external authentication systems (such as AWS via IRSA): how would I rotate that?

The cluster secret and bootstrap token: wouldn't knowing them allow anyone to join a node to the cluster?

[smira]

See my answer above - every secret can be rotated, and Talos will propagate the change down, but the question of gracefulness requires a separate investigation for each one of these.

[JacobMillward]

Is there a guide anywhere on how to do this non-graceful possibly-has-downtime rotation of these certs?