Talos cluster credentials expired, can't access cluster

[bcspragu]

Hi there,

I can no longer connect to my Talos cluster:

> kubectl get nodes
E1006 19:24:27.025594 1695958 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E1006 19:24:27.030988 1695958 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
[ ... repeated a few more times ... ]
error: You must be logged in to the server (the server has asked for the client to provide credentials)

Checking my ~/.kube/config, my users.user.client-certificate-data has expired. E.g. echo '<client certificate data>' | base64 -d | openssl x509 -text -noout contains:

        Validity
            Not Before: Oct  6 01:23:22 2023 GMT
            Not After : Oct  5 01:23:32 2024 GMT

So that explains that problem, I just don't know how to fix it. I tried to generate a new config with:

> talosctl kubeconfig -n ... -e ... --talosconfig ./talosconfig
error copying: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: remote error: tls: bad certificate"

Looking in my talosconfig, the server certificates look good until 2033, so I'm not sure what's actually going on here. Any ideas?

This is just a single-node cluster in my home, so I have physical access to the machine if need be.

[smira]

Kubernetes config can always be refreshed with talosctl kubeconfig, but you need Talos API access for that.

I guess both expired for you, you can check your talosconfig with talosctl --talosconfig=./talosconfig config info.

If your client-side talosconfig expired, the only option left is to recover it via secrets.yaml or machine configuration: https://www.talos.dev/v1.8/talos-guides/howto/cert-management/#generating-new-client-configuration

[bcspragu]

Thanks, that worked perfectly! For anyone else stumbling across this thread, here's what I did:

# Extract the CA cert + key from the control plane config
yq -r .machine.ca.crt controlplane.yaml | base64 -d > ca.crt
yq -r .machine.ca.key controlplane.yaml | base64 -d > ca.key

# Generate fresh credentials
talosctl gen key --name admin
talosctl gen csr --key admin.key --ip 127.0.0.1
talosctl gen crt --ca ca --csr admin.csr --name admin

# Update the `talosconfig` with the new values
# I generated them with:
# For ca: base64 -w0 ca.crt
# For crt: base64 -w0 admin.crt
# For key: base64 -w0 admin.key
vim talosconfig

# Finally, update your `~/.kube/config` with the new creds
talosctl kubeconfig -n ... -e ... --talosconfig ./talosconfig

Not sure if the docs should be updated or my version of yq is just weird, but the yq eval ... invocations didn't work for me:

> yq eval '.machine.ca.crt' controlplane.yaml | base64 -d > ca.crt
usage: yq [-h] [--yaml-output] [--yaml-roundtrip]
          [--yaml-output-grammar-version {1.1,1.2}] [--width WIDTH]
          [--indentless-lists] [--explicit-start] [--explicit-end]
          [--in-place] [--version]
          [jq_filter] [files ...]
yq: error: argument files: can't open '.machine.ca.crt': [Errno 2] No such file or directory: '.machine.ca.crt'

[daemonp]

Thank you for posting this @bcspragu and @smira for the answer

It's also worth mentioning that talosctl gen csr --key admin.key --ip 127.0.0.1 defaults to 24 hours and the --hours flag can be used to set for longer period.

This is the oneliner I'm using for updates:

yq -r .machine.ca.crt controlplane.yaml | base64 -d > ca.crt && \
yq -r .machine.ca.key controlplane.yaml | base64 -d > ca.key && \
talosctl gen key --name admin && \
talosctl gen csr --key admin.key --ip 127.0.0.1 && \
talosctl gen crt --ca ca.crt --csr admin.csr --name admin --days 8760 && \
yq eval '.contexts.home.ca = "'"$(base64 -w0 ca.crt)"'" | .contexts.home.crt = "'"$(base64 -w0 admin.crt)"'" | .contexts.home.key = "'"$(base64 -w0 admin.key)"'"' -i talosconfig

[bcspragu]

It's also worth mentioning that talosctl gen csr --key admin.key --ip 127.0.0.1 defaults to 24 hours and the --hours flag can be used to set for longer period.

I should really know more about how the PKI works in Talos/K8s, but it looks like even though the admin.crt is only valid for a day, the generated talosctl kubeconfig is valid for one year.

[dpoon]

The solution from @daemonp works, with a tweak. The command

talosctl gen crt --ca ca.crt --csr admin.csr --name admin --days 8760

… should be:

talosctl gen crt --ca ca --csr admin.csr --name admin --hours 8760

Also note that on a macOS client, base64 -w0 should instead be base64 -b0.

[markussiebert]

Thanks for your solution, but I get 'error decoding cert PEM' if I try to execute your fixed commands - any ideas?

[camrossi]

Thank you @bcspragu you saved me some major pain here!
Just one difference I had in my case is that the talosconfig didn't have the home section so just replaced the ca/crt/key and that fixed it. Adding home.crt etc... resulted in a non working config

context: turing
contexts:
  turing:
    endpoints:
      - E
    nodes:
      -  N
    ca:  XXX
    crt: XXX
    key: XXX

[markbaumgarten]

I am in a similar situation: Unable to use talosctl due to expired client certificates.

The above commands does not seem to work for me. After running the "oneliner" from @daemonp i get "failed to verify certificate: x509: certificate signed by unknown authority":

This is my command:

talosctl --talosconfig talosconfig -n 10.20.30.40 dmesg

The cluster was bootstrapped using the sidero controlplane docker based management cluster which is still running locally.

I am wondering if I might have to get the relevant controlplane.yaml, secrets.yaml etc from within this docker container?

The current controlplane.yaml files I have was extracted ...more than a year ago(I think) from the docker based sidero management container(I dont have access to the chat in slack which provided me a helpful answer a year or to ago)...

Can I use the "docker management cluster" in my docker container to fix my situation?

[smira]

If you have your CAPI cluster, you can get talosconfig secret from it

[markbaumgarten]

Would you mind elaborating?

…

On Fri, Aug 29, 2025 at 3:31 PM Andrey Smirnov ***@***.***> wrote: If you have your CAPI cluster, you can get talosconfig secret from it — Reply to this email directly, view it on GitHub <#9457 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAUZW2XYUZWRIPTS6FRAWKT3QBI25AVCNFSM6AAAAACFDZ45C2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTIMRVGYYTCOI> . You are receiving this because you commented.Message ID: ***@***.***>

[smira]

I'm not sure what I need to elaborate on. https://github.com/siderolabs/cluster-api-bootstrap-provider-talos/#retrieving-talosconfig