[Help] step ca config error -> "ssh certificate not found: please run step ssh login <identity>"

[mf-in-mun]

Hello,

I'm using smallstep step ca for a couple of months, started with an automated SSL provisioning, set up a Borg Backup remote server with automated SSH certificates provisioning and now I want to move a step further.
I want to change my SSH key handling using step ca and an OIDC provider Zitadel.

I'm working mainly with this blog https://smallstep.com/blog/diy-single-sign-on-for-ssh/

I've setup a new oidc privisioner and running

step ssh login my.name@my.domain.tld --provisioner=my-provisioner

works.
I've got a SSH certificate, with an entry in the ssh-agent.
ssh-add -L shows that key, step ssh list shows that exact entry and step ssh list --raw | step ssh inspect seems to be fine.

But when I try that command

step ssh config

for configuration I always got an error

ssh certificate not found: please run `step ssh login <identity>`

Searching here the only thing I found was a code snippet from the project:

		// Force a user to have a username
		if _, ok := data["User"]; !ok {
			return errors.New("ssh certificate not found: please run `step ssh login <identity>`")
		}
	}

I'm sorry, but that's nothing which helped me.

I've tried this on Win 11, Powershell 7.5.3 and on Fedora 42, ghostty. On both I got that error.

Please, may I ask for some help?
I'm quite sure that I'm missing some configuration.
Hopefully somebody can point me into the right direction.

BR SMF

System: Win 11 24h2, update on Tuesday, Smallstep CLI/0.28.7 installed via winget.
Fedora 42 updated today, Smallstep CLI 0.28.7 installed via the Smallstep repo.
step ca, containerized using Podman, image based on Alpinelinux 3.22.1, running on an Almalinux 10 server

[erikboberg]

I have the exact same problem. I followed the same blog post and everything works until I try the step ssh config. I get the same error message "ssh certificate not found: please run step ssh login <identity>". The output from ssh-add -L and step ssh list commands looks fine.

[mf-in-mun]

@erikboberg

I've given up trying this step ssh config.
When I use this command, a few configuration files with strange permissions are created.
I can't get that right.

I've created manually a ssh configuration.

1. Init step

step ca bootstrap --ca-url=https://srv-ca.my.domain.tld:20000 --fingerprint [Fingerprint of my root certificate]

2. Creating a .ssh config file

Add-Content -Path ~\.ssh\known_hosts -Value("@cert-authority * " + (step ssh config --host --roots))

3. Creating a certificate

step ssh login my.user@my.domain.tld --provisioner=[my provisioner]

4. Login via ssh

ssh my.user@my.domain.tld

Step 3 must be repeated every time the certificate becomes invalidatet!
Once in the morning, maybe after rebooting...
For me, it's a minor inconvenience.

Double check your user name settings. Check the principal your are using.

ssh-add -L | step ssh inspect

BR SMF

[erikboberg]

@mf-in-mun
Thanks so much for the reply!
I checked ssh-add -L | step ssh inspect and the principal is empty. Principals: (none).
Sorry I am new to this, but perhaps my username is not correctly recognized? The principals should contain my user name and email, right? (as seen in the initial example here) https://smallstep.com/blog/clever-uses-of-ssh-certificate-templates/

Perhaps my configuration is not correct on the OIDC provider? I am using my self-hosted authelia instance instead of google as in the example from the blog.

[mf-in-mun]

@erikboberg

I'm using a self hosted Zitadel instance. I have no knowledge on Authelia.

Be prepard for a steep learning curve.

Please do not expect a ready to use copy and paste approach.

1. Double checking your configuration is always a good idea.
2. In Zitadel I've had to enable some enhanced user information.
3. Username and useremail isn't necessarily transfered.
4. Double check your scope settings.
5. Double check in general your user settings in Authelia.
6. Double check your step provisioner.
   Mine is looking like that:

step ca provisioner add "[provisioner name" \
    --type=oidc \
    --ssh \
    --client-id="XXXXXXXXXXXXXXXX" \
    --client-secret="" \
    --configuration-endpoint="https://srv-zitadel.my.domain.tld/.well-known/openid-configuration" \
    --listen-address="127.0.0.1:20001" \
    --domain="my.domain.tld" \
    --domain="domain.tld"

Some links I find helpful:
https://smallstep.com/blog/diy-single-sign-on-for-ssh/
https://smallstep.com/docs/step-ca/provisioners/#oauthoidc-single-sign-on
https://smallstep.com/docs/tutorials/keycloak-oidc-provisioner/
https://gist.github.com/tashian/244fc69ccb7ceec433c7811e91cbf0b7
https://gist.github.com/tashian/fde43668cbf6e3227fb13ef51db650b8

BR SMF

[erikboberg]

Many thanks!
I had to dig around, I was missing a claims_policy in my OIDC client configuration so my email address was not sent.
Also the step ssh config is a bit wierd, it changed files mainly in ~/.step but it insisted on placing the "includes" file in /etc/step-ca instead. I had to move it and set ownership to my regular user.

Finally, the step ssh config command did not add the --provisioner flag to the ~/.step/ssh/config. I had to add it in manually to use my OIDC provisioner.

Now it actually works for me it seems.
Cool!

[mf-in-mun]

@erikboberg

Happy to hear it's working.

Let's close this issue marking your reply as an answer.

BR SMF

[erikboberg]

Sorry, had one more small issue, see below, since you seem to know how this works very well, I just wanted to ask you about it too before closing. Thank you again!

[erikboberg]

The only thing that still does not work is that I need to still verify the authenticity of the host the first time I connect to it.
Do you know where I should start looking for why? I did not use the Amazon AWS provisioner from the blog post but instead the workaround further down using X5C and ACME (already had ACME set up):

step ca certificate internal.smallstep.com internal.crt internal.key
step ssh certificate --host --x5c-cert internal.crt --x5c-key internal.key internal.smallstep.com internal

Perhaps that is the cause?

[mf-in-mun]

@erikboberg

You must ensure that your host provides it's own certificate.

I'm not working with the ACME protocoll. I'm using this for some servers without a http-server listening on port 80. So I'm using the step-cli software.

Generate a ssh key:

rm -f /etc/ssh/ssh_host_ecdsa_key # Remove an old one if existing
/usr/bin/ssh-keygen -t ecdsa -f /etc/ssh/ssh_host_ecdsa_key -N '' # No  Passphrase

Then the key must be signed

/usr/bin/step ssh certificate $(hostname) /etc/ssh/ssh_host_ecdsa_key.pub --host --sign --not-after="24h02m" --force --password-file="[your password file" --provisioner=[your provisioner]

That flag --not-after must be inline with your provisioner; typically 24h. But...

Next your authorized_keys file must be changed:

echo "@cert-authority $(hostname) $(/usr/bin/step ssh config --roots)" > /etc/ssh/authorized_keys

Check your SSH daemon configuration very, very carefully.
Otherwise you will be locked.
Restart your sshd.

Again this is some serious stuff.

[erikboberg]

Yes serious stuff, I like the challenge!
I ended up kind of using the method described in the blog:

sudo step ca certificate $HOSTNAME internal.crt internal.key \
    --ca-url https://ca-url:443 \
    --root path/to/root_ca.crt \
    --provisioner acme

sudo step ssh certificate --host \
    --x5c-cert internal.crt \
    --x5c-key internal.key $HOSTNAME ssh_host_ecdsa_key \
    --ca-url https://ca-url:443 \
    --root /path/to/root_ca.crt \
    --insecure --no-password

This worked!
I now no longer see the message about having to verify the authenticity of the host when connecting.
Great!

[mf-in-mun]

@erikboberg

Happy to hear it's working.

For some further checking you can set debug loglevel LogLevel DEBUG3 for your SSH daemon.
And you can set the verbosity of the ssh command to -vvv.

You will get a lot of information and you will see which key respectively which certficate is used.

[erikboberg]

Thanks again! I actually found another bug in my setup today. The host script (https://gist.github.com/tashian/fde43668cbf6e3227fb13ef51db650b8) sets automatic renewal of the host ssh certificate weekly (/etc/cron.weekly/rotate-ssh-certificate), but my acme has the default 24-hour expiration of certificates. I ended up prolonging the expiry in the ca.json to 48 hours and set the renewal script to run daily. There is probably a more elegant solution, but if it works I am ok with it.

[mf-in-mun]

@erikboberg

Well, you need some sort of trigger to create or renew your certificate(s). Right?

For now I'm working with 2 triggers:

• When the host is booting or the container pod is starting.
• When a timer triggers a renewal.

And I have decided to renew my certificates (SSL and SSH) every day. Which is the default setting for a step ca provider. This is a pragmatic solution that meets the demand for short-lived certificates on the one hand and the demand for stable operation on the other.
I create always a new certificate (key and certificate).
I don't care about a revocation list.

I'm using a Systemd service during boot time and a Systemd timer with it's service for renewal.

Tuning this to a stable system needs some time.

[erikboberg]

Great, it seems I am using a sensible strategy then by renewing certs once per day. The difference between your setup and mine seems to be that I renew the cert step ssh renew ssh_host_ecdsa_key-cert.pub ssh_host_ecdsa_key --force but I do not create a new key every time. Is there a downside to this approach compared to creating a new key and certificate every day?

I use a cron job that gets run every day instead of a systemd timer.

[mf-in-mun]

@erikboberg

The idea behind getting a new certificate AND a new key is, to get rid of everything which could be silently compromized.
Let's say it's the same idea as using an OTP (one time password).

Using systemd or crond is a technical implementation detail.
I like monitoring my services using journalctl and systemctl.
But hey, there are many roads to L.A.