Flux source-controller suddenly rejecting GPG signatures on commits?

[johanneskastl]

Dear all,

not sure what has changed where, but suddenly the source-controller is rejecting commits due to GPG signatures.

{"level":"error","ts":"2025-06-08T06:14:41.662Z","msg":"Reconciler error","controller":"gitrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"GitRepository","GitRepository":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"b3f97afb-d6df-41b0-847a-688d3c119e99","error":"signature verification of commit '6fd686b65f300cc2a3f0eb7f83958f5c5465559a' failed: unable to verify Git commit: unable to verify payload with any of the given key rings"}

• There was no update to the flux components from a manifest perspective, the installation was using the components from 2.5.1. An update to 2.6.1 did not improve this.
• The secret containing the GPG public keys was unchanged
• Both git locally and the WebUIs (Gitlab or Codeberg) show valid and verified GPG signatures on the commits
• The signatures are from one of the keys in the public key secret

Funnily enough, this has happened on two different clusters, both running 2.5.1. One of the was updated to 2.6.1.

Does anyone have any pointers on where I could look?

Kind Regards,
Johannes

[matheuscscp]

Ok so just to be sure, this started happening before upgrading to 2.6.x?

[johanneskastl]

Yes, I made changes to things unrelated to flux itself, i.e. actual workloads. Then those were never applied and I found this in the logs.

After upgrading to 2.6.1 there was another issue (#5385 ) first, after solving that issue, the above error is still present.

[johanneskastl]

Normally I would mistrust the GPG signatures on those commits, as it happened on two clusters.
But I found no trace that those are broken.

[johanneskastl]

Unfortunately this is still an issue. Both on 2.5.1 and 2.6.1.

Any ideas how to debug this?

[johanneskastl]

This is strange. I am using jj (https://jj-vcs.github.io/jj/latest/) currently as an alternative to git. It is using the git backend and is git compatible.

I had another cluster where flux was still running fine and GPG validation succeeded. I made a no-change commit using jj, signing the commit with the same GPG key and promptly the commit verification in the source-controller pod failed.

I force-pushed the branch to the previous commit and made a commit with git, using the same key. And this time the commit was accepted.

Funny thing is: Git shows both commits as having a valid signature.
(Sorry for the German output, but you get the idea).

commit f8b83e7eec1da7a9e3d88ed1a82618f205d11a40
gpg: Signatur vom Do 19 Jun 2025 08:05:37 CEST
gpg:                mittels EDDSA-Schlüssel C46E9A9CE29C497AAD9A7435D7E71ECBC2CA29E1
gpg: Korrekte Signatur von "Johannes Kastl <[email protected]>" [ultimativ]
Haupt-Fingerabdruck  = C46E 9A9C E29C 497A AD9A  7435 D7E7 1ECB C2CA 29E1
Author: Johannes Kastl <[email protected]>
Date:   Thu Jun 19 08:05:09 2025 +0200

commit f6ed705bd34d41960d8cc53e9e8273d7294519df
gpg: Signatur vom Do 19 Jun 2025 08:08:51 CEST
gpg:                mittels EDDSA-Schlüssel C46E9A9CE29C497AAD9A7435D7E71ECBC2CA29E1
gpg: Korrekte Signatur von "Johannes Kastl <[email protected]>" [ultimativ]
Haupt-Fingerabdruck  = C46E 9A9C E29C 497A AD9A  7435 D7E7 1ECB C2CA 29E1
Author: Johannes Kastl <[email protected]>
Date:   Thu Jun 19 08:08:51 2025 +0200

Strange. Especially as all other tooling so far (GitHub, GitLab, Forgejo, ...) also shows the commits as properly signed.

[matheuscscp]

You error is coming from this code (where openpgp is github.com/ProtonMail/go-crypto/openpgp version v1.2.0):

func verifySignature(sig string, payload []byte, keyRings ...string) (string, error) {
	if sig == "" {
		return "", fmt.Errorf("unable to verify payload as the provided signature is empty")
	}

	for _, r := range keyRings {
		reader := strings.NewReader(r)
		keyring, err := openpgp.ReadArmoredKeyRing(reader)
		if err != nil {
			return "", fmt.Errorf("unable to read armored key ring: %w", err)
		}
		signer, err := openpgp.CheckArmoredDetachedSignature(keyring, bytes.NewBuffer(payload), bytes.NewBufferString(sig), nil)
		if err == nil {
			return signer.PrimaryKey.KeyIdString(), nil
		}
	}
	return "", fmt.Errorf("unable to verify payload with any of the given key rings")
}

Maybe if you execute this with your inputs we could get some insights? It's probably something to be addressed in the upstream library https://github.com/ProtonMail/go-crypto