OCIRepository authentication required

[bozho]

Hi, I'm doing something wrong, but I cannot find what it is...

I have an OCIRepository for a package built from our private Github repo. It's referencing a secret created for a classic PAT using kubectl -n flux-system create secret docker-registry ghcr-auth ...., but it's failing to fetch the artifact digest.

kubectl describe for the resource (I've replaced our org and repo names below):

Name:         infra
Namespace:    kyverno
Labels:       resourceset.fluxcd.controlplane.io/name=infra
              resourceset.fluxcd.controlplane.io/namespace=flux-system
Annotations:  <none>
API Version:  source.toolkit.fluxcd.io/v1beta2
Kind:         OCIRepository
Metadata:
  Creation Timestamp:  2025-05-08T16:24:27Z
  Finalizers:
    finalizers.fluxcd.io
  Generation:        2
  Resource Version:  28210
  UID:               a0837ad3-75da-42a0-ad9d-aeae22649eb7
Spec:
  Interval:  5m
  Provider:  generic
  Ref:
    Tag:  latest
  Secret Ref:
    Name:   ghcr-auth
  Timeout:  60s
  URL:      oci://ghcr.io/myorg/myinfra/admission
Status:
  Conditions:
    Last Transition Time:  2025-05-08T16:50:14Z
    Message:               building artifact
    Observed Generation:   2
    Reason:                ProgressingWithRetry
    Status:                True
    Type:                  Reconciling
    Last Transition Time:  2025-05-08T16:50:14Z
    Message:               failed to determine artifact digest: GET https://ghcr.io/token?scope=repository%!A(MISSING)myorg%!F(MISSING)myinfra%!F(MISSING)admission%!A(MISSING)pull&service=ghcr.io: UNAUTHORIZED: authentication required
    Observed Generation:   2
    Reason:                OCIArtifactPullFailed
    Status:                False
    Type:                  Ready
    Last Transition Time:  2025-05-08T16:24:29Z
    Message:               failed to determine artifact digest: GET https://ghcr.io/token?scope=repository%3Amyorg%2Fmyinfra%2Fadmission%3Apull&service=ghcr.io: UNAUTHORIZED: authentication required
    Observed Generation:   2
    Reason:                OCIArtifactPullFailed
    Status:                True
    Type:                  FetchFailed
  Observed Generation:     -1
Events:
  Type     Reason                 Age                   From               Message
  ----     ------                 ----                  ----               -------
  Warning  OCIArtifactPullFailed  8m55s (x12 over 34m)  source-controller  failed to determine artifact digest: GET https://ghcr.io/token?scope=repository%!A(MISSING)myorg%!F(MISSING)myinfra%!F(MISSING)admission%!A(MISSING)pull&service=ghcr.io: UNAUTHORIZED: authentication required

The secret exists in the kyverno namespace, it was copied from the flux-system namespace. The top level "fleet" OCIRepository in the flux-system uses the same secret to access packages from our "fleet" repo (built from our "fleet" GitHub repository).

I've checked the contents of the secret, PAT is correct. When I use oras to fetch the manifest from oci://ghcr.io/myorg/myinfra/admission using the PAT, that works...

I'm curious about the (MISSING) strings appearing in the top error message...

kubectl describe for the "fleet" OCIRepository:

Name:         flux-system
Namespace:    flux-system
Labels:       app.kubernetes.io/instance=flux-system
              app.kubernetes.io/managed-by=flux-operator
              app.kubernetes.io/part-of=flux
              app.kubernetes.io/version=v2.5.1
              fluxcd.controlplane.io/name=flux
              fluxcd.controlplane.io/namespace=flux-system
Annotations:  kustomize.toolkit.fluxcd.io/ssa: Ignore
API Version:  source.toolkit.fluxcd.io/v1beta2
Kind:         OCIRepository
Metadata:
  Creation Timestamp:  2025-05-08T15:16:29Z
  Finalizers:
    finalizers.fluxcd.io
  Generation:        1
  Resource Version:  25924
  UID:               cab4d3a3-1c53-4596-aabe-dddb1fd4dd74
Spec:
  Interval:  1m0s
  Provider:  generic
  Ref:
    Tag:  latest
  Secret Ref:
    Name:   ghcr-auth
  Timeout:  60s
  URL:      oci://ghcr.io/myorg/myfleet
Status:
  Artifact:
    Digest:            sha256:37c8cb4e162c6360bcc37d816603f771e1c066585f968ecc0680e598090bcf3f
    Last Update Time:  2025-05-08T16:40:50Z
    Metadata:
      org.opencontainers.image.created:   2025-05-08T16:39:52Z
      org.opencontainers.image.revision:  refs/heads/master@sha1:cdbcfda4bb0de674a973ab33a09213498175072d
      org.opencontainers.image.source:    https://github.com/myorg/myfleet
    Path:                                 ocirepository/flux-system/flux-system/sha256:8a33dae1944ee84090d9ec35929819b59e21d5d4a735f570f5fecffea42f9495.tar.gz
    Revision:                             latest@sha256:8a33dae1944ee84090d9ec35929819b59e21d5d4a735f570f5fecffea42f9495
    Size:                                 2028
    URL:                                  http://source-controller.flux-system.svc.cluster.local./ocirepository/flux-system/flux-system/sha256:8a33dae1944ee84090d9ec35929819b59e21d5d4a735f570f5fecffea42f9495.tar.gz
  Conditions:
    Last Transition Time:  2025-05-08T16:40:50Z
    Message:               stored artifact for digest 'latest@sha256:8a33dae1944ee84090d9ec35929819b59e21d5d4a735f570f5fecffea42f9495'
    Observed Generation:   1
    Reason:                Succeeded
    Status:                True
    Type:                  Ready
    Last Transition Time:  2025-05-08T16:40:50Z
    Message:               stored artifact for digest 'latest@sha256:8a33dae1944ee84090d9ec35929819b59e21d5d4a735f570f5fecffea42f9495'
    Observed Generation:   1
    Reason:                Succeeded
    Status:                True
    Type:                  ArtifactInStorage
  Observed Generation:     1
  URL:                     http://source-controller.flux-system.svc.cluster.local./ocirepository/flux-system/flux-system/latest.tar.gz
Events:
  Type    Reason                      Age                  From               Message
  ----    ------                      ----                 ----               -------
  Normal  NewArtifact                 110m                 source-controller  stored artifact with revision 'latest@sha256:ce84365175cb09e51ae7f5d3d20afe1495ccfa5dc85bb6bb19dc9a9b213181a8' from 'oci://ghcr.io/myorg/myfleet', origin source 'https://github.com/myorg/myfleet', origin revision 'refs/heads/master@sha1:a4b8d601d7f11aa8356e5f96a063326206a3bb74'
  Normal  ArtifactUpToDate            103m (x7 over 109m)  source-controller  artifact up-to-date with remote revision: 'latest@sha256:ce84365175cb09e51ae7f5d3d20afe1495ccfa5dc85bb6bb19dc9a9b213181a8'
  Normal  NewArtifact                 102m                 source-controller  stored artifact with revision 'latest@sha256:0860081dda8bcf2be431375b01fcb82368dea97ade9e8f60cc29dfa1c841ea1e' from 'oci://ghcr.io/myorg/myfleet', origin source 'https://github.com/myorg/myfleet', origin revision 'refs/heads/master@sha1:16941549c14159e9ec546f647f00f8c5941fb43b'
  Normal  ArtifactUpToDate            100m (x2 over 101m)  source-controller  artifact up-to-date with remote revision: 'latest@sha256:0860081dda8bcf2be431375b01fcb82368dea97ade9e8f60cc29dfa1c841ea1e'
  Normal  NewArtifact                 99m                  source-controller  stored artifact with revision 'latest@sha256:0e28e2e6d89a8e6b69aaeb2ffc795116c3091e1c6eec3557c0fbebd331ed2121' from 'oci://ghcr.io/myorg/myfleet', origin source 'https://github.com/myorg/myfleet', origin revision 'refs/heads/master@sha1:07aff6d1397c1486bea8bc906c1aff726d8678cd'
  Normal  GarbageCollectionSucceeded  98m (x2 over 101m)   source-controller  garbage collected 1 artifacts
  Normal  ArtifactUpToDate            29m (x69 over 98m)   source-controller  artifact up-to-date with remote revision: 'latest@sha256:0e28e2e6d89a8e6b69aaeb2ffc795116c3091e1c6eec3557c0fbebd331ed2121'
  Normal  ArtifactUpToDate            2s (x26 over 25m)    source-controller  artifact up-to-date with remote revision: 'latest@sha256:8a33dae1944ee84090d9ec35929819b59e21d5d4a735f570f5fecffea42f9495'

What am I missing here?

[matheuscscp]

If you fetch that docker config inside the secret, can you use it locally to pull the artifact with flux pull artifact?

[bozho]

When you copied the secret from the cluster to your machine to test with oras, from which namespace did you copy? Try copying from kyverno

@matheuscscp Yeah, I picked up the kyverno one and compared it with the one in flux-system (from which it was copied using the fluxcd.controlplane.io/copyFrom annotation).

[matheuscscp]

Then this is your issue: controlplaneio-fluxcd/flux-operator#248

[bozho]

@stefanprodan Ahh... The original secret has the type kubernetes.io/dockerconfigjson. The copy is Opaque.

I've updated Secret in my ResourceSet, pushed, I can see the ResourceSet having an updated definition of the secret:

    API Version:                          v1
    Kind:                                 Secret
    Metadata:
      Annotations:
        fluxcd.controlplane.io/copyFrom:  flux-system/ghcr-auth
      Name:                               ghcr-auth
      Namespace:                          << default inputs.tenant inputs.namespace >>
      Type:                               kubernetes.io/dockercfg

But, the secret still has the Opaque type... The ResourceSet has the reconcile annotation set: fluxcd.controlplane.io/reconcileEvery: 5m, I've waited for ~15 minutes, the ResourceSet ran reconciliation 3 times, claiming success:

Events:
  Type    Reason                   Age                  From           Message
  ----    ------                   ----                 ----           -------
  Normal  ReconciliationSucceeded  45m                  flux-operator  Reconciliation finished in 249ms
  Normal  ReconciliationSucceeded  15m                  flux-operator  Reconciliation finished in 241ms
  Normal  ReconciliationSucceeded  35s (x186 over 16h)  flux-operator  (combined from similar events): Reconciliation finished in 236ms

[matheuscscp]

Just filed a fix: controlplaneio-fluxcd/flux-operator#250

[bozho]

@matheuscscp Yup, that's it.

As I've mentioned in the previous reply, the type doesn't seem to get reconciled. I've even deleted the secret and had ResourceSet re-create it. It was still created with the Opaque type.

Should I add this info to the ticket?

[bozho]

Oh, right. Secret type field is immutable. Can ResourceSet use the kustomize.toolkit.fluxcd.io/force: Enabled annotation to force re-creation of resources whose immutable fields have been changed?

[stefanprodan]

We'll introduce this option in the next release: controlplaneio-fluxcd/flux-operator#251

[bozho]

Oh, cool! Does that mean we'd only annotate the ResourceSet and it will force-recreate any resource within that failed reconciliation due to immutable fields?

Can we currently still use the annotation on individual resources inside ResourceSets?

[stefanprodan]

No this is only supported on the ResourceSet, given that force should be used as a one off, I see no issues with this.