Unable to autheticate image-reflector-controller with google artifact registry repo

[shreyasarani23]

Hi I have created an Autopilot GKE cluster, which by default has the workload identity enabled. I have created a google service account and binded the role "roles/artifactregistry.reader" and created an IAM allow policy that gives the Kubernetes ServiceAccount access to impersonate the IAM service account. But after annotating the service account and restarting the image-reflector-controller pod I am getting this error

{"level":"error","ts":"2025-07-28T05:43:35.247Z","msg":"scan failed: GET https://us-central1-docker.pkg.dev/v2/token?scope=repository%3Ainbound-isotope-466709-d1%2Fsample-app%2Fsample-app%3Apull&service=: DENIED: Unauthenticated request. Unauthenticated requests do not have permission \"artifactregistry.repositories.downloadArtifacts\" on resource \"projects/<project-id>/locations/us-central1/repositories/sample-app\" (or it may not exist)","controller":"imagerepository","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageRepository","ImageRepository":{"name":"sample-app","namespace":"flux-system"},"namespace":"flux-system","name":"sample-app","reconcileID":"866c4739-77e2-48a7-b2cd-fe10e3c6f6a0","error":"ReadOperationFailed"}

FYI I have bootstraped my flux (v2.6.4) using terraform below is the main.tf

resource "github_repository_deploy_key" "this" {
  title      = "Flux"
  repository =  var.github_repository
  key        = tls_private_key.flux.public_key_openssh
  read_only  = "false"
}


resource "flux_bootstrap_git" "this" {
  depends_on = [github_repository_deploy_key.this]
  components_extra = ["image-reflector-controller","image-automation-controller"]
  embedded_manifests = true
  kustomization_override = file("${path.module}/../flux-slack-notifications/flux-kustomization-patch.yaml")
  path               = "clusters/staging/gke"
}

below is the provider.tf

provider "google" {
  project = var.project_id
  region  = var.region
}



provider "flux" {
  kubernetes = {
    host                   = var.gke_endpoint
    #token                  = data.google_client_config.default.access_token
    cluster_ca_certificate = base64decode(var.cluster_ca_certificate)
     exec = {
      api_version = "client.authentication.k8s.io/v1beta1"
      command = "gke-gcloud-auth-plugin"
    }
  }
  git = {
    url = "ssh://[email protected]/${var.github_owner}/${var.github_repository}.git"
    ssh = {
      username    = "git"
      private_key = tls_private_key.flux.private_key_pem
    }
  }
}

provider "github" {
  owner = var.github_owner
  token = var.github_token
}

below is the flux-kustomization-patch.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata: 
  name: image-reflector-controller-sa-kustomization
resources:
  - gotk-components.yaml
  - gotk-sync.yaml
patches:
  # Patch annotation for image-reflector-controller's ServiceAccount
  - patch: |
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: image-reflector-controller
        namespace: flux-system
        annotations:
          iam.gke.io/gcp-service-account: flux-sa@<project-id>.iam.gserviceaccount.com
    target:
      kind: ServiceAccount
      name: image-reflector-controller
      namespace: flux-system

Below is my imagerepository spec and image policy

apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
 name: sample-app
 namespace: flux-system
spec:
 image: us-central1-docker.pkg.dev/<project-id>/sample-app/sample-app
 interval: 1m
--- 
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata:
 name: sample-app-policy
 namespace: flux-system
spec:
 imageRepositoryRef:
   name: sample-app
 policy:
   semver:
     range: ">0.0.0"
---

Confirmation of the annotation being added to image-reflector-controller service account

 kubectl describe sa image-reflector-controller -n flux-system
Name:                image-reflector-controller
Namespace:           flux-system
Labels:              app.kubernetes.io/component=image-reflector-controller
                     app.kubernetes.io/instance=flux-system
                     app.kubernetes.io/part-of=flux
                     app.kubernetes.io/version=v2.6.4
                     kustomize.toolkit.fluxcd.io/name=flux-system
                     kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations:         iam.gke.io/gcp-service-account: flux-sa@<project-id>.iam.gserviceaccount.com
Image pull secrets:  <none>
Mountable secrets:   <none>
Tokens:              <none>
Events:              <none>

Also checked that image-reflector-controller is referencing the same serviceAccountName i.e. ( image-reflector-controller). Also confirmed if the kubernetes service account is binded to the google service account using the below gcloud cli command and output is as expected

gcloud iam service-accounts get-iam-policy flux-sa@<project-id>.iam.gserviceaccount.com

bindings:
- members:
 - serviceAccount:<project-id>.svc.id.goog[flux-system/image-reflector-controller]
 role: roles/iam.workloadIdentityUser
etag: BwY69mGhyDk=
version: 1

Can someone please help me to debug the issue am I missing something? Any inputs on further troubleshooting will be appreciated

[matheuscscp]

https://fluxcd.io/flux/components/image/imagerepositories/#provider

[shreyasarani23]

Thanks @matheuscscp for pointing out I missed configuring the provider in the image repository and also need to enable the ObjectLevelWorkloadIdentity when using spec.provider/spec.serviceAccountName as mentioned in the docs. Below are the updated yaml's for imagerepository.yaml and flux-kustomization-patch.yaml

---
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
  name: sample-app
  namespace: flux-system
spec:
  provider: gcp
  image: us-central1-docker.pkg.dev/<project-id>/sample-app/sample-app
  interval: 1m
  serviceAccountName: image-reflector-controller
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
  name: image-reflector-controller-kustomization
resources:
  - gotk-components.yaml
  - gotk-sync.yaml
patches:
  # Patch annotation for image-reflector-controller's ServiceAccount
  - patch: |
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: image-reflector-controller
        namespace: flux-system
        annotations:
          iam.gke.io/gcp-service-account: [email protected]
    target:
      kind: ServiceAccount
      name: image-reflector-controller
      namespace: flux-system
  - target:
      kind: Deployment
      name: image-reflector-controller
    patch: |
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --feature-gates=ObjectLevelWorkloadIdentity=true
---