New image policy markers not picked up when using different checkout and push branch

[haayv]

I am not sure if this is a bug, but at least the behavior is rather unintuitive. When using the ImageUpdateAutomation with different spec.git.checkout.ref.branch and spec.git.push.branch, newly added image policy markers (on the checkout branch) are not picked up by the image-automation-controller, if the push branch already exists.

Steps to reproduce:

1. Create an ImageUpdateAutomation similar to this one:

apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageUpdateAutomation
metadata:
  name: buggy-image-update-automation
  namespace: flux-system
spec:
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: gitrepo
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        email: fluxcdbot@users.noreply.github.com
        name: fluxcdbot
    push:
      branch: flux-image-updates
  update:
    path: .
    strategy: Setters

2. Wait for the automation to pickup changes for existing image policies and to create the flux-image-updates branch.
3. Add a new image repo & image policy and image policy markers referencing those.
4. On the next reconciliation only the image policy markers existing before step 3 will be picked up by the image-automation-contoller. (can be seen when running the image-automation-controller with --log-level=trace)

Expected Behavior

The image policy markers added in step 4 should be picked up by the image-automation-controller as well.

Flux Setup

flux: v2.6.4

► checking prerequisites
✔ Kubernetes 1.32.6 >=1.31.0-0
► checking version in cluster
✔ distribution: flux-v2.6.4
✔ bootstrapped: false
► checking controllers
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.41.2
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.35.2
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v1.6.1
✔ source-controller: deployment ready
►ghcr.io/fluxcd/source-controller:v1.6.2
► checking crds
✔ buckets.source.toolkit.fluxcd.io/v1
✔ gitrepositories.source.toolkit.fluxcd.io/v1
✔ helmcharts.source.toolkit.fluxcd.io/v1
✔ helmrepositories.source.toolkit.fluxcd.io/v1
✔ imagepolicies.image.toolkit.fluxcd.io/v1beta2
✔ imagerepositories.image.toolkit.fluxcd.io/v1beta2
✔ imageupdateautomations.image.toolkit.fluxcd.io/v1beta2
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1
✔ ocirepositories.source.toolkit.fluxcd.io/v1
✔ all checks passed

Git Provider

Enterprise GitHub

Container Registry Provider

Azure Container Registry

Context

For context:

• The image repo is able to retrieve all image tags.
• The image policy is able to decide the latest semver tag.
• The image update automation is able to pickup the latest semver tag from the image policy.
• The image-automation-controller has the appropriate permission to push/create the branch.

I have already had a look into the source code and from my limited study of the source code this behavior might stem from the following code in (internal/source/source.go:229-233):

func (sm *SourceManager) CheckoutSource(ctx context.Context, options ...CheckoutOption) (*git.Commit, error) {
	//...
	if sm.srcCfg.switchBranch {
		if err := sm.gitClient.SwitchBranch(gitOpCtx, sm.srcCfg.pushBranch); err != nil {
			return nil, err
		}
	}
	//...
}

This code seems to run before scanning for image policy markers and switches the cloned repository to the push branch. This causes the image-automation-controller to miss the newly added image-policy-marker available only on the checkout branch.

If i am mistaken and this is not a bug, but rather intended behavior and/or it is already part of the documentation and i just missed it or if further information/clarifications are necessary please let me know.

[dchien234]

I also ran into problems with a similar setup, i.e. using the ImageUpdateAutomation with different spec.git.checkout.ref.branch and spec.git.push.branch.

Flux detects image version changes (at least, image-automation-controller log shows that), it's just that the spec.git.push.branch and the spec.git.checkout.ref.branch are at the same commit. Seems like Flux knows that there's new tags / image version update but it doesn't change the file (marked with markers), and commit.

It just pushes the spec.git.push.branch without any changes. Not sure how can I get around this.

[dchien234]

hi @matheuscscp , do you have any suggestions for this?

[matheuscscp]

My suggestion is to find a reproducible setup that you can use for investigating the image-automation-controller logic line by line in the Go code to find out the root cause. We can fix this for Flux 2.8, to be released early next year, hopefully end of Jan/beg. of Feb. I unfortunately can't look into this right now, but feel free to do so. If someone makes a thorough analysis and proposes a fix I can definitely review.