Kustomization reconciliation fails on multiple $patch: delete uses in a single strategic merge patch

[tdemin]

Environment

• Flux v2.4.0 (kustomize-controller v1.4.0)
• Kubernetes v1.29.3 (seemingly irrelevant)

Description

If multiple $patch: delete strategic merge patches are provided in a single spec.patches.patch in a kustomize.toolkit.fluxcd.io/v1.Kustomization resource, kustomize-controller fails to reconcile with the following status:

flux-system   kustomization.kustomize.toolkit.fluxcd.io/debug         3m20s   False     kustomize build failed: recovered from kustomize build panic: runtime error: invalid memory address or nil pointer dereference

kustomize-controller log reports the following:

{"level":"error","ts":"2024-12-09T18:41:35.677Z","msg":"Reconciliation failed after 108.007784ms, next try in 1h0m0s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{"name":"debug","namespace":"flux-system"},"namespace":"flux-system","name":"debug","reconcileID":"5eef40e9-8bf0-4a65-a178-eae10b04f877","revision":"main@sha1:4258282a038ab9506e1b37b3a44cc5ab4d38075e","error":"kustomize build failed: recovered from kustomize build panic: runtime error: invalid memory address or nil pointer dereference"}

The Kustomization resource needs to look like that:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: whatever
spec:
  patches:
    - patch: |-
        $patch: delete
        apiVersion: v1
        kind: Whatever
        metadata:
          name: whatever1
        ---
        $patch: delete
        apiVersion: v1
        kind: Whatever
        metadata:
          name: whatever2

Reproduction steps

0. Create a fresh Kubernetes cluster and install Flux v2.4.0 in whatever way convenient.
1. Create a Kustomization with the following files:

% cat > kustomization.yaml
resources:
  - configmaps.yml
% cat > configmaps.yml
apiVersion: v1
kind: Namespace
metadata:
  name: debug
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: debug1
  namespace: debug
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: debug2
  namespace: debug
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: debug3
  namespace: debug

2. Setup Flux to reconcile this Kustomization while deleting two of the ConfigMaps:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: debug
  namespace: flux-system
spec:
  interval: 1h
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./debug
  prune: true
  patches:
    - patch: |-
        $patch: delete
        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: debug1
          namespace: debug
        ---
        $patch: delete
        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: debug2
          namespace: debug

3. Wait for Flux to reconcile and observe the status above.

Additional details

Apparently closely related to kubernetes-sigs/kustomize#5552? The current Flux-specific workaround is simply to split every removal patch into its separate - patch:.

[stefanprodan]

Flux uses Kustomize Go SDK, if this doesn't work with the kustomize CLI then it can't work with Flux.

[cappyzawa]

I've added a test case for this issue in #1466. The test currently fails due to the panic described in this issue, but it will serve as a regression test once the upstream fix in kubernetes-sigs/kustomize#5552 is included in a future kustomize release.

This ensures we can verify the fix works correctly when the upstream dependency is updated.

[cappyzawa]

FYI: kustomize v5.7.0 has been released! 🎉
https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize/v5.7.0

It includes the following fix:

#5859: fix: Don't panic on multiple $patch: delete strategic merge patches in a single patch file

[matheuscscp]

@cappyzawa We need dedicated PRs to upgrade kustomize in these places:

https://github.com/search?q=org%3Afluxcd%20Pin%20kustomize%20to%20v5.6.0&type=code

[cappyzawa]

If no one is assigned to this yet, I’ll submit individual PRs starting tomorrow.