postBuild.substitute doesn't work in Flux CD

[Cloufish]

Hi,

I have following files on my public repo https://github.com/Cloufish/k8s-home-ops/tree/main/apps/base/code-server:

/app/release.yml

---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata: 
  name: &name ${appName}
spec:
  interval: 5m
  chart:
    spec: 
      chart: *name
      sourceRef: 
        kind: HelmRepository
        name: bjw-s-charts
        namespace: flux-system
      valuesFiles:
        - values.yml
      interval: 1m
---
apiVersion: v1
kind: Namespace
metadata:
  name: &name ${appName}
  labels:
    name: *name

ks.yml

---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: &appname code-server # Change this
  namespace: flux-system
spec:
  targetNamespace: *appname
  commonMetadata:
    labels:
      app.kubernetes.io/name: *appname
  interval: 30m
  timeout: 5m
  prune: true 
  path: "./kubernetes/apps/programming/code-server/app" # Change this
  sourceRef:
    kind: GitRepository
    name: flux-system
    namespace: flux-system
  wait: true
  postBuild:
    substitute:
      app_k8s_domain_name: k8s.cloufish.xyz
      default_k8s_cert_name: wildcard-k8s-cert-prod
      appName: *appname

Even though it is a very simple setup, the Flux doesn't resolve the variables

 $ flux get kustomizations --watch
NAME            REVISION                SUSPENDED       READY   MESSAGE
flux-system     main@sha1:5967c7a5      False           Unknown Reconciliation in progress
flux-system     main@sha1:5967c7a5      False   Unknown Reconciliation in progress
flux-system     main@sha1:5967c7a5      False   False   Namespace/${appName} dry-run failed (Invalid): Namespace "${appName}" is invalid: [metadata.name: Invalid value: "${appName}": a lowercase RFC 1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name',  or '123-abc', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?'), metadata.labels: Invalid value: "${appName}": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')]
✗ timed out waiting for the condition

It doesn't make any sense to me why it's not substituting.
The substitute is enabled by default, right?

spec:
  targetNamespace: *appname
  commonMetadata:
    labels:
      app.kubernetes.io/name: *appname
      kustomize.toolkit.fluxcd.io/substitute: enabled

The plan is simple

1. Declare substitute variables in ks.yml
2. Use them in release.yml (and values.yml in future)

When I execute flux check all checks passed

[stefanprodan]

Get rid of anchors from all these manifests and it should work.

[Cloufish]

Thanks for your response @stefanprodan !!!

Unfortunately previously I already tried this. I tried this now and didn't work
I noticed that flux get kustomizations show the same sha1
It's not the problem right? Because I also deleted Namespace resource declaration and the error changed, with sha1 being the same

ks.yml

---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata: 
  name: ${appName}
spec:
  interval: 5m
  chart:
    spec: 
      chart: ${appName}
      sourceRef: 
        kind: HelmRepository
        name: bjw-s-charts
        namespace: flux-system
      valuesFiles:
        - values.yml
      interval: 1m
---
apiVersion: v1
kind: Namespace
metadata:
  name: ${appName}
  labels:
    name: ${appName}

flux-system     main@sha1:73dd3cf2      False           False   Namespace/${appName} dry-run failed (Invalid): Namespace "${appName}" is invalid: [metadata.name: Invalid value: "${appName}": a lowercase RFC 1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name',  or '123-abc', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?'), metadata.labels: Invalid value: "${appName}": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')]
flux-system     main@sha1:73dd3cf2      False   Unknown Reconciliation in progress
flux-system     main@sha1:73dd3cf2      False   Unknown Reconciliation in progress
flux-system     main@sha1:73dd3cf2      False   Unknown Reconciliation in progress
flux-system     main@sha1:73dd3cf2      False   Unknown Reconciliation in progress
flux-system     main@sha1:73dd3cf2      False   False   Namespace/${appName} dry-run failed (Invalid): Namespace "${appName}" is invalid: [metadata.name: Invalid value: "${appName}": a lowercase RFC 1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name',  or '123-abc', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?'), metadata.labels: Invalid value: "${appName}": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')]
flux-system     main@sha1:73dd3cf2      False   Unknown Reconciliation in progress
flux-system     main@sha1:73dd3cf2      False   Unknown Reconciliation in progress
flux-system     main@sha1:73dd3cf2      False   Unknown Reconciliation in progress
flux-system     main@sha1:73dd3cf2      False   False   Namespace/${appName} dry-run failed (Invalid): Namespace "${appName}" is invalid: [metadata.name: Invalid value: "${appName}": a lowercase RFC 1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name',  or '123-abc', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?'), metadata.labels: Invalid value: "${appName}": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')]

[stefanprodan]

Run flux reconcile ks flux-system to pull the new commit

[stefanprodan]

Also the Flux Kustomization that applies the code-server dir, does it have substituions for ${appName}?

[Cloufish]

Also the Flux Kustomization that applies the code-server dir, does it have substituions for ${appName}?

Yes, Kustomization also

---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: ${appName}
  namespace: flux-system
spec:
  targetNamespace: ${appName}
  commonMetadata:
    labels:
      app.kubernetes.io/name: ${appName}
  interval: 30m
  timeout: 5m
  prune: true 
  path: "./kubernetes/apps/programming/code-server/app" # Change this
  sourceRef:
    kind: GitRepository
    name: flux-system
    namespace: flux-system
  wait: true
  postBuild:
    substitute:
      app_k8s_domain_name: k8s.cloufish.xyz
      default_k8s_cert_name: wildcard-k8s-cert-prod
      appName: code-server

Run flux reconcile ks flux-system to pull the new commit

Unfortunately this trigger same the error and timeouts eventually

flux reconcile ks flux-system
► annotating Kustomization flux-system in flux-system namespace
✔ Kustomization annotated
◎ waiting for Kustomization reconciliation
``` (3 minutes already passed. In flux get kustomizations --watch is the same error)

[stefanprodan]

The whole structure is wrong, you can’t have the Flux Kustomization in a dir like that, multiple Kustomizations are fighting over the same path

[Cloufish]

The whole structure is wrong, you can’t have the Flux Kustomization in a dir like that, multiple Kustomizations are fighting over the same path

So the flux-system directory must be under?

└── clusters
    ├── production

From what I read in https://fluxcd.io/flux/guides/repository-structure/

[Cloufish]

Okay I will use the template in https://github.com/fluxcd/flux2-kustomize-helm-example
Previously I just bootstraped flux in the blank repositosory

https://github.com/fluxcd/flux2-kustomize-helm-example

When I'm done and It will I will let you know :)
Thanks for being here and helping @stefanprodan

[Cloufish]

The whole structure is wrong, you can’t have the Flux Kustomization in a dir like that, multiple Kustomizations are fighting over the same path

Could you give a hint @stefanprodan where to place Flux Kustomization with postBuild.substitute for specific app?

I've deployed it using your template and it didn't fix the problem

In your https://github.com/fluxcd/flux2-kustomize-helm-example template I see you are declaring Kustomizations in clusters\[...] directory. Is this the only way?

In
clusters/production/apps.yaml I see

---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 10m0s
  dependsOn:
    - name: infra-configs
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./apps/production # <---
  prune: true
  wait: true
  timeout: 5m0s

Does that mean that in apps/production I need to have following kustomize Kustomization?
Is Flux Kustomization dependent on kustomize Kustomization files?

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ./APP1/ks.yaml
  - ./APP2/ks.yaml
  - ./APP3/ks.yaml
  - [...]

[stefanprodan]

Is this the only way?

There are many way to do it, this example is a for a simple setup where all apps are applied by a single Flux Kustomization. If you follow this example, then the substitutions must go into clusters/production/apps.yaml for all apps.

Check out the D2 example repos for a complex setup where substitutions are used https://github.com/controlplaneio-fluxcd/d2-fleet

[Cloufish]

Okay, but I still don't get it working @stefanprodan
I have following

clusters/production/apps.yaml

---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps-configs
  namespace: flux-system
spec:
  interval: 10m0s
  dependsOn:
    - name: infra-configs
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./apps/production
  prune: true
  wait: true
  timeout: 5m0s
  postBuild:
    substitute:
      app_k8s_domain_name: k8s.cloufish.xyz
      default_k8s_cert_name: wildcard-k8s-cert-prod
      code_server_app_name: code-server

And in
/apps/production/release.yaml

---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata: 
  name: ${code_server_app_name}
  namespace: ${code_server_app_name}
spec:
  releaseName: ${code_server_app_name}
  interval: 5m
  chart:
    spec: 
      chart: ${code_server_app_name}
      sourceRef: 
        kind: HelmRepository
        name: bjw-s-charts
        namespace: flux-system
      valuesFiles:
        - values.yml
      interval: 1m
---
apiVersion: v1
kind: Namespace
metadata:
  name: ${code_server_app_name}
  labels:
    name: ${code_server_app_name}

But I still get an error

apps    main@sha1:b19d75d5      False   Unknown Reconciliation in progress
apps    main@sha1:b19d75d5      False   Unknown Reconciliation in progress
apps    main@sha1:b19d75d5      False   Unknown Reconciliation in progress
apps    main@sha1:b19d75d5      False   Unknown Reconciliation in progress
apps    main@sha1:b19d75d5      False   False   Namespace/code-server dry-run failed (Invalid): Namespace "code-server" is invalid: metadata.labels: Invalid value: "${code_server_app_name}": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')

[stefanprodan]

Isn't it obvious? You have duplicated Flux Kustomizations applying the same stuff, the error comes from apps while you also apply the same path with apps-configs.