token recognition error in kustomization on valid CRD

[mloiseleur]

Describe the bug

When applying infrastructure components yaml from AWS provider of Cluster API, flux kustomization fails with this error message:

error message

Kustomization reconciliation failed: CustomResourceDefinition/rosacontrolplanes.controlplane.cluster.x-k8s.io dry-run failed (Invalid): CustomResourceDefinition.apiextensions.k8s.io "rosacontrolplanes.controlplane.cluster.x-k8s.io" is invalid: spec.validation.openAPIV3Schema.properties[spec].properties[version].x-kubernetes-validations[0].rule: Invalid value: apiextensions.ValidationRule{Rule:"self.matches('^(0|[1-9]\\d*)\\.(0|[1-9]\\d*)\\.(0|[1-9]\\d*)$')", Message:"version must be a valid semantic version", MessageExpression:"", Reason:(*apiextensions.FieldValueErrorReason)(nil), FieldPath:"", OptionalOldSelf:(*bool)(nil)}: compilation failed: ERROR: <input>:1:14: Syntax error: token recognition error at: ''^(0|[1-9]\d'
 | self.matches('^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)$')
 | .............^
ERROR: <input>:1:26: Syntax error: extraneous input '*' expecting {'[', '{', '(', ')', '.', '-', '!', 'true', 'false', 'null', NUM_FLOAT, NUM_INT, NUM_UINT, STRING, BYTES, IDENTIFIER}
 | self.matches('^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)$')
 | .........................^
ERROR: <input>:1:28: Syntax error: token recognition error at: '\'
 | self.matches('^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)$')
 | ...........................^
ERROR: <input>:1:30: Syntax error: no viable alternative at input '.('
 | self.matches('^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)$')
 | .............................^
ERROR: <input>:1:32: Syntax error: token recognition error at: '|['
 | self.matches('^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)$')
 | ...............................^
ERROR: <input>:1:37: Syntax error: mismatched input ']' expecting <EOF>
 | self.matches('^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)$')
 | ....................................^
ERROR: <input>:1:38: Syntax error: token recognition error at: '\'
 | self.matches('^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)$')
 | .....................................^
ERROR: <input>:1:42: Syntax error: token recognition error at: '\'
 | self.matches('^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)$')
 | .........................................^
ERROR: <input>:1:46: Syntax error: token recognition error at: '|['
 | self.matches('^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)$')
 | .............................................^
ERROR: <input>:1:52: Syntax error: token recognition error at: '\'
 | self.matches('^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)$')
 | ...................................................^
ERROR: <input>:1:56: Syntax error: token recognition error at: '$'
 | self.matches('^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)$')
 | .......................................................^
ERROR: <input>:1:57: Syntax error: token recognition error at: '')'
 | self.matches('^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)$')
 | ........................................................^

When I try to load this YAML with kubernetes, it works

$ kubectl apply -f infrastructure-components.yaml
[...]
customresourcedefinition.apiextensions.k8s.io/rosamachinepools.infrastructure.cluster.x-k8s.io configured
customresourcedefinition.apiextensions.k8s.io/rosacontrolplanes.controlplane.cluster.x-k8s.io configured

Strange things, this yaml section works fine:

              billingAccount:
                description: BillingAccount is an optional AWS account to use for
                  billing the subscription fees for ROSA clusters. The cost of running
                  each ROSA cluster will be billed to the infrastructure account in
                  which the cluster is running.
                type: string
                x-kubernetes-validations:
                - message: billingAccount is immutable
                  rule: self == oldSelf
                - message: billingAccount must be a valid AWS account ID
                  rule: self.matches('^[0-9]{12}$')

it seems it's only for this specific /spec/versions/0/schema/openAPIV3Schema/properties/spec/properties/version/x-kubernetes-validations field

              version:
                description: OpenShift semantic version, for example "4.14.5".
                type: string
                x-kubernetes-validations:
                - message: version must be a valid semantic version
                  rule: self.matches('^(0|[1-9]\\d*)\\.(0|[1-9]\\d*)\\.(0|[1-9]\\d*)$')

Steps to reproduce

Just add this YAML in a kustomization file, and it throws parser token error.

Expected behavior

A YAML accepted by kubernetes client should also be accepted by Flux.

Screenshots and recordings

No response

OS / Distro

Kubernetes

Flux version

v2.2.3

Flux check

► checking prerequisites
✔ Kubernetes 1.29.0-eks-c417bb3 >=1.26.0-0
► checking version in cluster
✔ distribution: flux-v2.2.3
✔ bootstrapped: true
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.37.4
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.37.1
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.31.2
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v1.2.2
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v1.2.4
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v1.2.4
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta3
✔ buckets.source.toolkit.fluxcd.io/v1beta2
✔ gitrepositories.source.toolkit.fluxcd.io/v1
✔ helmcharts.source.toolkit.fluxcd.io/v1beta2
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta2
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2
✔ imagepolicies.image.toolkit.fluxcd.io/v1beta2
✔ imagerepositories.image.toolkit.fluxcd.io/v1beta2
✔ imageupdateautomations.image.toolkit.fluxcd.io/v1beta1
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta3
✔ receivers.notification.toolkit.fluxcd.io/v1
✔ all checks passed

Additional context

It's possible to workaround this issue by removing this field with a json patch:

- patch: |-
    - op: remove
      path: /spec/versions/0/schema/openAPIV3Schema/properties/spec/properties/version/x-kubernetes-validations
  target:
    kind: CustomResourceDefinition
    name: rosamachinepools.infrastructure.cluster.x-k8s.io
- patch: |-
    - op: remove
      path: /spec/versions/0/schema/openAPIV3Schema/properties/spec/properties/version/x-kubernetes-validations
  target:
    kind: CustomResourceDefinition
    name: rosacontrolplanes.controlplane.cluster.x-k8s.io

Code of Conduct

• I agree to follow this project's Code of Conduct

[stefanprodan]

Hey @mloiseleur I can't reproduce this with Kubernetes upstream v1.29.3 PR here fluxcd/kustomize-controller#1124

[mloiseleur]

It's annoying and no surprising at the same time : this test is like the strange things section in the issue description.

There is two possible difference between config of this issue and your PR.

1. Did e2e test runs on arm ? We are using ghcr.io/fluxcd/kustomize-controller:v1.2.2 on arm64 kubernete node
2. You should try the whole file. It's triggered by a succession of something and this section.

since I cannot commit in your PR, I tried option 2 in fluxcd/kustomize-controller#1125