Feature request: Flux Kustomization to have a DriftDetection option

[barret0224]

In the Helm controller the HelmRelease has an option to specify DriftDetection configuration:

https://fluxcd.io/flux/components/helm/helmreleases/#drift-detection

It would be useful to have a similar configuration option for the Kustomization resources to control whether the kustomize-controller should only detect drifts/changes or should actively fix those.

I know the helm controller and the HelmRelease resource is a bit different form the Kustomization, so it might not be that obvious to take the same approach, but this functionality could be extremely useful in scenarios when the Kustomization is suspended and then turned on again.

[tnm-]

Exactly what I'm looking for!

[barret0224]

may I ask any of the kustomization-controller's maintainers opinion on how this could be implemented if it is possible at all?
this functionality is there in other gitops controllers and also supported in one of flux's own controllers, but there is a functionality gap in case of the Kustomization based controller...

[stefanprodan]

The drift feature in helm-controller is very specific to how Helm works, Helm stores a fully copy of all manifests in-cluster in Kubernetes Secrets (this is called Helm storage), the drift detection runs for the storage vs cluster, not for Git vs cluster.

In kustomize-controller we don't duplicate the manifests in etcd like Helm, this would be extremely wasteful, there are no plans to do what Helm does.

[tnm-]

When flux is used cross-cluster the connection may be lost temporarily and manual changes may be done on the target resource bypassing flux. After this for a grateful recovery it would be good to have a flag in the Kustomization that warns for drift but does not overwrite it. (Then based on the delta we could decide to update git content or let flux overwrite the drift.)

[matheuscscp]

When flux is used cross-cluster the connection may be lost temporarily and manual changes may be done on the target resource bypassing flux. After this for a grateful recovery it would be good to have a flag in the Kustomization that warns for drift but does not overwrite it. (Then based on the delta we could decide to update git content or let flux overwrite the drift.)

This is not GitOps

[stefanprodan]

We didn't designed the Flux controllers to works like a CLI in an imperative way. If you want to do that, then suspend the Flux Kustomizations and use the Flux CLI. There is a flux diff kustomization, it shows you the diff, you review it, then you run flux resume ks then flux suspend ks.

[tnm-]

It's emergency handling.

[matheuscscp]

It's emergency handling.

And Flux is obviously prepared for emergencies, just not the way it's being requested here. GitOps is a paradigm shift, it does support "manual interventions" but it does not support "manual approvals". See the right way to handle emergencies in Stefan's comment above

[barret0224]

The drift feature in helm-controller is very specific to how Helm works, Helm stores a fully copy of all manifests in-cluster in Kubernetes Secrets (this is called Helm storage), the drift detection runs for the storage vs cluster, not for Git vs cluster.

In kustomize-controller we don't duplicate the manifests in etcd like Helm, this would be extremely wasteful, there are no plans to do what Helm does.

the kustomize controller doesnt need to "duplicate" the manifests in etcd... it might utilize/extend the existing inventory info...
even the documentation mentions drift detection:
https://fluxcd.io/flux/components/kustomize/kustomizations/#inventory

but it seems this information is not exposed on the resource...

[tnm-]

I understand it won't be a priority for you, but would you be open for PRs (that would do pretty much what the cli does if a Kustomization has a special flag).

[tnm-]

It's emergency handling.

And Flux is obviously prepared for emergencies, just not the way it's being requested here. GitOps is a paradigm shift, it does support "manual interventions" but it does not support "manual approvals". See the right way to handle emergencies in Stefan's comment above

Those steps are manual. I'm looking for something that's easy to automate for the detection part.

[barret0224]

We didn't designed the Flux controllers to works like a CLI in an imperative way. If you want to do that, then suspend the Flux Kustomizations and use the Flux CLI. There is a flux diff kustomization, it shows you the diff, you review it, then you run flux resume ks then flux suspend ks.

I have checked that, and flux diff kustomization apparently doesnt work for example if you have a distributed setup (running the flux controller in a "management" cluster and deploy the workloads into other clusters)

[matheuscscp]

the kustomize controller doesnt need to "duplicate" the manifests in etcd... it might utilize/extend the existing inventory info...

Please review this message one more time, it doesn't add up. The inventory is in etcd, it's part of .status. And you can only detect a diff when you compare whole manifests, so, yes, you are basically telling us to store the manifests in etcd.

[1] Then based on the delta we could decide to update git content or let flux overwrite the drift.
[2] Those steps are manual. I'm looking for something that's easy to automate for the detection part.

These two sentences also don't add up. How would you automate your decision? And CLI commands allow for a lot of bash automation, there are already commands that you can use.

I have checked that, and flux diff kustomization apparently doesnt work for example if you have a distributed setup (running the flux controller in a "management" cluster and deploy the workloads into other clusters)

This is a feature request we would be willing to accept contributions on. Let's improve the existing model, which follows GitOps principles. Let's not introduce models that go against it