How do I force reconciliation of all resources?

[mkoval]

I frequently need to delete and re-create resources while iterating on a change. For example, this is necessary to re-create PVCs that may have been initialized with incorrect data. It is also useful when iterating on a replicaset or statefulset, which (mostly) prohibit edits.

This can leave the cluster in an inconsistent state, where some resources don't exist because they have manually been deleted (Kustomize) or uninstalled (Helm). I can force the reconciliation of individual Kustomize and HelmRelease resources using the flux reconcile CLI. But that requires that I know which exact flux artifacts need reconciliation.

I was hoping to find a command like:

• flux reconcile --all that reconciles all resources, regardless of whether Flux thinks they have changed and/or
• flux reconcile -l my-label=my-label-value to reconcile resources that have a label, regardless of whether Flux thinks they have changed and/or
• flux reconcile git source my-source to recursively reconcile all resources within the repository, regardless of whether Flux thinks they have changed.

As far as I can tell, none of those exist -- but flux reconcile does force reconciliation of a resource if manually specify it on the command-line. I have worked around this by writing a shell script that runs a separate flux reconcile command for each resource that I have touched, but that is tedious and error-prone to keep that script up-to-date. How do ya'll deal with this?

P.S. I am open to changing my workflow if there is a better way to iterate on changes that doesn't require solving this problem. I just haven't found one yet!

[souleb]

flux resources are chained. Wouldn't reconciling the top level kustomization be enough?

[mkoval]

I thought so, but it doesn't seem to work on a my HelmRelease. I have the following dependency chain:

• GitRepository flux-system/home-kubernetes
  • Kustomization flux-system/cluster-apps
    • Kustomization flux-system/home-assistant
      • HelmRelease smart-home/home-assistant

I was able to replicate the behavior I described:

1. Uninstall the Helm release:

   $ helm uninstall -n smart-home home-assistant
   release "home-assistant" uninstalled

2. Try reconciling the GitRepository:

   $ flux reconcile source git -n flux-system home-kubernetes
   ► annotating GitRepository home-kubernetes in flux-system namespace
   ✔ GitRepository annotated
   ◎ waiting for GitRepository reconciliation
   ✔ fetched revision main@sha1:4c880fbf49dbdf049b2624baea4aad26d153457b

   ❗ Returns immediately, doesn't install the Helm chart.

3. Try reconciling the first-level Kustomization

   $ flux reconcile -n flux-system kustomization cluster-apps
   ► annotating Kustomization cluster-apps in flux-system namespace
   ✔ Kustomization annotated
   ◎ waiting for Kustomization reconciliation
   ✔ applied revision main@sha1:4c880fbf49dbdf049b2624baea4aad26d153457b

   ❗ Returns immediately, doesn't install the Helm chart.

4. Try reconciling the second-level Kustomization:

   $ flux reconcile -n flux-system kustomization home-assistant
   ► annotating Kustomization home-assistant in flux-system namespace
   ✔ Kustomization annotated
   ◎ waiting for Kustomization reconciliation
   ✔ applied revision main@sha1:4c880fbf49dbdf049b2624baea4aad26d153457b

   ❗ Returns immediately, doesn't install the Helm chart.

5. Try reconciling the HelmRelease:

   $ flux reconcile -n smart-home helmrelease home-assistant
   ► annotating HelmRelease home-assistant in smart-home namespace
   ✔ HelmRelease annotated
   ◎ waiting for HelmRelease reconciliation
   ✔ applied revision 3.1.0

   🎉 Pauses for a few seconds, installs the Helm chart!

I would have expected all of these commands to have the same effect as (5). Am I doing something wrong?

[Aubermean]

I have hit the same dead end. What did you settle on?

[gecube]

$ helm uninstall -n smart-home home-assistant
release "home-assistant" uninstalled

and what did you expected? I understand if you complained if you did

$ kubectl delete helmrelease -n smart-home home-assistant

and nothing happened...

[Alexander-N]

Interestingly, the suspend and resume commands have an --all option, so you can e.g. force reconciliation of all kustomizations with

flux suspend kustomization --all
flux resume kustomization --all

There seems to be no --all-namespaces option for these commands.

[jonathanfoster]

This is the answer. Also, you can speed up dependency reconciliation with this patch, #4739 (comment).

[KThompso]

Good option. This would resume any resources you had suspended for other reasons though, which might not be desired.