Pulling atomically correct manifests from S3 bucket

[matthewwoodruff]

Hey, currently we're using git repositories (from github) as the source for our cluster manifests but in order to reduce egress we'd like to use S3 buckets. We're in the process of our migration and all is going well however we have a query around a potential issue with S3.

Using git our manifests can be pulled at a specific commit/ref and we have full confidence the resources referenced by our kustomization.yaml are correct as a unit. With S3, as each file (~3000) in our manifests directory is updated independently, we cannot guarantee the update is applied atomically so the source controller could generate an invalid artifact for the kustomization controller to use.

Is this understanding correct? If so are there any recommended solutions for this? For example we could package our manifests directory and store into S3, but how could that be used in flux?

Thanks 😄

[stefanprodan]

You could run kustomize build in your CI and push the resulting multi-doc YAML to a dir in S3, being a single file that you override every time you can ensure Flux will apply all objects belonging to a commit.

We are planing to add a new source type (OCI images), that may be better suitable for your when this lands fluxcd/source-controller#450