Issues when destroying modules via the gruntworks pipeline

[umm0n]

We're using the Gruntworks Ref arch, and the Gruntworks pipeline (and associated scripts) for deploying our infra, with GHA.

We encounter this issue only when attempting to delete modules, and only under specific circumstances. To reproduce:

1. We delete a bunch of modules, create a PR and push our changes.
2. The pipeline runs a destroy plan on each deleted module.
3. We merge the PR and the pipeline begins to run destroy on each deleted directory, as shown here:

The following modules were deleted:
<redacted>
Running apply  on each deleted module.
Destroying <redacted>

5. Another PR is merged while this pipe is running. The moment this happens, the pipe picks up on the latest commit (which no longer has any of the modules, as they were deleted) and fails.

This is the message we get:

[ecs-deploy-runner][2023-03-10T16:48:44+0000] Note: switching to 'HEAD^'.
[ecs-deploy-runner][2023-03-10T16:48:44+0000] You are in 'detached HEAD' state. You can look around, make experimental
[ecs-deploy-runner][2023-03-10T16:48:44+0000] changes and commit them, and you can discard any commits you make in this
[ecs-deploy-runner][2023-03-10T16:48:44+0000] state without impacting any branches by switching back to a branch.
[ecs-deploy-runner][2023-03-10T16:48:44+0000] If you want to create a new branch to retain commits you create, you may
[ecs-deploy-runner][2023-03-10T16:48:44+0000] do so (now or later) by using -c with the switch command. Example:
[ecs-deploy-runner][2023-03-10T16:48:44+0000]   git switch -c <new-branch-name>
[ecs-deploy-runner][2023-03-10T16:48:44+0000] Or undo this operation with:
[ecs-deploy-runner][2023-03-10T16:48:44+0000]   git switch -
[ecs-deploy-runner][2023-03-10T16:48:44+0000] Turn off this advice by setting config variable advice.detachedHead to false
[ecs-deploy-runner][2023-03-10T16:48:44+0000] HEAD is now at 03729b5 delete <redacted> (#1090)
[ecs-deploy-runner][2023-03-10T16:48:44+0000] [INFO] [infrastructure-deploy-script] 2023-03-10 16:48:44  Successfully checked out HEAD^
[ecs-deploy-runner][2023-03-10T16:48:44+0000] Error: Expected path /tmp/tmph4op5wbi/<redacted>/<redacted>/<redacted>/cdns/<redacted> to exist

To be clear, if no PR is merged before the pipeline finishes, it runs perfectly well. We're not totally sure what's going on, but we suspect the pipe is picking the latest commit in master as a reference when starting each fargate task. When a new commit is added while the pipe runs, it picks this one up, and it doesn't have any of the modules, as they were deleted in a previous commit.

How can we fix this?

r:terraform-aws-ci

---

Tracked in ticket #109987

[gitsstewart]

@umm0n this has been reported by another customer as well and we have an issue for it here https://github.com/gruntwork-io/terraform-aws-service-catalog/issues/1798 as well as an open PR for it here
https://github.com/gruntwork-io/terraform-aws-service-catalog/pull/1799
We are actively working on fixing this.

[MoonMoon1919]

Hi @umm0n, thanks for the report!

We've done extensive testing for a fix for this. In short, we've made a change to calculate the source ref earlier in the process, to ensure that the ref used in the ecs-deploy-runner during delete operations will not change due to subsequent PR merges prior to the delete operation occuring.

You will need to make this same change in the ci configuration for your CI tool of choice (e.g., GH Actions, Gitlab) in your repo. For example, if you're using GH Actions, you would need to modify infrastructure-live/.github/scripts/source-ref.sh.

We are working to update our examples in the service-catalog, but for now, you can see how we accomplished this, in the following PR: https://github.com/gruntwork-io/terraform-aws-service-catalog/pull/1799/files#diff-406976bb201d23780f59dab210db1c29210c5e32fa2e5fe08472236f4e8f611b

[umm0n]

Thanks! We'll implement it straight away.