Advice around pre-seeding plugin cache

[jmreicha]

Attempting to leverage Terraform plugin caching in our Terragrunt environment and running into race conditions with concurrency.

I saw a few workarounds in gruntwork-io/terragrunt#1212 but was wondering if there are other ways around the problem as that issue is nearly a year old and I’m not sure if there have been any new features that could help.

I’m thinking maybe an error_hook and/or some kind of retry but could use some other ideas.

---

Tracked in ticket #108687

[yorinasub17]

Unfortunately, we or the community hasn't really made progress on this so what's stated in the issue is still relevant at this point in time.

[yorinasub17]

Nope unfortunately I don't have an example script that automates this. This was more meant to be a manual approach.

[jmreicha]

Oh I see, ok thanks. Does an error_hook approach sound plausible as a work around for this?

[yorinasub17]

error_hook doesn't work here because it's not a retry mechanism.

I think https://terragrunt.gruntwork.io/docs/features/auto-retry/ is what you are looking for, which might work but I haven't tried it so I am not sure.

[jmreicha]

That’s good info, I will see if I can twist this into working. Appreciate the guidance.

[jmreicha]

So it looks like the retry mechanism is working, as I see the failures now multiple times in plan outputs, but it appears I need a way to clear the .terraform.hcl.lock and .terragrunt-cache directories that get created the first time gruntwork-io/terragrunt#1960.

[jmreicha]

Think I'm getting close to a solution with the following, the problem with this is that when you get into a weird state with providers and run a terragrunt plan it never recognizes that it needs to run an init again first to grab the providers it needs. Any ideas?

terraform {
  extra_arguments "defaults" {
    commands  = ["init"]
    arguments = ["-no-color"]
  }

  extra_arguments "cache" {
    commands = [
      "init",
      "apply",
      "refresh",
      "import",
      "plan",
      "taint",
      "untaint",
      "show"
    ]
    env_vars = {
      TF_PLUGIN_CACHE_DIR = "/tmp/plugins"
    }
  }

  error_hook "clear_cache" {
    commands = ["init", "plan", "apply"]
    execute  = ["rm", "-rf", "/tmp/plugins/registry.terraform.io"]
    on_errors = [
      "Inconsistent dependency lock file",
      "Required plugins are not installed",
      "Failed to install provider from shared cache",
    ]
  }
}

retryable_errors = [
  "Inconsistent dependency lock file",
  "Required plugins are not installed",
  "Failed to install provider from shared cache",
]

[jmreicha]

Hrm, it looks like this error_hook is maybe never being called 🤔

[jmreicha]

@yorinasub17 The on_errors block doesn't appear to work with normal regex syntax (as with retryable_errors), is this expected? The only syntax I can get working is ".*" but I can't figure out how to filter for specific errors. Is that how this is supposed to work?

[jmreicha]

Just saw this gruntwork-io/terragrunt#2101.

[jmreicha]

If anyone else runs across this we were able to get the configuration working in newer Terragrunt versions (v0.36.12+).

terraform {
    extra_arguments "cache" {
      commands = [
        "init",
        "apply",
        "refresh",
        "import",
        "plan",
        "taint",
        "untaint",
        "show"
      ]
      env_vars = {
        TF_PLUGIN_CACHE_DIR = "/tmp/plugins"
      }
    }

    error_hook "mirror_providers" {
      commands = ["init", "plan"]
      execute  = ["terragrunt", "providers", "mirror", "/tmp/plugins"]
      on_errors = [
        "Required plugins are not installed",
      ]
    }
  }

  retryable_errors = [
    "Required plugins are not installed",
  ]

  retry_max_attempts = 3
  retry_sleep_interval_sec = 10
  ...
}