When resolving Terraform modules from registry, Trivy does not utilize remote service discovery

[marcinbelczewski]

Description

This is the same problem as described in closed discussion #7777 however, here I can provide all the necessary details.

When scanning Terraform code containing definition of a module source from the private Terraform registry, Trivy assumes certain endpoints URLs, which are invalid for example for Terraform registries deployed on JFrog Artifactory.

Desired Behavior

Trivy is able to utilize Terraform HTTP API's Remote Service Discovery to resolve endpoints of private Terraform registries, with URL schemes different than those of public Terraform Registry.

Actual Behavior

Trivy fails with HTTP 404 because it assumes private Terraform registries will have URL scheme exactly the same as public Terraform Registry.

Reproduction Steps

1. Have a Terraform module in private Terraform registry in JFrog Artifactory
2. Create Terraform root module invoking the module:

./main.tf


module "required" {
    source  = "repo.example.com/terraform/my-module"
}


3. Configure authentication token - for example via env variables so that Trivy can authenticate with private registry.

4. Run `trivy config . --debug` and observe following entries in the output log:

```console
DEBUG [terraform evaluator] Locating non-initialized module	source="repo.example.com/terraform/my-module"
DEBUG [module resolver] Resolving module	name="module.required" source="repo.example.com/terraform/my-module"
DEBUG [module resolver] Trying to resolve module via cache	key="f93c531a13cff7905df192c7652d91ff"
DEBUG [module resolver] Found a token for the registry	hostname="repo.example.com"
DEBUG [module resolver] Requesting module source from registry	url="https://repo.example.com/v1/modules/terraform/my-module/download"
ERROR [terraform evaluator] Failed to load module. Maybe try 'terraform init'?	err="unexpected status code: 404"

As can be seen, Trivy assumes https://repo.example.com/v1/modules/ URL scheme for the registry to download the module, however, the actual URL scheme can be learned by utilizing Terraform Registry HTTP API Remote Service Discovery:

❯ curl -H "Authorization: Bearer ${ARTIFACTORY_API_KEY}" \
     "https://repo.example.com/.well-known/terraform.json"
{
  "modules.v1" : "https://repo.example.com/artifactory/api/terraform/v1/modules/",
  "state.v2" : "https://repo.example.com/artifactory/api/terraform/remote/v2",
  "tfe.v2" : "https://repo.example.com/artifactory/api/terraform/remote/v2",
  "tfe.v2.1" : "https://repo.example.com/artifactory/api/terraform/remote/v2",
  "tfe.v2.2" : "https://repo.example.com/artifactory/api/terraform/remote/v2",
  "login.v1" : {
    "client" : "terraform-cli",
    "authz" : "https://repo.example.com/ui/terraform/oauth2/authorize",
    "token" : "https://repo.example.com/artifactory/api/oauth2/token",
    "grant_types" : [ "authz_code" ]
  }
}

As can be seen above, the correct URL scheme to download the module should be:

https://repo.example.com/artifactory/api/terraform/v1/modules/
rather than https://repo.example.com/v1/modules/ Trivy used.

Trivy should preferrably utilize the same discovery mechanism that Terraform CLI does:

DiscoverServiceUrl method from terraform-svchost package

Terraform CLI code for endpoints discovery

### Target

Config

### Scanner

None

### Output Format

None

### Mode

None

### Debug Output

```bash
2024-11-21T19:32:21+01:00	DEBUG	No plugins loaded
2024-11-21T19:32:21+01:00	INFO	Loaded	file_path="trivy.yaml"
2024-11-21T19:32:21+01:00	DEBUG	Cache dir	dir="/Users/marcin.belczewski/Library/Caches/trivy"
2024-11-21T19:32:21+01:00	DEBUG	Cache dir	dir="/Users/marcin.belczewski/Library/Caches/trivy"
2024-11-21T19:32:21+01:00	DEBUG	Parsed severities	severities=[UNKNOWN LOW MEDIUM HIGH CRITICAL]
2024-11-21T19:32:21+01:00	INFO	[misconfig] Misconfiguration scanning is enabled
2024-11-21T19:32:21+01:00	DEBUG	[misconfig] Checks successfully loaded from disk
2024-11-21T19:32:21+01:00	DEBUG	Enabling misconfiguration scanners	scanners=[azure-arm cloudformation dockerfile helm kubernetes terraform terraformplan-json terraformplan-snapshot]
2024-11-21T19:32:21+01:00	DEBUG	Initializing scan cache...	type="memory"
2024-11-21T19:32:21+01:00	DEBUG	Skipping path	path=".git"
2024-11-21T19:32:21+01:00	DEBUG	[misconfig] Scanning files for misconfigurations...	scanner="Terraform"
2024-11-21T19:32:21+01:00	DEBUG	[terraform scanner] Scanning directory	file_path="."
2024-11-21T19:32:22+01:00	DEBUG	[rego] Overriding filesystem for checks
2024-11-21T19:32:22+01:00	DEBUG	[rego] Embedded libraries are loaded	count=15
2024-11-21T19:32:22+01:00	DEBUG	[rego] Embedded checks are loaded	count=509
2024-11-21T19:32:22+01:00	DEBUG	[rego] Checks from disk are loaded	count=524
2024-11-21T19:32:22+01:00	DEBUG	[rego] Overriding filesystem for data
2024-11-21T19:32:22+01:00	DEBUG	[terraform parser] Setting project/module root	module="root" file_path="."
2024-11-21T19:32:22+01:00	DEBUG	[terraform parser] Parsing FS	module="root" file_path="."
2024-11-21T19:32:22+01:00	DEBUG	[terraform parser] Parsing	module="root" file_path="main.tf"
2024-11-21T19:32:22+01:00	DEBUG	[terraform parser] Added file	module="root" file_path="main.tf"
2024-11-21T19:32:22+01:00	INFO	[terraform scanner] Scanning root module	file_path="."
2024-11-21T19:32:22+01:00	DEBUG	[terraform parser] Setting project/module root	module="root" file_path="."
2024-11-21T19:32:22+01:00	DEBUG	[terraform parser] Parsing FS	module="root" file_path="."
2024-11-21T19:32:22+01:00	DEBUG	[terraform parser] Parsing	module="root" file_path="main.tf"
2024-11-21T19:32:22+01:00	DEBUG	[terraform parser] Added file	module="root" file_path="main.tf"
2024-11-21T19:32:22+01:00	DEBUG	[terraform parser] Loading module	module="root" module="root"
2024-11-21T19:32:22+01:00	DEBUG	[terraform parser] Read block(s) and ignore(s)	module="root" blocks=1 ignores=0
2024-11-21T19:32:22+01:00	DEBUG	[terraform parser] Added input variables from tfvars	module="root" count=0
2024-11-21T19:32:22+01:00	DEBUG	[terraform parser] Working directory for module evaluation	module="root" file_path="/Users/marcin.belczewski/spikes/trivy-fun"
2024-11-21T19:32:22+01:00	DEBUG	[terraform evaluator] Starting module evaluation...	path="."
2024-11-21T19:32:22+01:00	DEBUG	[terraform evaluator] Starting iteration	iteration=0
2024-11-21T19:32:22+01:00	DEBUG	[terraform evaluator] Starting iteration	iteration=1
2024-11-21T19:32:22+01:00	DEBUG	[terraform evaluator] Context unchanged	iteration=1
2024-11-21T19:32:22+01:00	DEBUG	[terraform evaluator] Locating non-initialized module	source="repo.example.com/terraform/my-module"
2024-11-21T19:32:22+01:00	DEBUG	[module resolver] Resolving module	name="module.required" source="repo.example.com/terraform/my-module"
2024-11-21T19:32:22+01:00	DEBUG	[module resolver] Trying to resolve module via cache	key="f93c531a13cff7905df192c7652d91ff"
2024-11-21T19:32:22+01:00	DEBUG	[module resolver] Found a token for the registry	hostname="repo.example.com"
2024-11-21T19:32:22+01:00	DEBUG	[module resolver] Requesting module source from registry	url="https://repo.example.com/v1/modules/terraform/my-module/download"
2024-11-21T19:32:22+01:00	ERROR	[terraform evaluator] Failed to load module. Maybe try 'terraform init'?	err="unexpected status code: 404"
2024-11-21T19:32:22+01:00	DEBUG	[terraform evaluator] Starting post-submodules evaluation...
2024-11-21T19:32:22+01:00	DEBUG	[terraform evaluator] Starting iteration	iteration=0
2024-11-21T19:32:22+01:00	DEBUG	[terraform evaluator] Starting iteration	iteration=1
2024-11-21T19:32:22+01:00	DEBUG	[terraform evaluator] Context unchanged	iteration=1
2024-11-21T19:32:22+01:00	DEBUG	[terraform evaluator] Module evaluation complete.
2024-11-21T19:32:22+01:00	DEBUG	[terraform parser] Finished parsing module	module="root"
2024-11-21T19:32:22+01:00	DEBUG	[terraform executor] Adapting modules...
2024-11-21T19:32:22+01:00	DEBUG	[terraform executor] Adapted module(s) into state data.	count=1
2024-11-21T19:32:22+01:00	DEBUG	[terraform executor] Using max routines	count=11
2024-11-21T19:32:22+01:00	DEBUG	[terraform executor] Initialized Go check(s).	count=482
2024-11-21T19:32:22+01:00	DEBUG	[rego] Scanning inputs	count=1
2024-11-21T19:32:22+01:00	DEBUG	[terraform executor] Finished applying rules.
2024-11-21T19:32:22+01:00	DEBUG	[terraform executor] Applying ignores...
2024-11-21T19:32:22+01:00	DEBUG	[misconfig] Scanning files for misconfigurations...	scanner="Helm"
2024-11-21T19:32:22+01:00	DEBUG	[rego] Overriding filesystem for checks
2024-11-21T19:32:22+01:00	DEBUG	[rego] Embedded libraries are loaded	count=15
2024-11-21T19:32:23+01:00	DEBUG	[rego] Embedded checks are loaded	count=509
2024-11-21T19:32:23+01:00	DEBUG	[rego] Checks from disk are loaded	count=524
2024-11-21T19:32:23+01:00	DEBUG	[rego] Overriding filesystem for data
2024-11-21T19:32:23+01:00	DEBUG	OS is not detected.
2024-11-21T19:32:23+01:00	INFO	Detected config files	num=1
2024-11-21T19:32:23+01:00	DEBUG	Scanned config file	file_path="."
2024-11-21T19:32:23+01:00	DEBUG	[vex] VEX filtering is disabled

Operating System

MacOs 14

Version

2024-11-21T19:34:40+01:00	INFO	Loaded	file_path="trivy.yaml"
Version: 0.57.0
Check Bundle:
  Digest: sha256:b381d8e123c2568845a65f751635033051b076e66c460ab0037b4084845c19de
  DownloadedAt: 2024-11-21 08:44:39.470903 +0000 UTC

Checklist

• Run trivy clean --all
• Read the troubleshooting

[nikpivkin]

That's right, Trivy does not currently support remote service discovery.

/cc @simar7 @itaysk

[marcinbelczewski]

I can take a stab at implementing support for remote service discovery - would that be helpful?

[simar7]

@marcinbelczewski my only concern is if RSD implementation has a bug, it'll seem as if Trivy is at fault. If the registries are private, it'll be nearly impossible for us to test and debug such a case.

[marcinbelczewski]

Understood @simar7. Would it make sense to provide opt-in then? Another config flag like --tf-discover-registry-endpoints enabling RSD?

[markekibbe]

I'm experiencing basically what @marcinbelczewski has said

I'm working on trying to migrate from tfsec to trivy currently, and we utilize GitLab self-hosted with GitLab Terraform Registry. I help with devsecops pipelines for a team with hundreds upon hundreds of published internal module versions inside a private registry.

With Trivy, I'm seeing the following log

2024-12-05T00:46:43Z    DEBUG   [module resolver] Resolving module      name="module.deployment" source="gitlab.redacted.com/namespace/internal-module-name/aws"
2024-12-05T00:46:43Z    DEBUG   [module resolver] Found a token for the registry        hostname="gitlab.redacted.com"
2024-12-05T00:46:43Z    DEBUG   [module resolver] Requesting module versions from registry using        url="https://gitlab.redacted.com/v1/modules/namespace/internal-module-name/aws/versions"
2024-12-05T00:46:45Z    ERROR   [terraform evaluator] Failed to load module. Maybe try 'terraform init'?        err="unexpected status code for versions endpoint: 400"

Gitlab's documentation also shows a different endpoint schema https://docs.gitlab.com/ee/api/packages/terraform-modules.html#list-available-versions-for-a-specific-module :

curl --header "Authorization: Bearer <personal_access_token>" "https://gitlab.example.com/api/v4/packages/terraform/modules/v1/group/hello-world/local/versions"

However, if I were to run

export TF_LOG=trace
terraform init

The logs from terraform show the proper versions endpoint:

2024-12-05T00:52:04.852Z [DEBUG] Module installer: begin deployment.redacted_instance
2024-12-05T00:52:04.852Z [TRACE] ModuleInstaller: deployment.redacted_instance is not yet installed
2024-12-05T00:52:04.852Z [TRACE] ModuleInstaller: cleaning directory .terraform/modules/deployment.redacted_instance prior to install of deployment.redacted_instance
2024-12-05T00:52:04.852Z [TRACE] ModuleInstaller: deployment.redacted_instance is a registry module at gitlab.redacted.com/namespace/internal-module-name/aws
2024-12-05T00:52:04.852Z [DEBUG] deployment.redacted_instance listing available versions of gitlab.redacted.com/namespace/internal-module-name/aws at gitlab.redacted.com
2024-12-05T00:52:04.852Z [DEBUG] fetching module versions from "https://gitlab.redacted.com/api/v4/packages/terraform/modules/v1/namespace/internal-module-name/aws/versions"
2024-12-05T00:52:04.852Z [DEBUG] GET https://gitlab.redacted.com/api/v4/packages/terraform/modules/v1/namespace/internal-module-name/aws/versions

would you concur this is the same issue?

[nikpivkin]

Yeah, it's the same problem.

[Krad-dono]

Hello, sooo any updates when this bug will be fixed? We also have private JFrog Artifactory registry setup and have the same error...
Btw - you're not alone in this. Checkov has the same issues...

[simar7]

Apologies this is not a bug but a feature request instead. Currently we don't have any plans to support RSD but we can keep this discussion open.

[Krad-dono]

Understandable, we can look at this as new feature request. So I've two suggestions/points:

1. A lot of larger companies (mine included) use registry caching/proxy solutions like jFrog Artifactory. So it would make sense to have support for those in some capacity. Maybe not everything (as this is clearly can be a huge task) but I think you will definitely can find people in this community (me included) that can help you troubleshoot and run some nightly build with this new feature added and provide logs if needed to make this feature reality and working (at least for jfrog artifactory).
2. If there is no capacity and indeed no appetite to have such functionality added and this tool used by larger corporations then at least I would expect better documentation added to https://trivy.dev/latest/docs/coverage/iac/terraform/ or https://trivy.dev/latest/docs/scanner/misconfiguration/config/config/#private-terraform-registries Now nowhere it is written in documentation that your scanner does not support private registry proxy setup and if you use more modules from https://registry.terraform.io/ that are proxied/cached though artifactory you don't get that code scanned.

Well those are my points. Hope that this makes some sense.

[nikpivkin]

@simar7 FYI there is the svchost library, which contains a client implementation of RSD.