[aws] Update inspector data stream for Cloud Detection and Response (CDR) workflow #14306

brijesh-elastic · 2025-06-24T15:13:24Z

Proposed commit message

aws: Update inspector data stream for Cloud Detection and Response (CDR) workflow

Add ECS mappings and latest transform to aws.inspector datastream 
to help with the Cloud Native Vulnerability Management (CNVM)[1] workflow.

Enabled agentless deployment for the `inspector` datastream.
Parse and map the newly introduced fields in the `inspector` datastream.

[1] https://www.elastic.co/guide/en/security/current/vuln-management-overview.html

Note

To Reviewers:

Please refer to this guide: https://docs.elastic.dev/security-solution/cloud-security/cdr/3p-dev-guide which was used for proposed changes.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/aws directory.
Run the following command to run tests.

elastic-package test

Related issues

…CDR) workflow

elasticmachine · 2025-06-24T15:13:29Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

brijesh-elastic · 2025-06-24T15:38:06Z

@maxcold, I've updated the integration title and all other references of AWS Inspector to Amazon Inspector. See Product Page and Documentation Page.

elastic-vault-github-plugin-prod · 2025-06-24T16:58:37Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

packages/aws/_dev/build/docs/inspector.md

packages/aws/changelog.yml

packages/aws/data_stream/inspector/_dev/deploy/docker/files/config.yml

packages/aws/data_stream/inspector/elasticsearch/ingest_pipeline/default.yml

packages/aws/kibana/dashboard/aws-131a1550-5a0b-11ed-a807-bd2da8f2e79b.json

packages/aws/kibana/dashboard/aws-383d4630-63df-11ed-be08-4b4db5223139.json

packages/aws/kibana/dashboard/aws-60881ab0-63e0-11ed-be08-4b4db5223139.json

packages/aws/kibana/dashboard/aws-63984b70-63e1-11ed-be08-4b4db5223139.json

brijesh-elastic · 2025-06-25T06:21:26Z

packages/aws/elasticsearch/transform/latest_cdr_vulnerabilities/transform.yml

+      move_on_creation: true
+latest:
+  unique_key:
+    - aws.inspector.transform_unique_id


@maxcold, I'm using the aws.inspector.transform_unique_id field as the unique key in the transform because if I use individual fields (vulnerability.id, resource.id, package.id, package.version), the transforms behave strangely. The destination index has a higher event count than the source index, which seems to be caused by the fields resource.id, package.id, and package.version being arrays.

@maxcold, here's the logic for calculating aws.inspector.transform_unique_id

integrations/packages/aws/data_stream/inspector/elasticsearch/ingest_pipeline/default.yml

Lines 1150 to 1154 in bbd4c37

- set:

field: aws.inspector.transform_unique_id

tag: set_transform_unique_id

value: '{{vulnerability.id}}|{{resource.id}}|{{package.name}}|{{package.version}}'

if: ctx.aws?.inspector?.type == 'PACKAGE_VULNERABILITY'

thanks! I think it makes sense. I wonder when resource.id can be an array. I guess we had similar situation with AWS Security Hub when resource was defined as array in the docs but we couldn't find any example in the real data. Did you see this case with Amazon Inspector in the wild?
anyway, the unique key logic seems reasonable to me!

In Amazon Inspector also we're getting the single element in resources.

guess we had similar situation with AWS Security Hub when resource was defined as array in the docs but we couldn't find any example in the real data. Did you see this case with Amazon Inspector in the wild?

It is similar to AWS SecurityHub.
Added a comment here: https://github.com/elastic/security-team/issues/10753#issuecomment-2921417448

kcreddy · 2025-06-25T08:21:59Z

packages/aws/_dev/build/docs/inspector.md

@@ -1,15 +1,19 @@
 # Inspector

-The [AWS Inspector](https://docs.aws.amazon.com/inspector/) integration collects and parses data from AWS Inspector [Findings](https://docs.aws.amazon.com/inspector/v2/APIReference/API_ListFindings.html) REST APIs.
+The [Amazon Inspector](https://docs.aws.amazon.com/inspector/) integration collects and parses data from Amazon Inspector [Findings](https://docs.aws.amazon.com/inspector/v2/APIReference/API_ListFindings.html) REST APIs.


@jamiehynds @cpascale43 , we are changing AWS Inspector into Amazon Inspector as it should be with this change. We are only changing UI elements and not updating any field names, so the custom fields are still aws.inspector.*.
Let me know if you have any concerns.

kcreddy · 2025-06-25T08:34:42Z

packages/aws/manifest.yml

@maxcold, current kibana version constraint is following:

kibana: version: "~8.16.6 || ~8.17.4 || ^8.18.0 || ^9.0.0"

In Inspector, we don't have missing CVE values for package vulnerabilities. Also, it is not an array and always contain single values. For example: vulnerability.id: CVE-2025-38000

Do we still need to bump the minimum versions here? Related to multiple/empty CVEs: #14079 (comment)

@kcreddy thanks for highlighting it! I need to test the integration with 8.18 and 9.0 to check that. Right now I'm testing with the latest 9.1 snapshot, will play with 8.18/9.0 version later

When testing 8.18 I realised that we don't support multiple packages in this version and also have some other rough edges. I think we will need to bump to ^8.19.0 || ^9.1.0" but let's wait for product feedback as well

aws integration has shared ownership. If we are completely removing support for 8.17 and 8.18, we need everyone's approval on this.
Is it possible to backport this support multiple packages to 8.17.9 and 8.18.4?

cc: @nick-alayil

@brijesh-elastic, please update the versions to: ^8.19.0 || ^9.1.0 similar to Tenable and Rapid7.

cc: @maxcold

packages/aws/data_stream/inspector/fields/package.yml

packages/aws/data_stream/inspector/fields/vulnerability.yml

packages/aws/data_stream/inspector/fields/resource.yml

packages/aws/data_stream/inspector/elasticsearch/ingest_pipeline/default.yml

kcreddy · 2025-06-25T08:45:36Z

packages/aws/elasticsearch/transform/latest_cdr_vulnerabilities/transform.yml

+      move_on_creation: true
+latest:
+  unique_key:
+    - aws.inspector.transform_unique_id


@maxcold, here's the logic for calculating aws.inspector.transform_unique_id

integrations/packages/aws/data_stream/inspector/elasticsearch/ingest_pipeline/default.yml

Lines 1150 to 1154 in bbd4c37

- set:

field: aws.inspector.transform_unique_id

tag: set_transform_unique_id

value: '{{vulnerability.id}}|{{resource.id}}|{{package.name}}|{{package.version}}'

if: ctx.aws?.inspector?.type == 'PACKAGE_VULNERABILITY'

maxcold · 2025-06-25T08:53:20Z

@kcreddy @brijesh-elastic While testing I see that we don't have the resource.name , I guess it was discussed during the mapping, but I realised that for EC2 instances we can rely on the Name tag (it might not be present, but when it is it represents the instance name). Wdyt about using it as resource.name?

kcreddy · 2025-06-26T10:16:56Z

@kcreddy @brijesh-elastic While testing I see that we don't have the resource.name , I guess it was discussed during the mapping, but I realised that for EC2 instances we can rely on the Name tag (it might not be present, but when it is it represents the instance name). Wdyt about using it as resource.name?

@maxcold, that sounds reasonable. We will also populate host.name based on this tag (only if present) as we don't have that as well.

brijesh-elastic · 2025-06-27T07:21:03Z

@kcreddy @brijesh-elastic While testing I see that we don't have the resource.name , I guess it was discussed during the mapping, but I realised that for EC2 instances we can rely on the Name tag (it might not be present, but when it is it represents the instance name). Wdyt about using it as resource.name?

@maxcold, that sounds reasonable. We will also populate host.name based on this tag (only if present) as we don't have that as well.

Populated resource.name and host.name in the 34a8fd6

kcreddy

Waiting on product confirmation for: #14306 (comment)

packages/aws/data_stream/inspector/_dev/test/pipeline/test-inspector.log-expected.json

packages/aws/data_stream/inspector/elasticsearch/ingest_pipeline/default.yml

packages/aws/data_stream/inspector/_dev/deploy/docker/files/config.yml

packages/aws/changelog.yml

packages/aws/data_stream/inspector/_dev/test/pipeline/test-inspector.log-expected.json

packages/aws/data_stream/inspector/elasticsearch/ingest_pipeline/default.yml

efd6

LGTM, but please wait for @kcreddy.

kcreddy · 2025-07-08T08:11:50Z

packages/aws/data_stream/inspector/_dev/test/pipeline/test-inspector.log-expected.json

+            "package": {
+                "architecture": [
+                    "X86_64"
+                ],
+                "fixed_version": [
+                    "0.38.0",
+                    "0:2.0.5-1.amzn2.0.1"
+                ],
+                "name": [
+                    "golang.org/x/net",
+                    "nerdctl"
+                ],
+                "path": [
+                    "vol-0e47545061282cd35:/p1:opt/cni/bin/aws-cni",
+                    "vol-0e47545061282cd35:/p1:etc/eks/image-credential-provider/ecr-credential-provider",
+                    "vol-0e47545061282cd35:/p1:opt/cni/bin/dhcp",
+                    "vol-0e47545061282cd35:/p1:usr/bin/aws-iam-authenticator",
+                    "vol-0e47545061282cd35:/p1:usr/bin/kubelet"
+                ],
+                "version": [
+                    "v0.1.0",
+                    "v0.30.0",
+                    "2.0.4"
+                ]


@maxcold, the array sizes of package.* fields are not equal. For example, package.name.size() != package.version.size(). Can you confirm if it doesn't cause issues within Findings page?
This happens because the package.* arrays are made not to contain duplicates (for example, golang.org/x/net below).

The nested version of this package field is here.

"package_nested": [ { "epoch": 0, "file_path": "vol-0e47545061282cd35:/p1:opt/cni/bin/aws-cni", "fixed_in_version": "0.38.0", "name": "golang.org/x/net", "package_manager": "GOBINARY", "version": "v0.1.0" }, { "epoch": 0, "file_path": "vol-0e47545061282cd35:/p1:etc/eks/image-credential-provider/ecr-credential-provider", "fixed_in_version": "0.38.0", "name": "golang.org/x/net", "package_manager": "GOBINARY", "version": "v0.30.0" }, { "epoch": 0, "file_path": "vol-0e47545061282cd35:/p1:opt/cni/bin/dhcp", "fixed_in_version": "0.38.0", "name": "golang.org/x/net", "package_manager": "GOBINARY", "version": "v0.30.0" }, { "epoch": 0, "file_path": "vol-0e47545061282cd35:/p1:usr/bin/aws-iam-authenticator", "fixed_in_version": "0.38.0", "name": "golang.org/x/net", "package_manager": "GOBINARY", "version": "v0.30.0" }, { "epoch": 0, "file_path": "vol-0e47545061282cd35:/p1:usr/bin/kubelet", "fixed_in_version": "0.38.0", "name": "golang.org/x/net", "package_manager": "GOBINARY", "version": "v0.30.0" }, { "arch": "X86_64", "epoch": 0, "fixed_in_version": "0:2.0.5-1.amzn2.0.1", "name": "nerdctl", "package_manager": "OS", "release": "1.amzn2.0.1", "remediation": "yum update nerdctl", "version": "2.0.4" } ],

@kcreddy as we don't correlate these arrays it shouldn't be a problem. @alexreal1314 am I correct?

maxcold · 2025-07-09T08:38:17Z

packages/aws/manifest.yml

@@ -11,7 +11,7 @@ conditions:
  elastic:
    subscription: basic
  kibana:
-    version: "~8.16.6 || ~8.17.4 || ^8.18.0 || ^9.0.0"
+    version: "~8.16.6 || ~8.17.4 || ^8.19.0 || ^9.1.0"


@brijesh-elastic @kcreddy as discussed this should be just ^8.19.0 || ^9.1.0, support for 8.16.x and 8.17.x will need to be removed with the adaptation of CDR transform

kcreddy

LGTM. Thanks!

alexreal1314 · 2025-07-09T12:49:01Z

packages/aws/data_stream/inspector/elasticsearch/ingest_pipeline/default.yml

+        if (resources.size() == 1){
+          def res = resources[0];
+
+          if (res.type == 'AWS_EC2_INSTANCE') {


@brijesh-elastic @kcreddy I have a case where resource is not present on the finding, can we have a a case where resources.size() == 1 and res.type !== 'AWS_EC2_INSTANCE'?

might be related to this error im seeing in the document

"error": { "message": "Processor date with tag date_resources_details_awsEc2Instance_launchedAt in pipeline logs-aws.inspector-4.0.0 failed with message: field [awsEc2Instance] not present as part of path [_ingest._value.details.awsEc2Instance.launchedAt]" },

cc @maxcold

important finding @alexreal1314 ! As discussed in our sync, the resource.id is defined as MUST_HAVE mostly because it's used for transform uniqueness in our native integration, but for Amazon Inspector, we chose another field for the uniqueness. If the field can be empty, we need to see if we can fix things that break, eg. opening the flyout from the contextual table. All the rest I think should already be agnostic to not having a resource id

@brijesh-elastic @maxcold @kcreddy PR to handle missing resource id gracefully in main (9.2.0) and 9.1.0 have been merged. Still i'm curious what may be the reason for the missing resource field, this is the document:

{ "agent": { "name": "alex-worker2", "id": "9c0d1500-5aef-4999-9af5-855ff7c37652", "ephemeral_id": "3c4db4e7-9692-46d6-aa4c-4ab00e8105d0", "type": "filebeat", "version": "9.0.3" }, "package": { "fixed_version": [ "0:59.6.0-2.amzn2023.0.6" ], "name": [ "python3-setuptools", "python3-setuptools-wheel" ], "version": [ "59.6.0" ], "architecture": [ "NOARCH" ] }, "elastic_agent": { "id": "9c0d1500-5aef-4999-9af5-855ff7c37652", "version": "9.0.3", "snapshot": false }, "vulnerability": { "reference": [ "https://alas.aws.amazon.com/AL2023/ALAS-2025-1005.html", "https://alas.aws.amazon.com/AL2/ALAS-2025-2877.html", "https://alas.aws.amazon.com/AL2023/ALAS-2025-1003.html", "https://alas.aws.amazon.com/cve/json/v1/CVE-2025-47273.json", "https://alas.aws.amazon.com/AL2023/ALAS-2025-1004.html", "https://alas.aws.amazon.com/AL2/ALAS-2025-2876.html" ], "severity": "High", "score": { "version": "3.1", "base": 7.5 }, "scanner": { "vendor": "Amazon Inspector" }, "description": "setuptools is a package that allows users to download, build, install, upgrade, and uninstall Python packages. A path traversal vulnerability in `PackageIndex` is present in setuptools prior to version 78.1.1. An attacker would be allowed to write files to arbitrary locations on the filesystem with the permissions of the process running the Python code, which could escalate to remote code execution depending on the context. Version 78.1.1 fixes the issue.", "id": "CVE-2025-47273", "title": "CVE-2025-47273 - python3-setuptools, python3-setuptools-wheel", "published_date": "2025-05-17T00:00:00.000Z" }, "error": { "message": "Processor date with tag date_resources_details_awsEc2Instance_launchedAt in pipeline logs-aws.inspector-4.0.0 failed with message: field [awsEc2Instance] not present as part of path [_ingest._value.details.awsEc2Instance.launchedAt]" }, "message": "setuptools is a package that allows users to download, build, install, upgrade, and uninstall Python packages. A path traversal vulnerability in `PackageIndex` is present in setuptools prior to version 78.1.1. An attacker would be allowed to write files to arbitrary locations on the filesystem with the permissions of the process running the Python code, which could escalate to remote code execution depending on the context. Version 78.1.1 fixes the issue.", "tags": [ "forwarded", "aws-inspector", "preserve_original_event" ], "cloud": { "provider": "aws", "account": { "id": "704479110758" } }, "input": { "type": "httpjson" }, "observer": { "vendor": "Amazon Inspector" }, "@timestamp": "2025-06-26T12:42:32.515Z", "ecs": { "version": "8.11.0" }, "related": { "hash": [ "sha256:023cba81b02358aa89023184475accbaf4d8b7edba68d1c8981e46747029cc80", "sha256:f2e0a07f71934e38981adf8055d7ac9a0562b6b33ef05c5a9be25f6efc9a5bca" ] }, "data_stream": { "namespace": "default", "type": "logs", "dataset": "aws.inspector" }, "host": { "name": "classic-rim.net" }, "aws": { "inspector": { "severity": "HIGH", "inspector_score": 7.5, "package_vulnerability_details": { "related_vulnerabilities": [ "ALAS2-2025-2877", "ALAS2-2025-2876", "ALAS2023-2025-1004", "ALAS2023-2025-1005", "ALAS2023-2025-1003" ], "vendor": { "severity": "Important", "updated_at": "2025-06-02T00:00:00.000Z" }, "vulnerable_packages": [ { "remediation": "sudo dnf check-update", "package_manager": "OS", "release": "2.amzn2023.0.5", "name": "python3-setuptools", "epoch": 0, "fixed_in_version": "0:59.6.0-2.amzn2023.0.6", "source_layer_hash": "sha256:023cba81b02358aa89023184475accbaf4d8b7edba68d1c8981e46747029cc80", "arch": "NOARCH", "version": "59.6.0" }, { "remediation": "sudo dnf check-update", "package_manager": "OS", "release": "2.amzn2023.0.5", "name": "python3-setuptools-wheel", "epoch": 0, "fixed_in_version": "0:59.6.0-2.amzn2023.0.6", "source_layer_hash": "sha256:023cba81b02358aa89023184475accbaf4d8b7edba68d1c8981e46747029cc80", "arch": "NOARCH", "version": "59.6.0" } ], "source": { "value": "AMAZON_CVE", "url": { "path": "/cve/json/v1/CVE-2025-47273.json", "extension": "json", "original": "https://alas.aws.amazon.com/cve/json/v1/CVE-2025-47273.json", "scheme": "https", "domain": "alas.aws.amazon.com" } }, "cvss": [ { "base_score": 7.5, "scoring_vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:H/A:N", "source": "AMAZON_CVE", "version": "3.1" }, { "base_score": 8.8, "scoring_vector": "CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H", "source": "NVD", "version": "3.1" } ] }, "epss": { "score": 0.0012 }, "package_nested": [ { "remediation": "sudo dnf check-update", "package_manager": "OS", "release": "2.amzn2023.0.5", "name": "python3-setuptools", "epoch": 0, "fixed_in_version": "0:59.6.0-2.amzn2023.0.6", "source_layer_hash": "sha256:023cba81b02358aa89023184475accbaf4d8b7edba68d1c8981e46747029cc80", "arch": "NOARCH", "version": "59.6.0" }, { "remediation": "sudo dnf check-update", "package_manager": "OS", "release": "2.amzn2023.0.5", "name": "python3-setuptools-wheel", "epoch": 0, "fixed_in_version": "0:59.6.0-2.amzn2023.0.6", "source_layer_hash": "sha256:023cba81b02358aa89023184475accbaf4d8b7edba68d1c8981e46747029cc80", "arch": "NOARCH", "version": "59.6.0" } ], "inspector_score_details": { "adjusted_cvss": { "score": { "source": "AMAZON_CVE" }, "scoring_vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:H/A:N", "cvss_source": "AMAZON_CVE" } }, "resources": [ { "partition": "aws", "details": { "aws": { "ecr_container_image": { "image": { "hash": "sha256:f2e0a07f71934e38981adf8055d7ac9a0562b6b33ef05c5a9be25f6efc9a5bca", "tags": [ "latest" ] }, "registry": "704479110758", "pushed_at": "2025-04-17T12:24:47.924Z", "platform": "AMAZON_LINUX_2023", "repository_name": "orestis-onweek-2", "architecture": "amd64" } } }, "id": "arn:aws:ecr:us-east-1:704479110758:repository/orestis-onweek-2/sha256:f2e0a07f71934e38981adf8055d7ac9a0562b6b33ef05c5a9be25f6efc9a5bca", "region": "us-east-1", "type": "AWS_ECR_CONTAINER_IMAGE" } ], "type": "PACKAGE_VULNERABILITY", "transform_unique_id": "CVE-2025-47273||{0=python3-setuptools, 1=python3-setuptools-wheel}|{0=59.6.0}", "finding_arn": "arn:aws:inspector2:us-east-1:704479110758:finding/194f71676960026044e67e7c1cf1f0a5", "exploit_available": "NO", "remediation": { "recommendation": { "text": "None Provided" } }, "last_observed_at": "2025-06-26T12:42:32.515Z", "fix_available": "YES", "first_observed_at": "2025-06-13T08:46:12.050Z", "status": "ACTIVE" } }, "event": { "agent_id_status": "auth_metadata_missing", "ingested": "2025-07-09T12:05:54Z", "original": "{\"awsAccountId\":\"704479110758\",\"description\":\"setuptools is a package that allows users to download, build, install, upgrade, and uninstall Python packages. A path traversal vulnerability in `PackageIndex` is present in setuptools prior to version 78.1.1. An attacker would be allowed to write files to arbitrary locations on the filesystem with the permissions of the process running the Python code, which could escalate to remote code execution depending on the context. Version 78.1.1 fixes the issue.\",\"epss\":{\"score\":0.0012},\"exploitAvailable\":\"NO\",\"findingArn\":\"arn:aws:inspector2:us-east-1:704479110758:finding/194f71676960026044e67e7c1cf1f0a5\",\"firstObservedAt\":1749804372.05,\"fixAvailable\":\"YES\",\"inspectorScore\":7.5,\"inspectorScoreDetails\":{\"adjustedCvss\":{\"adjustments\":[],\"cvssSource\":\"AMAZON_CVE\",\"score\":7.5,\"scoreSource\":\"AMAZON_CVE\",\"scoringVector\":\"CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:H/A:N\",\"version\":\"3.1\"}},\"lastObservedAt\":1750941752.515,\"packageVulnerabilityDetails\":{\"cvss\":[{\"baseScore\":7.5,\"scoringVector\":\"CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:H/A:N\",\"source\":\"AMAZON_CVE\",\"version\":\"3.1\"},{\"baseScore\":8.8,\"scoringVector\":\"CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H\",\"source\":\"NVD\",\"version\":\"3.1\"}],\"referenceUrls\":[\"https://alas.aws.amazon.com/AL2023/ALAS-2025-1005.html\",\"https://alas.aws.amazon.com/AL2/ALAS-2025-2877.html\",\"https://alas.aws.amazon.com/AL2023/ALAS-2025-1003.html\",\"https://alas.aws.amazon.com/cve/json/v1/CVE-2025-47273.json\",\"https://alas.aws.amazon.com/AL2023/ALAS-2025-1004.html\",\"https://alas.aws.amazon.com/AL2/ALAS-2025-2876.html\"],\"relatedVulnerabilities\":[\"ALAS2-2025-2877\",\"ALAS2-2025-2876\",\"ALAS2023-2025-1004\",\"ALAS2023-2025-1005\",\"ALAS2023-2025-1003\"],\"source\":\"AMAZON_CVE\",\"sourceUrl\":\"https://alas.aws.amazon.com/cve/json/v1/CVE-2025-47273.json\",\"vendorCreatedAt\":1747440000,\"vendorSeverity\":\"Important\",\"vendorUpdatedAt\":1748822400,\"vulnerabilityId\":\"CVE-2025-47273\",\"vulnerablePackages\":[{\"arch\":\"NOARCH\",\"epoch\":0,\"fixedInVersion\":\"0:59.6.0-2.amzn2023.0.6\",\"name\":\"python3-setuptools\",\"packageManager\":\"OS\",\"release\":\"2.amzn2023.0.5\",\"remediation\":\"sudo dnf check-update\",\"sourceLayerHash\":\"sha256:023cba81b02358aa89023184475accbaf4d8b7edba68d1c8981e46747029cc80\",\"version\":\"59.6.0\"},{\"arch\":\"NOARCH\",\"epoch\":0,\"fixedInVersion\":\"0:59.6.0-2.amzn2023.0.6\",\"name\":\"python3-setuptools-wheel\",\"packageManager\":\"OS\",\"release\":\"2.amzn2023.0.5\",\"remediation\":\"sudo dnf check-update\",\"sourceLayerHash\":\"sha256:023cba81b02358aa89023184475accbaf4d8b7edba68d1c8981e46747029cc80\",\"version\":\"59.6.0\"}]},\"remediation\":{\"recommendation\":{\"text\":\"None Provided\"}},\"resources\":[{\"details\":{\"awsEcrContainerImage\":{\"architecture\":\"amd64\",\"imageHash\":\"sha256:f2e0a07f71934e38981adf8055d7ac9a0562b6b33ef05c5a9be25f6efc9a5bca\",\"imageTags\":[\"latest\"],\"platform\":\"AMAZON_LINUX_2023\",\"pushedAt\":1744892687.924,\"registry\":\"704479110758\",\"repositoryName\":\"orestis-onweek-2\"}},\"id\":\"arn:aws:ecr:us-east-1:704479110758:repository/orestis-onweek-2/sha256:f2e0a07f71934e38981adf8055d7ac9a0562b6b33ef05c5a9be25f6efc9a5bca\",\"partition\":\"aws\",\"region\":\"us-east-1\",\"tags\":{},\"type\":\"AWS_ECR_CONTAINER_IMAGE\"}],\"severity\":\"HIGH\",\"status\":\"ACTIVE\",\"title\":\"CVE-2025-47273 - python3-setuptools, python3-setuptools-wheel\",\"type\":\"PACKAGE_VULNERABILITY\",\"updatedAt\":1750941752.515}", "created": "2025-07-09T09:57:27.977Z", "kind": "pipeline_error", "id": "CVE-2025-47273||{0=python3-setuptools, 1=python3-setuptools-wheel}|{0=59.6.0}|2025-06-26T12:42:32.515Z", "category": [ "vulnerability" ], "type": [ "info" ], "dataset": "aws.inspector" } }

"error": { "message": "Processor date with tag date_resources_details_awsEc2Instance_launchedAt in pipeline logs-aws.inspector-4.0.0 failed with message: field [awsEc2Instance] not present as part of path [_ingest._value.details.awsEc2Instance.launchedAt]" },

This error is not related to the case mentioned, but will resolve it next commit.

I have a case where resource is not present on the finding, can we have a a case where resources.size() == 1 and res.type !== 'AWS_EC2_INSTANCE'?

@alexreal1314, what is the expected fields/logic when resources.size() == 1 and res.type !== 'AWS_EC2_INSTANCE'? populating resource.id with aws.inspector.resources[].id?

PR to handle missing resource id gracefully in main (9.2.0) and 9.1.0 have been merged. Still i'm curious what may be the reason for the missing resource field, this is the document:

resource.id is missing because aws.inspector.resources[].type = "AWS_ECR_CONTAINER_IMAGE".

@brijesh-elastic regarding:

@alexreal1314, what is the expected fields/logic when resources.size() == 1 and res.type !== 'AWS_EC2_INSTANCE'? populating resource.id with aws.inspector.resources[].id?

i don't have the documentation of aws inspector so im no 100% sure, but looking at the sample events provided looks like yes, why do we check for res.type before that?

Can the issue stem from the fact that we want host.id and host.name only for EC2 instances?

Yes, I think that may've been the case here. @brijesh-elastic , we should populate resource.id and resource.type for all resources from resources[].id and resources[].type. Similar to Security Hub: https://github.com/elastic/integrations/blob/main/packages/aws/data_stream/securityhub_findings/elasticsearch/ingest_pipeline/default.yml#L1604-L1605

cloud.instance.id (from res.id)

cloud.instance.id should only be populated when res.Type = AWS_EC2_INSTANCE. I think you got the rest.

Thanks @kcreddy, @maxcold, @alexreal1314
Resolve in the commit.

@alexreal1314, I got confirmation from @nick-alayil that it is okay to remove the transform part from this PR and add it close to 9.2.0 release.
Can you confirm if this issue can be marked resolved? Are there any more tests you would like to do before we remove transform?

@kcreddy @brijesh-elastic @maxcold issue is resolved, there where no vulnerabilities with missing resource field. Thanks everyone.

Awesome, thanks @alexreal1314 for confirming

@brijesh-elastic, you can go ahead and remove the transform from the PR and any changes related to it (version constraint, documentation, etc.)

efd6 · 2025-07-14T01:52:50Z

packages/aws/changelog.yml

+    - description: Update the kibana constraint to support ^8.19.0 || ^9.1.0
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/14306


I don't think this is needed, and as it exists is misleading. The text here says (suggests?) that we are adding support for 8.19.0 and 9.1.0, but this is not what we are doing, we are restricting to these. I would instead suggest that this be removed since the behaviour is already implicitly documented by the integrations addition page for users who care about this.

elastic-sonarqube · 2025-07-14T08:30:42Z

Quality Gate failed

Failed conditions
68.3% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

elasticmachine · 2025-07-14T08:30:48Z

💚 Build Succeeded

Buildkite Build
Commit: 66c196a

History

💚 Build #28414 succeeded 740956b
💚 Build #28373 succeeded ec76fc1
💔 Build #28258 failed 4c117c1
💚 Build #28222 succeeded 70439b2
💚 Build #28139 succeeded 91bc1d0
💔 Build #28035 failed 34fca8d

cc @brijesh-elastic

Update inspector datastream to support Cloud Detection and Response (…

9ff387c

…CDR) workflow

brijesh-elastic self-assigned this Jun 24, 2025

brijesh-elastic added the Integration:aws AWS label Jun 24, 2025

brijesh-elastic requested review from a team as code owners June 24, 2025 15:13

Update changelog entry

bbd4c37

efd6 reviewed Jun 25, 2025

View reviewed changes

brijesh-elastic commented Jun 25, 2025

View reviewed changes

kcreddy requested a review from a team June 25, 2025 08:18

kcreddy reviewed Jun 25, 2025

View reviewed changes

address pr comment

671472d

brijesh-elastic requested review from efd6 and kcreddy June 26, 2025 09:59

brijesh-elastic and others added 2 commits June 27, 2025 12:48

Populate resource.name and host.name when tags[].Name present

34a8fd6

Merge branch 'main' into aws-4.0.0

b9b2494

brijesh-elastic requested a review from maxcold June 27, 2025 07:26

kcreddy reviewed Jul 1, 2025

View reviewed changes

andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Jul 1, 2025

Merge branch 'main' into aws-4.0.0

729c0ce

update null reference condition in scripts

75cb248

andrewkroh added bugfix Pull request that fixes a bug issue enhancement New feature or request labels Jul 2, 2025

efd6 reviewed Jul 2, 2025

View reviewed changes

address pr comment

34fca8d

brijesh-elastic requested review from kcreddy and efd6 July 3, 2025 09:14

Merge branch 'main' into aws-4.0.0

91bc1d0

efd6 approved these changes Jul 7, 2025

View reviewed changes

kcreddy reviewed Jul 8, 2025

View reviewed changes

brijesh-elastic added 2 commits July 8, 2025 16:14

Merge branch 'main' into aws-4.0.0

29880df

Update the kibana constraint to support ^8.19.0 || ^9.1.0

70439b2

maxcold reviewed Jul 9, 2025

View reviewed changes

brijesh-elastic and others added 2 commits July 9, 2025 14:33

update kibana constraints

31c5699

Merge branch 'main' into aws-4.0.0

4c117c1

kcreddy approved these changes Jul 9, 2025

View reviewed changes

alexreal1314 reviewed Jul 9, 2025

View reviewed changes

brijesh-elastic and others added 2 commits July 10, 2025 19:09

address pr comments and skip the guardduty system test to pass the ci

ec76fc1

Merge branch 'main' into aws-4.0.0

740956b

efd6 reviewed Jul 14, 2025

View reviewed changes

remove changelog entry related to kibana constraint

66c196a

alexreal1314 approved these changes Jul 14, 2025

View reviewed changes

	- set:
	field: aws.inspector.transform_unique_id
	tag: set_transform_unique_id
	value: '{{vulnerability.id}}\|{{resource.id}}\|{{package.name}}\|{{package.version}}'
	if: ctx.aws?.inspector?.type == 'PACKAGE_VULNERABILITY'

[aws] Update inspector data stream for Cloud Detection and Response (CDR) workflow #14306

Are you sure you want to change the base?

[aws] Update inspector data stream for Cloud Detection and Response (CDR) workflow #14306

Conversation

brijesh-elastic commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Related issues

Uh oh!

elasticmachine commented Jun 24, 2025

Uh oh!

brijesh-elastic commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kcreddy Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kcreddy Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kcreddy Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maxcold commented Jun 25, 2025

Uh oh!

kcreddy commented Jun 26, 2025

Uh oh!

brijesh-elastic commented Jun 27, 2025

Uh oh!

kcreddy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

brijesh-elastic commented Jun 24, 2025 •

edited

Loading

brijesh-elastic commented Jun 24, 2025 •

edited

Loading

elastic-vault-github-plugin-prod bot commented Jun 24, 2025 •

edited

Loading

kcreddy Jun 25, 2025 •

edited

Loading

kcreddy Jun 25, 2025 •

edited

Loading

kcreddy Jun 26, 2025 •

edited

Loading

alexreal1314 Jul 9, 2025 •

edited

Loading

kcreddy Jul 10, 2025 •

edited

Loading

kcreddy Jul 11, 2025 •

edited

Loading