[Google Threat Intelligence] Add Integration Package #13189

niraj-crest · 2025-03-20T12:12:20Z

Enhancement

What does this PR do?

Added 4 data streams (cryptominer, first_stage_delivery_vectors, infostealer, iot).
Added data collection logic for the data streams.
Added the ingest pipeline for the data streams.
Mapped fields according to the ECS schema and added Fields metadata in the appropriate yaml files.
Added dashboards and visualizations.
Added system test cases for the data stream.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Clone integrations repo.
Install elastic-package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/ti_google_threat_intelligence) directory.
Run the following command to run tests. elastic-package test

elasticmachine · 2025-03-20T16:10:47Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

niraj-crest · 2025-03-26T06:02:42Z

Hey @jamiehynds & @andrewkroh
Can anyone please add reviewers to this PR?

kcreddy · 2025-03-28T09:01:56Z

/test

kcreddy

@niraj-crest, could you fix the CI error?

Error: there is no owner for "packages/ti_google_threat_intelligence" in ".github/CODEOWNERS"

by adding a line into .github/CODEOWNERS

niraj-crest · 2025-04-01T04:45:05Z

Hello @kcreddy,
Added codeowners, can please run the tests again?

kcreddy · 2025-04-01T05:29:29Z

/test

elastic-vault-github-plugin-prod · 2025-04-01T05:48:42Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

kcreddy · 2025-04-01T09:54:53Z

@niraj-crest, looks like the PR is failing Check for blocked path changes.
The git diff git diff --name-only f9794072abccfc51e4549ecd7510d0d2248f2c72 1b3d6ddc5affe5f7215b2668ecdd1db7682202e9 is identifying files from blocked_paths.

Can you re-sync your branch with elastic:main?

niraj-crest · 2025-04-02T05:40:14Z

@niraj-crest, looks like the PR is failing Check for blocked path changes. The git diff git diff --name-only f9794072abccfc51e4549ecd7510d0d2248f2c72 1b3d6ddc5affe5f7215b2668ecdd1db7682202e9 is identifying files from blocked_paths.

Can you re-sync your branch with elastic:main?

Done @kcreddy.

kcreddy · 2025-04-02T05:57:09Z

@niraj-crest, I don't think its done.

kcreddy · 2025-04-02T09:10:20Z

/test

elastic-sonarqube · 2025-04-02T09:37:47Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
82.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

niraj-crest · 2025-04-03T08:55:21Z

Hello @kcreddy,
CI is passing now, i guess you can start reviewing the PR.

packages/ti_google_threat_intelligence/_dev/build/build.yml

...elligence/data_stream/first_stage_delivery_vectors/elasticsearch/ingest_pipeline/default.yml

packages/ti_google_threat_intelligence/_dev/build/docs/README.md

packages/ti_google_threat_intelligence/data_stream/cryptominer/agent/stream/cel.yml.hbs

kcreddy · 2025-04-03T10:30:16Z

@kcreddy,
CI is passing now, i guess you can start reviewing the PR.

@niraj-crest I've added few comments. I haven't looked into the entire PR yet. Will do.

packages/ti_google_threat_intelligence/_dev/deploy/docker/files/config.yml

packages/ti_google_threat_intelligence/data_stream/cryptominer/agent/stream/cel.yml.hbs

...google_threat_intelligence/data_stream/cryptominer/elasticsearch/ingest_pipeline/default.yml

...t_intelligence/data_stream/cryptominer/_dev/test/pipeline/test-cryptominer.log-expected.json

packages/ti_google_threat_intelligence/_dev/build/docs/README.md

packages/ti_google_threat_intelligence/elasticsearch/transform/domain_ioc/transform.yml

kcreddy · 2025-06-04T10:29:26Z

/test

niraj-crest · 2025-06-04T10:40:03Z

Hello @kcreddy / @efd6 /@andrewkroh
Please let me know if we have any other review comments.

efd6 · 2025-06-04T22:56:33Z

packages/ti_google_threat_intelligence/data_stream/cryptominer/agent/stream/cel.yml.hbs

+  state.with({
+    "start_time": state.?cursor.last_timestamp.orValue((now - duration(state.initial_interval)).format("2006010215"))
+  }).as(state, state.with(


Suggested change

state.with({

"start_time": state.?cursor.last_timestamp.orValue((now - duration(state.initial_interval)).format("2006010215"))

}).as(state, state.with(

state.?cursor.last_timestamp.orValue((now - duration(state.initial_interval)).format("2006010215")).as(start_time,

since this is overwritten for every eval it doesn't need to be put in state. This will need bracket fixups.

Though, I would also reconsider doing the format here. Instead, that way you don't need to reparse each time you make an adjustment.

Thank you for the suggestion!
We’ve updated the code to use start_time directly as recommended, which definitely helped simplify things.

Regarding your additional point about reconsidering the formatting, we explored that approach but ran into a timing issue due to the API's behavior. Specifically, the API enforces a 1-hour delay, and if we don’t round precisely to the hour, we risk calling a future window too early.

For example, if the current time is 10:21 and we start with an initial interval of 2h, we’ll fetch data for 08 and 09. When checking whether to proceed to the next hour, (09:21 + 1h) becomes 10:21, which might still be less than now (say 10:22) by the time the check happens. This leads us to attempt a 10:00 call prematurely and receive an error.

To avoid this, we’ve kept the strict hour-to-hour comparison using formatted values.
Please let us know if you have any further thoughts.

OK. That is also unfortunate. What we'd really like is a timestamp round function. elastic/mito#92

efd6 · 2025-06-04T22:57:27Z

packages/ti_google_threat_intelligence/data_stream/cryptominer/agent/stream/cel.yml.hbs

+  }).as(state, state.with(
+    request(
+      "GET",
+      state.url.trim_right("/") + "/api/v3/threat_lists/cryptominer/" + state.start_time + "?" + {


Suggested change

state.url.trim_right("/") + "/api/v3/threat_lists/cryptominer/" + state.start_time + "?" + {

state.url.trim_right("/") + "/api/v3/threat_lists/cryptominer/" + start_time + "?" + {

packages/ti_google_threat_intelligence/data_stream/cryptominer/agent/stream/cel.yml.hbs

efd6 · 2025-06-04T22:58:18Z

packages/ti_google_threat_intelligence/data_stream/cryptominer/agent/stream/cel.yml.hbs

+        "cursor": {
+          "last_timestamp": (state.start_time.parse_time("2006010215") + duration("1h")).format("2006010215"),
+        },
+        "want_more": (state.start_time.parse_time("2006010215") + duration("1h")) < (now.format("2006010215")).parse_time("2006010215")


Suggested change

"want_more": (state.start_time.parse_time("2006010215") + duration("1h")) < (now.format("2006010215")).parse_time("2006010215")

"want_more": (start_time.parse_time("2006010215") + duration("1h")) < (now.format("2006010215")).parse_time("2006010215")

efd6 · 2025-06-04T22:59:59Z

...google_threat_intelligence/data_stream/first_stage_delivery_vectors/agent/stream/cel.yml.hbs

Same comments as above.

… manifest and CEL

niraj-crest · 2025-06-06T09:35:08Z

Hello @kcreddy / @efd6 ,
We’ve addressed all the review comments.
Kindly let us know if you have any additional suggestions. If everything looks good to you, we’d appreciate it if we could proceed with merging this PR.

kcreddy · 2025-06-09T09:53:38Z

/test

packages/ti_google_threat_intelligence/elasticsearch/transform/rule/transform.yml

kcreddy · 2025-06-09T10:05:16Z

packages/ti_google_threat_intelligence/elasticsearch/transform/rule/transform.yml

+      must_not:
+        - match_phrase:
+            kibana.alert.rule.tags: IOC Stream


This is indeed confusing as this PR change is dependent on future PRs. We should avoid this if possible in the future.

What is the difference between IOCs ingested from this PR's API i..e., Get Hourly Threat List vs the /ioc_stream API thats added in #13449?
Is one API better than the other, if so why don't we add only the better one?

kcreddy · 2025-06-09T10:09:18Z

packages/ti_google_threat_intelligence/elasticsearch/transform/ip_ioc/fields/iot_fields.yml

@@ -0,0 +1,264 @@
+- name: gti


Please kindly make a note of all these after items and create a follow up PR. Thanks!

packages/ti_google_threat_intelligence/elasticsearch/transform/domain_ioc/fields/ecs.yml

packages/ti_google_threat_intelligence/elasticsearch/transform/url_ioc/fields/ecs.yml

kcreddy · 2025-06-09T10:21:41Z

packages/ti_google_threat_intelligence/elasticsearch/transform/ip_ioc/transform.yml

+  # Bump this version to delete, reinstall, and restart the transform during
+  # package installation.
+  fleet_transform_version: 1.0.0
+  run_as_kibana_system: false


After adding the pipeline, the transform turns Healthy as expected.

This behaviour is very strange. run_as_kibana_system shouldn't impact the transform health from a pipeline not being present, especially when we don't even have a pipeline by default #13189 (comment).

If its not too much, could you please kindly add a video capturing this behaviour?

packages/ti_google_threat_intelligence/elasticsearch/transform/rule/transform.yml

efd6

My concerns have been addressed, but please wait for @kcreddy.

efd6 · 2025-06-12T23:38:09Z

packages/ti_google_threat_intelligence/data_stream/cryptominer/agent/stream/cel.yml.hbs

+        ?"query": has(state.query) ? optional.of([state.query]) : optional.none(),
+        "limit": ["4000"],
+        "x-tool": ["Elastic"],
+        "User-Agent": ["v0.1.0"],  // Include integration version here; must match the 'version' in package level manifest.yml


Suggested change

"User-Agent": ["v0.1.0"], // Include integration version here; must match the 'version' in package level manifest.yml

"User-Agent": ["v0.1.0"], // Keep this in sync with 'version' in package level manifest.yml.

Same change elsewhere that this comment exists.

efd6 · 2025-06-12T23:39:30Z

packages/ti_google_threat_intelligence/manifest.yml

+# This version must match the User-Agent version used in CEL code.
+# Remember to update the User-Agent in CEL code when changing this version.


efd6 · 2025-06-12T23:53:31Z

packages/ti_google_threat_intelligence/data_stream/cryptominer/agent/stream/cel.yml.hbs

+  state.with({
+    "start_time": state.?cursor.last_timestamp.orValue((now - duration(state.initial_interval)).format("2006010215"))
+  }).as(state, state.with(


OK. That is also unfortunate. What we'd really like is a timestamp round function. elastic/mito#92

… event.kind in enriched index

kcreddy · 2025-06-17T06:05:00Z

/test

kcreddy

Only waiting on 1 outstanding comment: #13189 (comment)

kcreddy · 2025-06-23T05:49:01Z

/test

elasticmachine · 2025-06-23T06:12:57Z

💚 Build Succeeded

Buildkite Build
Commit: addde76

History

💚 Build #27316 succeeded e3bab0f
💚 Build #27046 succeeded b4ebf3e
💚 Build #26864 succeeded 87040d1
💚 Build #26467 succeeded 595a25b
💚 Build #26231 succeeded 86489a7
💚 Build #26104 succeeded 8b256a6

elastic-sonarqube · 2025-06-23T06:13:06Z

Quality Gate failed

Failed conditions
64.3% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

kcreddy

LGTM. Thank you!
#14287 tracks list of pending issues for making the integration GA.

elastic-vault-github-plugin-prod · 2025-06-23T07:16:48Z

Package ti_google_threat_intelligence - 0.1.0 containing this change is available at https://epr.elastic.co/package/ti_google_threat_intelligence/0.1.0/

- Added 4 data streams (cryptominer, first_stage_delivery_vectors, infostealer, iot). - Added data collection logic for the data streams. - Added the ingest pipeline for the data streams. - Mapped fields according to the ECS schema and added Fields metadata in the appropriate yaml files. - Added dashboards and visualizations. - Added system test cases for the data stream.

niraj-crest added 4 commits March 20, 2025 16:07

add ti_google_threat_intelligence package

525f1eb

update readme.md

a98f5b1

update changelog entry

3e01aa8

update changelog

aa66795

andrewkroh added New Integration Issue or pull request for creating a new integration package. dashboard Relates to a Kibana dashboard bug, enhancement, or modification. labels Mar 20, 2025

jamiehynds added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Mar 20, 2025

update screenshots of dashboards

5e45c90

jamiehynds requested a review from a team March 26, 2025 08:25

andrewkroh added the Crest Contributions from Crest developement team. label Mar 27, 2025

kcreddy reviewed Mar 28, 2025

View reviewed changes

add codeowner

1b3d6dd

niraj-crest force-pushed the ti_google_threat_intelligence branch from 0078531 to 1b3d6dd Compare April 1, 2025 04:42

Merge branch 'elastic:main' into ti_google_threat_intelligence

d976d10

niraj-crest mentioned this pull request Apr 3, 2025

[Google Threat Intelligence] Add linux, malicious_network_infrastructure, malware, mobile and osx data streams #13190

Merged

4 tasks

kcreddy reviewed Apr 3, 2025

View reviewed changes

kcreddy reviewed Apr 4, 2025

View reviewed changes

efd6 reviewed Jun 4, 2025

View reviewed changes

Refactor CEL to use start_time directly; add version sync comments in…

0e16360

… manifest and CEL

niraj-crest requested a review from efd6 June 5, 2025 11:42

minor changes in readme.

b4ebf3e

kcreddy reviewed Jun 9, 2025

View reviewed changes

efd6 mentioned this pull request Jun 12, 2025

Add timestamp round method to Time library elastic/mito#92

Closed

efd6 approved these changes Jun 13, 2025

View reviewed changes

Update rule transform source index, add fields in ecs.yml, and update…

e3bab0f

… event.kind in enriched index

niraj-crest requested a review from kcreddy June 13, 2025 06:10

kcreddy reviewed Jun 17, 2025

View reviewed changes

niraj-crest requested a review from kcreddy June 19, 2025 06:08

niraj-crest mentioned this pull request Jun 20, 2025

Source Index Queries in Transform not Applied During System Tests elastic/elastic-package#2676

Open

niraj-crest added 2 commits June 20, 2025 18:42

Add issue in transform.yml comment

8cc2a7c

update comments on transform.yml

addde76

This was referenced Jun 23, 2025

[ti_google_threat_intelligence] Remove manual pipeline and field mappings once pipeline-transform dependency handling is supported #14286

Open

[ti_google_threat_intelligence] Promote package to GA after resolving pending issues #14287

Open

kcreddy approved these changes Jun 23, 2025

View reviewed changes

kcreddy merged commit 01ed0b5 into elastic:main Jun 23, 2025
6 of 7 checks passed

andrewkroh added the Integration:ti_google_threat_intelligence Google Threat Intelligence (Partner supported) label Jun 23, 2025

andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Jul 1, 2025

	state.url.trim_right("/") + "/api/v3/threat_lists/cryptominer/" + state.start_time + "?" + {
	state.url.trim_right("/") + "/api/v3/threat_lists/cryptominer/" + start_time + "?" + {

	"want_more": (state.start_time.parse_time("2006010215") + duration("1h")) < (now.format("2006010215")).parse_time("2006010215")
	"want_more": (start_time.parse_time("2006010215") + duration("1h")) < (now.format("2006010215")).parse_time("2006010215")

	"User-Agent": ["v0.1.0"], // Include integration version here; must match the 'version' in package level manifest.yml
	"User-Agent": ["v0.1.0"], // Keep this in sync with 'version' in package level manifest.yml.

		# This version must match the User-Agent version used in CEL code.
		# Remember to update the User-Agent in CEL code when changing this version.

[Google Threat Intelligence] Add Integration Package #13189

[Google Threat Intelligence] Add Integration Package #13189

Conversation

niraj-crest commented Mar 20, 2025

What does this PR do?

Checklist

How to test this PR locally

Uh oh!

elasticmachine commented Mar 20, 2025

Uh oh!

niraj-crest commented Mar 26, 2025

Uh oh!

kcreddy commented Mar 28, 2025

Uh oh!

kcreddy left a comment

Choose a reason for hiding this comment

Uh oh!

niraj-crest commented Apr 1, 2025

Uh oh!

kcreddy commented Apr 1, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Apr 1, 2025

🚀 Benchmarks report

Uh oh!

kcreddy commented Apr 1, 2025

Uh oh!

niraj-crest commented Apr 2, 2025

Uh oh!

kcreddy commented Apr 2, 2025

Uh oh!

kcreddy commented Apr 2, 2025

Uh oh!

elastic-sonarqube bot commented Apr 2, 2025

Quality Gate passed

Uh oh!

niraj-crest commented Apr 3, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kcreddy commented Apr 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kcreddy commented Jun 4, 2025

Uh oh!

niraj-crest commented Jun 4, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

niraj-crest commented Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kcreddy commented Jun 9, 2025

Uh oh!

kcreddy commented Apr 3, 2025 •

edited

Loading

niraj-crest commented Jun 6, 2025 •

edited

Loading

kcreddy Jun 9, 2025 •

edited

Loading

kcreddy Jun 9, 2025 •

edited

Loading

kcreddy left a comment •

edited

Loading