Domaintools Feeds Integration v1.1.0 #14423

briluza · 2025-07-04T13:06:22Z

Enhancement

Proposed commit message

Added more feeds from DomainTools.

WHAT: Additional datastream, ingest pipeline, fields, saved searches and dashboard for DomainTools Feeds.
WHY: Enhancement of DomainTools Feeds Integration

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

[ ]

How to test this PR locally

Clone integrations repo.
Install the elastic package locally.
Start the elastic stack using the elastic package.
Move to integrations/packages/<integration_name> directory.
Run the following command to run tests.

elastic-package test -v

Related issues

N/A

Screenshots

…e docs

…ntegration

cla-checker-service · 2025-07-04T13:06:27Z

💚 CLA has been signed

elasticmachine · 2025-07-04T16:00:26Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

efd6 · 2025-07-06T22:09:29Z

@briluza Please ensure that you have signed the CLA and that the email address used for signing agrees with the email addresses used in the commits. When this is done, we can take a look at the changes here.

briluza · 2025-07-07T12:59:54Z

done @efd6 , thank you!

packages/ti_domaintools/changelog.yml

packages/ti_domaintools/data_stream/domaindiscovery/_dev/test/system/test-default-config.yml

packages/ti_domaintools/data_stream/domaindiscovery/agent/stream/cel.yml.hbs

efd6 · 2025-07-07T22:03:33Z

.../ti_domaintools/data_stream/domaindiscovery/_dev/test/pipeline/test-event.json-expected.json

+            }
+        }
+    ]
+}


I suspect that you are using an older version of elastic-package. Please upgrade to the latest version and regenerate the test expectations.

packages/ti_domaintools/data_stream/nod_feed/elasticsearch/ingest_pipeline/default.yml

packages/ti_domaintools/data_stream/nod_feed/fields/ecs.yml

packages/ti_domaintools/data_stream/nod_feed/manifest.yml

efd6 · 2025-07-07T22:26:22Z

packages/ti_domaintools/elasticsearch/transform/latest_domaindiscovery/transform.yml

Versions need to be bumped here, but I don't see that for the existing data stream.

@kcreddy What is the convention for fleet_transform_version values wrt package versions?

There is no convention wrt package versions. But it maybe nice to GA (1.0.0) the transform if the integration is also in GA. Just the customers who are aware/not aware of our versioning could understand/misunderstand the transform as Beta feature otherwise.

…ngest-pipeline

efd6

There are some outstanding issues from the previous review:

efd6 · 2025-07-09T01:41:43Z

packages/ti_domaintools/data_stream/domainrdap/elasticsearch/ingest_pipeline/default.yml

+    on_failure:
+      - set:
+          field: error.message
+          value: "Failed to parse JSON from message: {{ ctx.message }} {{ error.message }}"


ISTM that if we fail at this processor, the game is lost. So is there a point in having an on_failure handler? I think not.

efd6 · 2025-07-09T01:42:19Z

packages/ti_domaintools/data_stream/nad_feed/elasticsearch/ingest_pipeline/default.yml

+description: Pipeline for processing nad feed
+processors:
+- json:
+    if: ctx?.message != null


Suggested change

if: ctx?.message != null

if: ctx.message != null

(throughout since ctx is always non-null)

efd6 · 2025-07-09T01:45:38Z

.../ti_domaintools/data_stream/domaindiscovery/_dev/test/pipeline/test-event.json-expected.json

+          "indicator"
+        ]
+      },
+      "message": "{\"timestamp\":\"2025-01-11T08:42:46Z\",\"domain\":\"ccnitsolution.com\"}",


This should be put in event.original.

efd6 · 2025-07-09T01:47:23Z

packages/ti_domaintools/data_stream/domainrdap/elasticsearch/ingest_pipeline/default.yml

+        Processor '{{{ _ingest.on_failure_processor_type }}}'
+        {{#_ingest.on_failure_processor_tag}}with tag '{{{ _ingest.on_failure_processor_tag }}}'
+        {{/_ingest.on_failure_processor_tag}}failed with message '{{{ _ingest.on_failure_message }}}'


Sorry, I must have pointed you to an example with double stache.

Suggested change

Processor '{{{ _ingest.on_failure_processor_type }}}'

{{#_ingest.on_failure_processor_tag}}with tag '{{{ _ingest.on_failure_processor_tag }}}'

{{/_ingest.on_failure_processor_tag}}failed with message '{{{ _ingest.on_failure_message }}}'

Processor '{{{ _ingest.on_failure_processor_type }}}'

{{{#_ingest.on_failure_processor_tag}}}with tag '{{{ _ingest.on_failure_processor_tag }}}'

{{{/_ingest.on_failure_processor_tag}}}failed with message '{{{ _ingest.on_failure_message }}}'

(throughout)

efd6 · 2025-07-09T01:48:05Z

packages/ti_domaintools/data_stream/domainrdap/_dev/test/pipeline/test-event.json-expected.json

+                    "indicator"
+                ]
+            },
+            "message": "{\"timestamp\":\"2025-06-12T20:34:31Z\",\"domain\":\"unlockyourlifehere.com\",\"raw_record\":{\"first_request_timestamp\":\"2025-06-12T20:34:24Z\",\"requests\":[{\"data\":\"{\\\"objectClassName\\\":\\\"domain\\\",\\\"handle\\\":\\\"2894681047_DOMAIN_COM-VRSN\\\",\\\"ldhName\\\":\\\"UNLOCKYOURLIFEHERE.COM\\\",\\\"links\\\":[{\\\"value\\\":\\\"https:\\\\/\\\\/rdap.verisign.com\\\\/com\\\\/v1\\\\/domain\\\\/UNLOCKYOURLIFEHERE.COM\\\",\\\"rel\\\":\\\"self\\\",\\\"href\\\":\\\"https:\\\\/\\\\/rdap.verisign.com\\\\/com\\\\/v1\\\\/domain\\\\/UNLOCKYOURLIFEHERE.COM\\\",\\\"type\\\":\\\"application\\\\/rdap+json\\\"},{\\\"value\\\":\\\"https:\\\\/\\\\/rdap.godaddy.com\\\\/v1\\\\/domain\\\\/UNLOCKYOURLIFEHERE.COM\\\",\\\"rel\\\":\\\"related\\\",\\\"href\\\":\\\"https:\\\\/\\\\/rdap.godaddy.com\\\\/v1\\\\/domain\\\\/UNLOCKYOURLIFEHERE.COM\\\",\\\"type\\\":\\\"application\\\\/rdap+json\\\"}],\\\"status\\\":[\\\"redemption period\\\"],\\\"entities\\\":[{\\\"objectClassName\\\":\\\"entity\\\",\\\"handle\\\":\\\"146\\\",\\\"roles\\\":[\\\"registrar\\\"],\\\"publicIds\\\":[{\\\"type\\\":\\\"IANA Registrar ID\\\",\\\"identifier\\\":\\\"146\\\"}],\\\"vcardArray\\\":[\\\"vcard\\\",[[\\\"version\\\",{},\\\"text\\\",\\\"4.0\\\"],[\\\"fn\\\",{},\\\"text\\\",\\\"GoDaddy.com, LLC\\\"]]],\\\"entities\\\":[{\\\"objectClassName\\\":\\\"entity\\\",\\\"roles\\\":[\\\"abuse\\\"],\\\"vcardArray\\\":[\\\"vcard\\\",[[\\\"version\\\",{},\\\"text\\\",\\\"4.0\\\"],[\\\"fn\\\",{},\\\"text\\\",\\\"\\\"],[\\\"tel\\\",{\\\"type\\\":\\\"voice\\\"},\\\"uri\\\",\\\"tel:480-624-2505\\\"],[\\\"email\\\",{},\\\"text\\\",\\\"[email protected]\\\"]]]}]}],\\\"events\\\":[{\\\"eventAction\\\":\\\"registration\\\",\\\"eventDate\\\":\\\"2024-06-28T11:49:19Z\\\"},{\\\"eventAction\\\":\\\"expiration\\\",\\\"eventDate\\\":\\\"2025-06-28T11:49:19Z\\\"},{\\\"eventAction\\\":\\\"last changed\\\",\\\"eventDate\\\":\\\"2025-05-20T02:44:33Z\\\"},{\\\"eventAction\\\":\\\"last update of RDAP database\\\",\\\"eventDate\\\":\\\"2025-06-12T20:34:16Z\\\"}],\\\"secureDNS\\\":{\\\"delegationSigned\\\":false},\\\"rdapConformance\\\":[\\\"rdap_level_0\\\",\\\"icann_rdap_technical_implementation_guide_0\\\",\\\"icann_rdap_response_profile_0\\\"],\\\"notices\\\":[{\\\"title\\\":\\\"Terms of Use\\\",\\\"description\\\":[\\\"Service subject to Terms of Use.\\\"],\\\"links\\\":[{\\\"href\\\":\\\"https:\\\\/\\\\/www.verisign.com\\\\/domain-names\\\\/registration-data-access-protocol\\\\/terms-service\\\\/index.xhtml\\\",\\\"type\\\":\\\"text\\\\/html\\\"}]},{\\\"title\\\":\\\"Status Codes\\\",\\\"description\\\":[\\\"For more information on domain status codes, please visit https:\\\\/\\\\/icann.org\\\\/epp\\\"],\\\"links\\\":[{\\\"href\\\":\\\"https:\\\\/\\\\/icann.org\\\\/epp\\\",\\\"type\\\":\\\"text\\\\/html\\\"}]},{\\\"title\\\":\\\"RDDS Inaccuracy Complaint Form\\\",\\\"description\\\":[\\\"URL of the ICANN RDDS Inaccuracy Complaint Form: https:\\\\/\\\\/icann.org\\\\/wicf\\\"],\\\"links\\\":[{\\\"href\\\":\\\"https:\\\\/\\\\/icann.org\\\\/wicf\\\",\\\"type\\\":\\\"text\\\\/html\\\"}]}]}\",\"source_type\":\"registry\",\"timestamp\":\"2025-06-12T20:34:24Z\",\"url\":\"https://rdap.verisign.com/com/v1/domain/unlockyourlifehere.com\"}]},\"parsed_record\":{\"parsed_fields\":{\"conformance\":[\"rdap_level_0\",\"icann_rdap_technical_implementation_guide_0\",\"icann_rdap_response_profile_0\"],\"contacts\":[],\"creation_date\":\"2024-06-28T11: 49: 19+00: 00\",\"dnssec\":{\"signed\":false},\"domain\":\"UNLOCKYOURLIFEHERE.COM\",\"domain_statuses\":[\"redemption period\"],\"email_domains\":[\"godaddy.com\"],\"emails\":[\"[email protected]\"],\"expiration_date\":\"2025-06-28T11: 49: 19+00: 00\",\"handle\":\"2894681047_DOMAIN_COM-VRSN\",\"last_changed_date\":\"2025-05-20T02: 44: 33+00: 00\",\"links\":[{\"href\":\"https://rdap.verisign.com/com/v1/domain/UNLOCKYOURLIFEHERE.COM\",\"rel\":\"self\"},{\"href\":\"https://rdap.godaddy.com/v1/domain/UNLOCKYOURLIFEHERE.COM\",\"rel\":\"related\"}],\"registrar\":{\"contacts\":[{\"email\":\"[email protected]\",\"name\":\"\",\"phone\":\"tel:480-624-2505\",\"roles\":[\"abuse\"]}],\"iana_id\":\"146\",\"name\":\"GoDaddy.com, LLC\"},\"unclassified_emails\":[]},\"registrar_request_url\":null,\"registry_request_url\":\"https://rdap.verisign.com/com/v1/domain/unlockyourlifehere.com\"}}",


This should be put in event.original.

efd6 · 2025-07-09T01:51:57Z

packages/ti_domaintools/kibana/search/ti_domaintools-eb599614-25b6-48b2-9225-1e0789619305.json

Why was this deleted?

packages/ti_domaintools/data_stream/domaindiscovery/manifest.yml

efd6 · 2025-07-09T01:55:36Z

packages/ti_domaintools/data_stream/domainrdap/sample_event.json

+                    "name": "GoDaddy.com, LLC",
+                    "contacts": [
+                        {
+                            "name": "",


In that case, please add a comment near the bottom of the ingest pipelines so that the cleaner is not accidentally added in the future.

efd6 · 2025-07-09T01:58:34Z

packages/ti_domaintools/elasticsearch/transform/latest_domaindiscovery/transform.yml

@kcreddy What is the convention for fleet_transform_version values wrt package versions?

efd6 · 2025-07-09T02:01:00Z

packages/ti_domaintools/_dev/build/docs/README.md

+
+### Domain Discovery Feed
+
+The `domaindiscovery feed` data stream provides events from [DomainTools Domain Discovery Feed](https://www.domaintools.com/products/threat-intelligence-feeds/).


Suggested change

The `domaindiscovery feed` data stream provides events from [DomainTools Domain Discovery Feed](https://www.domaintools.com/products/threat-intelligence-feeds/).

The `domaindiscovery` feed data stream provides events from [DomainTools Domain Discovery Feed](https://www.domaintools.com/products/threat-intelligence-feeds/).

(the name of the data stream in the code is domaindiscovery, unless you want to have the "_feed" suffix on all of the data streams for consistency)

briluza and others added 10 commits June 27, 2025 21:05

[IDEV-1964_1988] Implement nad and domaindiscovery feed; update readm…

4192e2f

…e docs

Update the app naming to DomainTools Feeds

4c822f3

Update manifest file

06689eb

Update README

e450e16

[IDEV-1987] feat(domainrdap): Implement domainrdap feeds in elastic i…

9d45ddc

…ntegration

Update version from 0.4.0 to 1.0.1

5b3e799

Merge branch 'elastic:main' into ti_domaintools

9adcb34

Update kibana overview dashboard; addded more saved searches

9b70daa

Merge branch 'elastic:main' into ti_domaintools

1cce741

fix tests and update README

a052738

briluza requested a review from a team as a code owner July 4, 2025 13:06

briluza marked this pull request as draft July 4, 2025 13:07

briluza marked this pull request as ready for review July 4, 2025 13:10

Update PR link in changelog.yml

9a6c016

andrewkroh removed the needs CLA User must sign the Elastic Contributor License before review. label Jul 7, 2025

efd6 reviewed Jul 7, 2025

View reviewed changes

briluza and others added 5 commits July 8, 2025 21:21

Merge branch 'elastic:main' into ti_domaintools

4622fb5

address PR feedbacks

b7e7b2d

[pr feedbacks] - updated the cel.yml.hbs files; change condition in i…

b14d38e

…ngest-pipeline

added final new line

eb24e75

fix error in ingesting

70aa23f

briluza added 2 commits July 8, 2025 23:00

make the feed a constant keyword

957ad2c

update ingest pipeline for domainrdap

779f05f

briluza changed the title ~~Domaintools Feeds Integration v1.0.1~~ Domaintools Feeds Integration v1.1.0 Jul 8, 2025

efd6 reviewed Jul 9, 2025

View reviewed changes

andrewkroh added the enhancement New feature or request label Jul 14, 2025


		### Domain Discovery Feed

		The `domaindiscovery feed` data stream provides events from [DomainTools Domain Discovery Feed](https://www.domaintools.com/products/threat-intelligence-feeds/).

	The `domaindiscovery feed` data stream provides events from [DomainTools Domain Discovery Feed](https://www.domaintools.com/products/threat-intelligence-feeds/).
	The `domaindiscovery` feed data stream provides events from [DomainTools Domain Discovery Feed](https://www.domaintools.com/products/threat-intelligence-feeds/).

Domaintools Feeds Integration v1.1.0 #14423

Are you sure you want to change the base?

Domaintools Feeds Integration v1.1.0 #14423

Conversation

briluza commented Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

cla-checker-service bot commented Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Jul 4, 2025

Uh oh!

efd6 commented Jul 6, 2025

Uh oh!

briluza commented Jul 7, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

briluza commented Jul 4, 2025 •

edited

Loading

cla-checker-service bot commented Jul 4, 2025 •

edited

Loading