diff --git a/packages/ti_domaintools/_dev/build/docs/README.md b/packages/ti_domaintools/_dev/build/docs/README.md index 21da9b3ef80..9aa1cfbe9a6 100644 --- a/packages/ti_domaintools/_dev/build/docs/README.md +++ b/packages/ti_domaintools/_dev/build/docs/README.md @@ -1,40 +1,50 @@ -# DomainTools Real Time Unified Feeds +# DomainTools Feeds -The DomainTools Real Time Unified Feeds integration allows you to monitor DomainTools Newly Observed Domains. -The DomainTools NOD Feed provides real-time access to newly registered and observed domains, enabling proactive threat detection and defense. +DomainTools Feeds provide data on the different stages of the domain lifecycle: from first-observed in the wild, to newly re-activated after a period of quiet. Access current feed data in real-time or retrieve historical feed data through separate APIs. Some feeds also offer data for DNS firewalls in Response Policy Zone (RPZ) format. -With over 300,000 new domains observed daily, the feed empowers security teams to identify and block potentially malicious domains before they can be weaponized. -Ideal for threat hunting, phishing prevention, and brand protection, the NOD Feed delivers unparalleled visibility into emerging domain activity to stay ahead of evolving threats. +Summary of Available Feeds: -For example, if you wanted to monitor Newly Observed Domains (NOD) feed, you could ingest the DomainTools NOD feed. -Then you can reference domaintools.nod_feed when using visualizations or alerts. +- `Newly Active Domains (NAD)`: Apex-level domains (e.g. example.com but not ) that we observe based on the latest lifecycle of the domain. A domain may be seen either for the first time ever, or again after at least 10 days of inactivity (no observed resolutions in DNS). Populated with our global passive DNS (pDNS) sensor network. +- `Newly Observed Domains (NOD)`: Apex-level domains (e.g. example.com but not ) that we observe for the first time, and have not observed previously with our global DNS sensor network. +- `Domain Discovery`: New domains as they are either discovered in domain registration information, observed by our global sensor network, or reported by trusted third parties. +- `Domain RDAP`: Changes to global domain registration information, populated by the Registration Data Access Protocol (RDAP). Compliments the 5-Minute WHOIS Feed as registries and registrars switch from Whois to RDAP. + +With over 300,000 new domains observed daily, the feed empowers security teams to identify and block potentially malicious domains before they can be weaponized. +Ideal for threat hunting, phishing prevention, and brand protection. + +For example, if you wanted to monitor Newly Observed Domains (NOD) feed, you could ingest the DomainTools NOD feed. +Then you can reference ti_domaintools.nod_feed when using visualizations or alerts. ## Data streams -The DomainTools Real Time Unified Feeds integration collects one type of data streams: logs +The DomainTools Feeds integration collects one type of data streams: **logs** + +Log data streams collected by the DomainTools integration include the following feeds: -Log data streams collected by the DomainTools integration include the Newly Observed Domains (NOD) feed: Apex-level domains (e.g. Example Domain but not www.example.com) that we observe for the first time, and have not observed previously. -Populated with our global DNS sensor network. +- `Newly Observed Domains (NOD)` +- `Newly Active Domains (NAD)` +- `Domain Discovery` +- `Domain RDAP` ## Requirements -You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. +You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware. -You will require a license to one or more DomainTools feeds, and API credentials. -Your required API credentials will vary with your authentication method, detailed below. +You will require a license to one or more DomainTools feeds, and API credentials. +Your required API credentials will vary with your authentication method, detailed below. -Obtain your API credentials from your group’s API administrator. +Obtain your API credentials from your group’s API administrator. API administrators can manage their API keys at research.domaintools.com, selecting the drop-down account menu and choosing API admin. ## Setup For step-by-step instructions on how to set up an integration, see the Getting started guide. -### Newly Observed Domains (NOD) Feed +### Newly Observed Domains (NOD) Feed The `nod_feed` data stream provides events from [DomainTools Newly Observed Domains Feed](https://www.domaintools.com/products/threat-intelligence-feeds/). -This data is collected via the [DomainTools Real Time Feeds API](https://docs.domaintools.com/feeds/realtime/). +This data is collected via the [DomainTools Feeds API](https://docs.domaintools.com/feeds/realtime/). #### Example @@ -42,3 +52,35 @@ This data is collected via the [DomainTools Real Time Feeds API](https://docs.do {{fields "nod_feed"}} +### Newly Active Domains (NAD) Feed + +The `nod_feed` data stream provides events from [DomainTools Newly Active Domains Feed](https://www.domaintools.com/products/threat-intelligence-feeds/). +This data is collected via the [DomainTools Feeds API](https://docs.domaintools.com/feeds/realtime/). + +#### Example + +{{event "nad_feed"}} + +{{fields "nad_feed"}} + +### Domain Discovery Feed + +The `domaindiscovery feed` data stream provides events from [DomainTools Domain Discovery Feed](https://www.domaintools.com/products/threat-intelligence-feeds/). +This data is collected via the [DomainTools Feeds API](https://docs.domaintools.com/feeds/realtime/). + +#### Example + +{{event "domaindiscovery"}} + +{{fields "domaindiscovery"}} + +### Domain RDAP + +The `domainrdap feed` data stream provides events from [DomainTools Domain RDAP](https://www.domaintools.com/products/threat-intelligence-feeds/). +This data is collected via the [DomainTools Feeds API](https://docs.domaintools.com/feeds/realtime/). + +#### Example + +{{event "domainrdap"}} + +{{fields "domainrdap"}} diff --git a/packages/ti_domaintools/_dev/deploy/docker/docker-compose.yml b/packages/ti_domaintools/_dev/deploy/docker/docker-compose.yml index 40cb6883980..1092099e687 100644 --- a/packages/ti_domaintools/_dev/deploy/docker/docker-compose.yml +++ b/packages/ti_domaintools/_dev/deploy/docker/docker-compose.yml @@ -1,6 +1,6 @@ version: "2.3" services: - domaintools: + ti_domaintools: image: docker.elastic.co/observability/stream:v0.18.0 ports: - 8080 diff --git a/packages/ti_domaintools/_dev/deploy/docker/files/config.yml b/packages/ti_domaintools/_dev/deploy/docker/files/config.yml index ee46f1622f2..42e2b9b4d7b 100644 --- a/packages/ti_domaintools/_dev/deploy/docker/files/config.yml +++ b/packages/ti_domaintools/_dev/deploy/docker/files/config.yml @@ -6,3 +6,91 @@ rules: body: |- {"timestamp":"2025-01-11T08:42:46Z","domain":"test1.com"} {"timestamp":"2025-01-11T08:42:46Z","domain":"test2.com"} + - path: /v1/feed/nad/ + methods: [GET] + responses: + - status_code: 200 + body: |- + {"timestamp":"2025-01-11T08:42:46Z","domain":"test3.com"} + {"timestamp":"2025-01-11T08:42:46Z","domain":"test4.com"} + - path: /v1/feed/domaindiscovery/ + methods: [GET] + responses: + - status_code: 200 + body: |- + {"timestamp":"2025-01-11T08:42:46Z","domain":"test5.com"} + {"timestamp":"2025-01-11T08:42:46Z","domain":"test6.com"} + - path: /v1/feed/domainrdap/ + methods: [GET] + responses: + - status_code: 200 + body: |- + { + "timestamp": "2025-06-12T20:34:31Z", + "domain": "unlockyourlifehere.com", + "raw_record": { + "first_request_timestamp": "2025-06-12T20:34:24Z", + "requests": [ + { + "data": "{\"objectClassName\":\"domain\",\"handle\":\"2894681047_DOMAIN_COM-VRSN\",\"ldhName\":\"UNLOCKYOURLIFEHERE.COM\",\"links\":[{\"value\":\"https:\\/\\/rdap.verisign.com\\/com\\/v1\\/domain\\/UNLOCKYOURLIFEHERE.COM\",\"rel\":\"self\",\"href\":\"https:\\/\\/rdap.verisign.com\\/com\\/v1\\/domain\\/UNLOCKYOURLIFEHERE.COM\",\"type\":\"application\\/rdap+json\"},{\"value\":\"https:\\/\\/rdap.godaddy.com\\/v1\\/domain\\/UNLOCKYOURLIFEHERE.COM\",\"rel\":\"related\",\"href\":\"https:\\/\\/rdap.godaddy.com\\/v1\\/domain\\/UNLOCKYOURLIFEHERE.COM\",\"type\":\"application\\/rdap+json\"}],\"status\":[\"redemption period\"],\"entities\":[{\"objectClassName\":\"entity\",\"handle\":\"146\",\"roles\":[\"registrar\"],\"publicIds\":[{\"type\":\"IANA Registrar ID\",\"identifier\":\"146\"}],\"vcardArray\":[\"vcard\",[[\"version\",{},\"text\",\"4.0\"],[\"fn\",{},\"text\",\"GoDaddy.com, LLC\"]]],\"entities\":[{\"objectClassName\":\"entity\",\"roles\":[\"abuse\"],\"vcardArray\":[\"vcard\",[[\"version\",{},\"text\",\"4.0\"],[\"fn\",{},\"text\",\"\"],[\"tel\",{\"type\":\"voice\"},\"uri\",\"tel:480-624-2505\"],[\"email\",{},\"text\",\"abuse@godaddy.com\"]]]}]}],\"events\":[{\"eventAction\":\"registration\",\"eventDate\":\"2024-06-28T11:49:19Z\"},{\"eventAction\":\"expiration\",\"eventDate\":\"2025-06-28T11:49:19Z\"},{\"eventAction\":\"last changed\",\"eventDate\":\"2025-05-20T02:44:33Z\"},{\"eventAction\":\"last update of RDAP database\",\"eventDate\":\"2025-06-12T20:34:16Z\"}],\"secureDNS\":{\"delegationSigned\":false},\"rdapConformance\":[\"rdap_level_0\",\"icann_rdap_technical_implementation_guide_0\",\"icann_rdap_response_profile_0\"],\"notices\":[{\"title\":\"Terms of Use\",\"description\":[\"Service subject to Terms of Use.\"],\"links\":[{\"href\":\"https:\\/\\/www.verisign.com\\/domain-names\\/registration-data-access-protocol\\/terms-service\\/index.xhtml\",\"type\":\"text\\/html\"}]},{\"title\":\"Status Codes\",\"description\":[\"For more information on domain status codes, please visit https:\\/\\/icann.org\\/epp\"],\"links\":[{\"href\":\"https:\\/\\/icann.org\\/epp\",\"type\":\"text\\/html\"}]},{\"title\":\"RDDS Inaccuracy Complaint Form\",\"description\":[\"URL of the ICANN RDDS Inaccuracy Complaint Form: https:\\/\\/icann.org\\/wicf\"],\"links\":[{\"href\":\"https:\\/\\/icann.org\\/wicf\",\"type\":\"text\\/html\"}]}]}", + "source_type": "registry", + "timestamp": "2025-06-12T20:34:24Z", + "url": "https://rdap.verisign.com/com/v1/domain/unlockyourlifehere.com" + } + ] + }, + "parsed_record": { + "parsed_fields": { + "conformance": [ + "rdap_level_0", + "icann_rdap_technical_implementation_guide_0", + "icann_rdap_response_profile_0" + ], + "contacts": [], + "creation_date": "2024-06-28T11:49:19+00:00", + "dnssec": { + "signed": false + }, + "domain": "UNLOCKYOURLIFEHERE.COM", + "domain_statuses": [ + "redemption period" + ], + "email_domains": [ + "godaddy.com" + ], + "emails": [ + "abuse@godaddy.com" + ], + "expiration_date": "2025-06-28T11:49:19+00:00", + "handle": "2894681047_DOMAIN_COM-VRSN", + "last_changed_date": "2025-05-20T02:44:33+00:00", + "links": [ + { + "href": "https://rdap.verisign.com/com/v1/domain/UNLOCKYOURLIFEHERE.COM", + "rel": "self" + }, + { + "href": "https://rdap.godaddy.com/v1/domain/UNLOCKYOURLIFEHERE.COM", + "rel": "related" + } + ], + "registrar": { + "contacts": [ + { + "email": "abuse@godaddy.com", + "name": "", + "phone": "tel:480-624-2505", + "roles": [ + "abuse" + ] + } + ], + "iana_id": "146", + "name": "GoDaddy.com, LLC" + }, + "unclassified_emails": [] + }, + "registrar_request_url": null, + "registry_request_url": "https://rdap.verisign.com/com/v1/domain/unlockyourlifehere.com" + } + } diff --git a/packages/ti_domaintools/changelog.yml b/packages/ti_domaintools/changelog.yml index 249a4f08d98..80ef1276d98 100644 --- a/packages/ti_domaintools/changelog.yml +++ b/packages/ti_domaintools/changelog.yml @@ -1,4 +1,12 @@ # newer versions go on top +- version: "1.1.0" + changes: + - description: Added nad, domaindiscovery and domainrdap domaintools feeds. + type: enhancement + link: https://github.com/elastic/integrations/pull/14423 + - description: Update overview dashboard to include the newly added feeds. + type: enhancement + link: https://github.com/elastic/integrations/pull/14423 - version: "1.0.0" changes: - description: Release package as GA. diff --git a/packages/ti_domaintools/data_stream/domaindiscovery/_dev/test/pipeline/test-event.json b/packages/ti_domaintools/data_stream/domaindiscovery/_dev/test/pipeline/test-event.json new file mode 100644 index 00000000000..75451670f59 --- /dev/null +++ b/packages/ti_domaintools/data_stream/domaindiscovery/_dev/test/pipeline/test-event.json @@ -0,0 +1,13 @@ +{ + "events": [ + { + "message": "{\"timestamp\":\"2025-01-11T08:42:46Z\",\"domain\":\"ccnitsolution.com\"}" + }, + { + "message": "{\"timestamp\":\"2025-01-11T08:42:46Z\",\"domain\":\"ccnitsolution2.com\"}" + }, + { + "message": "{\"timestamp\":\"2025-01-11T08:42:46Z\",\"domain\":\"ccnitsolution3.com\"}" + } + ] +} \ No newline at end of file diff --git a/packages/ti_domaintools/data_stream/domaindiscovery/_dev/test/pipeline/test-event.json-expected.json b/packages/ti_domaintools/data_stream/domaindiscovery/_dev/test/pipeline/test-event.json-expected.json new file mode 100644 index 00000000000..52582640170 --- /dev/null +++ b/packages/ti_domaintools/data_stream/domaindiscovery/_dev/test/pipeline/test-event.json-expected.json @@ -0,0 +1,94 @@ +{ + "expected": [ + { + "domaintools": { + "domain": "ccnitsolution.com", + "timestamp": "2025-01-11T08:42:46Z" + }, + "ecs": { + "version": "8.11.0" + }, + "event": { + "category": [ + "threat" + ], + "kind": "enrichment", + "type": [ + "indicator" + ] + }, + "message": "{\"timestamp\":\"2025-01-11T08:42:46Z\",\"domain\":\"ccnitsolution.com\"}", + "threat": { + "feed": { + "description": "New domains as they are either discovered in domain registration information, observed by our global sensor network, or reported by trusted third parties.", + "name": "DomainTools domaindiscovery", + "reference": "https://docs.techdocs.ci.domaintools.cloud/feeds/realtime/userguide/" + }, + "indicator": { + "name": "ccnitsolution.com", + "type": "domain-name" + } + } + }, + { + "domaintools": { + "domain": "ccnitsolution2.com", + "timestamp": "2025-01-11T08:42:46Z" + }, + "threat": { + "indicator": { + "name": "ccnitsolution2.com", + "type": "domain-name" + }, + "feed": { + "reference": "https://docs.techdocs.ci.domaintools.cloud/feeds/realtime/userguide/", + "name": "DomainTools domaindiscovery", + "description": "New domains as they are either discovered in domain registration information, observed by our global sensor network, or reported by trusted third parties." + } + }, + "message": "{\"timestamp\":\"2025-01-11T08:42:46Z\",\"domain\":\"ccnitsolution2.com\"}", + "ecs": { + "version": "8.11.0" + }, + "event": { + "category": [ + "threat" + ], + "type": [ + "indicator" + ], + "kind": "enrichment" + } + }, + { + "domaintools": { + "domain": "ccnitsolution3.com", + "timestamp": "2025-01-11T08:42:46Z" + }, + "threat": { + "indicator": { + "name": "ccnitsolution3.com", + "type": "domain-name" + }, + "feed": { + "reference": "https://docs.techdocs.ci.domaintools.cloud/feeds/realtime/userguide/", + "name": "DomainTools domaindiscovery", + "description": "New domains as they are either discovered in domain registration information, observed by our global sensor network, or reported by trusted third parties." + } + }, + "message": "{\"timestamp\":\"2025-01-11T08:42:46Z\",\"domain\":\"ccnitsolution3.com\"}", + "ecs": { + "version": "8.11.0" + }, + "event": { + "category": [ + "threat" + ], + "type": [ + "indicator" + ], + "kind": "enrichment" + } + } + ] +} \ No newline at end of file diff --git a/packages/ti_domaintools/data_stream/domaindiscovery/_dev/test/system/test-default-config.yml b/packages/ti_domaintools/data_stream/domaindiscovery/_dev/test/system/test-default-config.yml new file mode 100644 index 00000000000..31a0f441817 --- /dev/null +++ b/packages/ti_domaintools/data_stream/domaindiscovery/_dev/test/system/test-default-config.yml @@ -0,0 +1,11 @@ +input: cel +service: ti_domaintools +vars: +data_stream: + vars: + api_url: http://{{Hostname}}:{{Port}}/v1 + interval: 10m + api_username: xxx + api_key: xxx +assert: + hit_count: 2 diff --git a/packages/ti_domaintools/data_stream/domaindiscovery/agent/stream/cel.yml.hbs b/packages/ti_domaintools/data_stream/domaindiscovery/agent/stream/cel.yml.hbs new file mode 100644 index 00000000000..3f710b8a3e3 --- /dev/null +++ b/packages/ti_domaintools/data_stream/domaindiscovery/agent/stream/cel.yml.hbs @@ -0,0 +1,63 @@ +config_version: "2" +interval: {{interval}} +resource.tracer: + enabled: {{enable_request_tracer}} + filename: "../../logs/cel/http-request-trace-*.ndjson" + maxbackups: 5 +resource.url: {{api_url}} +state: + api_username: {{api_username}} + api_key: {{api_key}} + session_id: {{session_id}} + app_name: elastic_feeds + app_partner: elastic + app_version: 0.1.0 +redact: + fields: + - api_key +program: | + state.with( + request( + "GET", + state.url.trim_right("/") + "/feed/domaindiscovery/?" + { + "api_username": [state.api_username], + "api_key": [state.api_key], + "sessionID": [state.session_id], + "app_name": [state.app_name], + "app_partner": [state.app_partner], + "app_version": [state.app_version], + }.format_query() + ).with( + { + "Header": { + "Accept": ["application/x-ndjson"], + }, + } + ).do_request().as(resp, (resp.StatusCode == 200 || resp.StatusCode == 206) ? + { + "events": string(resp.Body).split("\n").map(e, e!="", + { + "message": e, + } + ), + "want_more": resp.StatusCode == 206 + } + : + { + "events": { + "error": { + "code": string(resp.StatusCode), + "id": string(resp.Status), + "message": "GET: " + + ( + (size(resp.Body) != 0) ? + string(resp.Body) + : + string(resp.Status) + " (" + string(resp.StatusCode) + ")" + ), + }, + }, + "want_more": false, + } + ) + ) diff --git a/packages/ti_domaintools/data_stream/domaindiscovery/elasticsearch/ilm/default_policy.json b/packages/ti_domaintools/data_stream/domaindiscovery/elasticsearch/ilm/default_policy.json new file mode 100644 index 00000000000..68d2c5e57a6 --- /dev/null +++ b/packages/ti_domaintools/data_stream/domaindiscovery/elasticsearch/ilm/default_policy.json @@ -0,0 +1,23 @@ +{ + "policy": { + "phases": { + "hot": { + "actions": { + "rollover": { + "max_age": "2d", + "max_size": "50gb" + }, + "set_priority": { + "priority": 100 + } + } + }, + "delete": { + "min_age": "3d", + "actions": { + "delete": {} + } + } + } + } +} \ No newline at end of file diff --git a/packages/ti_domaintools/data_stream/domaindiscovery/elasticsearch/ingest_pipeline/default.yml b/packages/ti_domaintools/data_stream/domaindiscovery/elasticsearch/ingest_pipeline/default.yml new file mode 100644 index 00000000000..ca0d7dd87ba --- /dev/null +++ b/packages/ti_domaintools/data_stream/domaindiscovery/elasticsearch/ingest_pipeline/default.yml @@ -0,0 +1,65 @@ +--- +description: Pipeline for processing domaindiscovery feed +processors: +- json: + if: ctx?.message != null + field: message + target_field: domaintools + + ############################ + # Generic indicator fields # + ############################ + +- set: + field: threat.indicator.type + value: domain-name +- set: + if: ctx.domaintools?.domain != null + field: threat.indicator.name + copy_from: domaintools.domain + + ###################### + # Threat feed fields # + ###################### + +- set: + field: threat.feed.description + value: 'New domains as they are either discovered in domain registration information, observed by our global sensor network, or reported by trusted third parties.' +- set: + field: threat.feed.name + value: 'DomainTools domaindiscovery' +- set: + field: threat.feed.reference + value: https://docs.techdocs.ci.domaintools.cloud/feeds/realtime/userguide/ + + #################### + # Event ECS fields # + #################### + +- set: + field: ecs.version + value: '8.11.0' +- set: + field: event.kind + value: enrichment +- set: + field: event.category + value: ['threat'] +- set: + field: event.type + value: ['indicator'] + +on_failure: +- set: + field: event.kind + value: pipeline_error +- append: + field: tags + value: preserve_original_event + allow_duplicates: false +- append: + field: error.message + value: >- + Processor '{{{ _ingest.on_failure_processor_type }}}' + {{#_ingest.on_failure_processor_tag}}with tag '{{{ _ingest.on_failure_processor_tag }}}' + {{/_ingest.on_failure_processor_tag}}failed with message '{{{ _ingest.on_failure_message }}}' diff --git a/packages/ti_domaintools/data_stream/domaindiscovery/fields/base-fields.yml b/packages/ti_domaintools/data_stream/domaindiscovery/fields/base-fields.yml new file mode 100644 index 00000000000..0f3844069d4 --- /dev/null +++ b/packages/ti_domaintools/data_stream/domaindiscovery/fields/base-fields.yml @@ -0,0 +1,8 @@ +- name: data_stream.type + external: ecs +- name: data_stream.dataset + external: ecs +- name: data_stream.namespace + external: ecs +- name: "@timestamp" + external: ecs diff --git a/packages/ti_domaintools/data_stream/domaindiscovery/fields/ecs.yml b/packages/ti_domaintools/data_stream/domaindiscovery/fields/ecs.yml new file mode 100644 index 00000000000..664ff10ce6a --- /dev/null +++ b/packages/ti_domaintools/data_stream/domaindiscovery/fields/ecs.yml @@ -0,0 +1,24 @@ +- name: ecs.version + external: ecs +- name: error.message + external: ecs +- name: event.category + external: ecs +- name: event.id + external: ecs +- name: event.ingested + external: ecs +- name: event.kind + external: ecs +- name: event.type + external: ecs +- name: threat.indicator.type + external: ecs +- name: threat.indicator.name + external: ecs +- name: threat.feed.description + external: ecs +- name: threat.feed.name + external: ecs +- name: threat.feed.reference + external: ecs diff --git a/packages/ti_domaintools/data_stream/domaindiscovery/fields/fields.yml b/packages/ti_domaintools/data_stream/domaindiscovery/fields/fields.yml new file mode 100644 index 00000000000..3deb150fc16 --- /dev/null +++ b/packages/ti_domaintools/data_stream/domaindiscovery/fields/fields.yml @@ -0,0 +1,27 @@ +- name: domaintools + type: group + fields: + - name: domain + type: keyword + description: > + The Domain. + + - name: feed + type: constant_keyword + value: "domaindiscovery_feed" + description: > + The feed type. + + - name: timestamp + type: date + description: > + Timestamp when the domain was added to the DomainTools feed, in ISO 8601 UTC form. + +- name: message + external: ecs + description: > + The feed from DomainTools Feed API. + +- name: input.type + type: keyword + description: Type of filebeat input. diff --git a/packages/ti_domaintools/data_stream/domaindiscovery/fields/is-ioc-transform-source-true.yml b/packages/ti_domaintools/data_stream/domaindiscovery/fields/is-ioc-transform-source-true.yml new file mode 100644 index 00000000000..b1d45027981 --- /dev/null +++ b/packages/ti_domaintools/data_stream/domaindiscovery/fields/is-ioc-transform-source-true.yml @@ -0,0 +1,4 @@ +- name: labels.is_ioc_transform_source + type: constant_keyword + value: "true" + description: Indicates whether an IOC is in the raw source data stream, or the in latest destination index. diff --git a/packages/ti_domaintools/data_stream/domaindiscovery/lifecycle.yml b/packages/ti_domaintools/data_stream/domaindiscovery/lifecycle.yml new file mode 100644 index 00000000000..5a4af9095b7 --- /dev/null +++ b/packages/ti_domaintools/data_stream/domaindiscovery/lifecycle.yml @@ -0,0 +1 @@ +data_retention: "5d" diff --git a/packages/ti_domaintools/data_stream/domaindiscovery/manifest.yml b/packages/ti_domaintools/data_stream/domaindiscovery/manifest.yml new file mode 100644 index 00000000000..094e755c2dd --- /dev/null +++ b/packages/ti_domaintools/data_stream/domaindiscovery/manifest.yml @@ -0,0 +1,58 @@ +title: "DomainTools Domain Discovery Feed" +type: logs +ilm_policy: logs-ti_domaintools.domaindiscovery-default_policy +streams: + - input: cel + vars: + - name: api_url + type: text + title: DomainTools API URL + multi: false + required: true + show_user: true + default: https://api.domaintools.com/v1 + description: The URL of the DomainTools API. + - name: interval + type: text + title: Interval + multi: false + required: true + show_user: true + default: 10m + description: Interval at which the feed will be pulled. Supported units for this parameter are h/m/s. + - name: api_username + type: text + title: DomainTools API Username + multi: false + required: true + show_user: true + default: DomainTools API Username + description: DomainTools API Username + - name: api_key + type: password + title: DomainTools API Key + multi: false + required: true + show_user: true + secret: true + description: DomainTools API Key + - name: session_id + type: text + title: Session ID + multi: false + required: true + show_user: true + default: DomainToolsElasticSID + description: The Session ID to use in requesting feed. + - name: enable_request_tracer + type: bool + title: Enable request tracing + description: >- + The request tracer logs requests and responses to the agent's local file-system for debugging configurations. Enabling this request tracing compromises security and should only be used for debugging. Disabling the request tracer will delete any stored traces. See [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-cel.html#_resource_tracer_enable) for details. + default: false + multi: false + required: false + show_user: false + title: DomainTools Domain Discovery Feed + description: Subscribe to DomainTools Domain Discovery Feed + template_path: cel.yml.hbs diff --git a/packages/ti_domaintools/data_stream/domaindiscovery/sample_event.json b/packages/ti_domaintools/data_stream/domaindiscovery/sample_event.json new file mode 100644 index 00000000000..34bb6da07ab --- /dev/null +++ b/packages/ti_domaintools/data_stream/domaindiscovery/sample_event.json @@ -0,0 +1,77 @@ +{ + "input": { + "type": "cel" + }, + "agent": { + "name": "docker-fleet-agent", + "id": "da8d0a37-2d46-4788-96bd-e9ee19e332ec", + "ephemeral_id": "d1cbe648-0a1d-48e8-a161-cd82403e623e", + "type": "filebeat", + "version": "8.15.3" + }, + "@timestamp": "2025-01-30T20:15:25.396Z", + "ecs": { + "version": "8.11.0" + }, + "data_stream": { + "namespace": "default", + "type": "logs", + "dataset": "ti_domaintools.domaindiscovery" + }, + "host": { + "hostname": "docker-fleet-agent", + "os": { + "kernel": "6.10.11-linuxkit", + "codename": "focal", + "name": "Ubuntu", + "type": "linux", + "family": "debian", + "version": "20.04.6 LTS (Focal Fossa)", + "platform": "ubuntu" + }, + "containerized": false, + "ip": [ + "172.19.0.10" + ], + "name": "docker-fleet-agent", + "id": "cfae1e7244ae479b9b0968259c91b13a", + "mac": [ + "02-42-AC-13-00-0A" + ], + "architecture": "aarch64" + }, + "elastic_agent": { + "id": "da8d0a37-2d46-4788-96bd-e9ee19e332ec", + "version": "8.15.3", + "snapshot": false + }, + "domaintools": { + "domain": "tractorpoweredcoreaerator.com", + "timestamp": "2025-01-30T20:14:48Z", + "feed": "domaindiscovery" + }, + "threat": { + "indicator": { + "name": "tractorpoweredcoreaerator.com", + "type": "domain-name" + }, + "feed": { + "reference": "https://docs.techdocs.ci.domaintools.cloud/feeds/realtime/userguide/", + "name": "DomainTools domaindiscovery", + "description": "New domains as they are either discovered in domain registration information, observed by our global sensor network, or reported by trusted third parties." + } + }, + "message": "{\"timestamp\":\"2025-01-30T20:14:48Z\",\"domain\":\"tractorpoweredcoreaerator.com\"}", + "event": { + "agent_id_status": "verified", + "ingested": "2025-01-30T20:15:26Z", + "kind": "enrichment", + "category": [ + "threat" + ], + "type": [ + "indicator" + ], + "dataset": "ti_domaintools.domaindiscovery" + } +} \ No newline at end of file diff --git a/packages/ti_domaintools/data_stream/domainrdap/_dev/test/pipeline/test-event.json b/packages/ti_domaintools/data_stream/domainrdap/_dev/test/pipeline/test-event.json new file mode 100644 index 00000000000..fba40fb3f4f --- /dev/null +++ b/packages/ti_domaintools/data_stream/domainrdap/_dev/test/pipeline/test-event.json @@ -0,0 +1,7 @@ +{ + "events": [ + { + "message": "{\"timestamp\":\"2025-06-12T20:34:31Z\",\"domain\":\"unlockyourlifehere.com\",\"raw_record\":{\"first_request_timestamp\":\"2025-06-12T20:34:24Z\",\"requests\":[{\"data\":\"{\\\"objectClassName\\\":\\\"domain\\\",\\\"handle\\\":\\\"2894681047_DOMAIN_COM-VRSN\\\",\\\"ldhName\\\":\\\"UNLOCKYOURLIFEHERE.COM\\\",\\\"links\\\":[{\\\"value\\\":\\\"https:\\\\/\\\\/rdap.verisign.com\\\\/com\\\\/v1\\\\/domain\\\\/UNLOCKYOURLIFEHERE.COM\\\",\\\"rel\\\":\\\"self\\\",\\\"href\\\":\\\"https:\\\\/\\\\/rdap.verisign.com\\\\/com\\\\/v1\\\\/domain\\\\/UNLOCKYOURLIFEHERE.COM\\\",\\\"type\\\":\\\"application\\\\/rdap+json\\\"},{\\\"value\\\":\\\"https:\\\\/\\\\/rdap.godaddy.com\\\\/v1\\\\/domain\\\\/UNLOCKYOURLIFEHERE.COM\\\",\\\"rel\\\":\\\"related\\\",\\\"href\\\":\\\"https:\\\\/\\\\/rdap.godaddy.com\\\\/v1\\\\/domain\\\\/UNLOCKYOURLIFEHERE.COM\\\",\\\"type\\\":\\\"application\\\\/rdap+json\\\"}],\\\"status\\\":[\\\"redemption period\\\"],\\\"entities\\\":[{\\\"objectClassName\\\":\\\"entity\\\",\\\"handle\\\":\\\"146\\\",\\\"roles\\\":[\\\"registrar\\\"],\\\"publicIds\\\":[{\\\"type\\\":\\\"IANA Registrar ID\\\",\\\"identifier\\\":\\\"146\\\"}],\\\"vcardArray\\\":[\\\"vcard\\\",[[\\\"version\\\",{},\\\"text\\\",\\\"4.0\\\"],[\\\"fn\\\",{},\\\"text\\\",\\\"GoDaddy.com, LLC\\\"]]],\\\"entities\\\":[{\\\"objectClassName\\\":\\\"entity\\\",\\\"roles\\\":[\\\"abuse\\\"],\\\"vcardArray\\\":[\\\"vcard\\\",[[\\\"version\\\",{},\\\"text\\\",\\\"4.0\\\"],[\\\"fn\\\",{},\\\"text\\\",\\\"\\\"],[\\\"tel\\\",{\\\"type\\\":\\\"voice\\\"},\\\"uri\\\",\\\"tel:480-624-2505\\\"],[\\\"email\\\",{},\\\"text\\\",\\\"abuse@godaddy.com\\\"]]]}]}],\\\"events\\\":[{\\\"eventAction\\\":\\\"registration\\\",\\\"eventDate\\\":\\\"2024-06-28T11:49:19Z\\\"},{\\\"eventAction\\\":\\\"expiration\\\",\\\"eventDate\\\":\\\"2025-06-28T11:49:19Z\\\"},{\\\"eventAction\\\":\\\"last changed\\\",\\\"eventDate\\\":\\\"2025-05-20T02:44:33Z\\\"},{\\\"eventAction\\\":\\\"last update of RDAP database\\\",\\\"eventDate\\\":\\\"2025-06-12T20:34:16Z\\\"}],\\\"secureDNS\\\":{\\\"delegationSigned\\\":false},\\\"rdapConformance\\\":[\\\"rdap_level_0\\\",\\\"icann_rdap_technical_implementation_guide_0\\\",\\\"icann_rdap_response_profile_0\\\"],\\\"notices\\\":[{\\\"title\\\":\\\"Terms of Use\\\",\\\"description\\\":[\\\"Service subject to Terms of Use.\\\"],\\\"links\\\":[{\\\"href\\\":\\\"https:\\\\/\\\\/www.verisign.com\\\\/domain-names\\\\/registration-data-access-protocol\\\\/terms-service\\\\/index.xhtml\\\",\\\"type\\\":\\\"text\\\\/html\\\"}]},{\\\"title\\\":\\\"Status Codes\\\",\\\"description\\\":[\\\"For more information on domain status codes, please visit https:\\\\/\\\\/icann.org\\\\/epp\\\"],\\\"links\\\":[{\\\"href\\\":\\\"https:\\\\/\\\\/icann.org\\\\/epp\\\",\\\"type\\\":\\\"text\\\\/html\\\"}]},{\\\"title\\\":\\\"RDDS Inaccuracy Complaint Form\\\",\\\"description\\\":[\\\"URL of the ICANN RDDS Inaccuracy Complaint Form: https:\\\\/\\\\/icann.org\\\\/wicf\\\"],\\\"links\\\":[{\\\"href\\\":\\\"https:\\\\/\\\\/icann.org\\\\/wicf\\\",\\\"type\\\":\\\"text\\\\/html\\\"}]}]}\",\"source_type\":\"registry\",\"timestamp\":\"2025-06-12T20:34:24Z\",\"url\":\"https://rdap.verisign.com/com/v1/domain/unlockyourlifehere.com\"}]},\"parsed_record\":{\"parsed_fields\":{\"conformance\":[\"rdap_level_0\",\"icann_rdap_technical_implementation_guide_0\",\"icann_rdap_response_profile_0\"],\"contacts\":[],\"creation_date\":\"2024-06-28T11: 49: 19+00: 00\",\"dnssec\":{\"signed\":false},\"domain\":\"UNLOCKYOURLIFEHERE.COM\",\"domain_statuses\":[\"redemption period\"],\"email_domains\":[\"godaddy.com\"],\"emails\":[\"abuse@godaddy.com\"],\"expiration_date\":\"2025-06-28T11: 49: 19+00: 00\",\"handle\":\"2894681047_DOMAIN_COM-VRSN\",\"last_changed_date\":\"2025-05-20T02: 44: 33+00: 00\",\"links\":[{\"href\":\"https://rdap.verisign.com/com/v1/domain/UNLOCKYOURLIFEHERE.COM\",\"rel\":\"self\"},{\"href\":\"https://rdap.godaddy.com/v1/domain/UNLOCKYOURLIFEHERE.COM\",\"rel\":\"related\"}],\"registrar\":{\"contacts\":[{\"email\":\"abuse@godaddy.com\",\"name\":\"\",\"phone\":\"tel:480-624-2505\",\"roles\":[\"abuse\"]}],\"iana_id\":\"146\",\"name\":\"GoDaddy.com, LLC\"},\"unclassified_emails\":[]},\"registrar_request_url\":null,\"registry_request_url\":\"https://rdap.verisign.com/com/v1/domain/unlockyourlifehere.com\"}}" + } + ] +} \ No newline at end of file diff --git a/packages/ti_domaintools/data_stream/domainrdap/_dev/test/pipeline/test-event.json-expected.json b/packages/ti_domaintools/data_stream/domainrdap/_dev/test/pipeline/test-event.json-expected.json new file mode 100644 index 00000000000..6f38f2b8037 --- /dev/null +++ b/packages/ti_domaintools/data_stream/domainrdap/_dev/test/pipeline/test-event.json-expected.json @@ -0,0 +1,67 @@ +{ + "expected": [ + { + "domaintools": { + "domain": "unlockyourlifehere.com", + "first_request_timestamp": "2025-06-12T20:34:24Z", + "parsed_record": { + "parsed_fields": { + "contacts": [], + "creation_date": "2024-06-28T11: 49: 19+00: 00", + "email_domains": [ + "godaddy.com" + ], + "emails": [ + "abuse@godaddy.com" + ], + "expiration_date": "2025-06-28T11: 49: 19+00: 00", + "handle": "2894681047_DOMAIN_COM-VRSN", + "last_changed_date": "2025-05-20T02: 44: 33+00: 00", + "registrar": { + "contacts": [ + { + "email": "abuse@godaddy.com", + "name": "", + "phone": "tel:480-624-2505", + "roles": [ + "abuse" + ] + } + ], + "iana_id": "146", + "name": "GoDaddy.com, LLC" + } + } + }, + "requests_url": [ + "https://rdap.verisign.com/com/v1/domain/unlockyourlifehere.com" + ], + "timestamp": "2025-06-12T20:34:31Z" + }, + "ecs": { + "version": "8.11.0" + }, + "event": { + "category": [ + "threat" + ], + "kind": "enrichment", + "type": [ + "indicator" + ] + }, + "message": "{\"timestamp\":\"2025-06-12T20:34:31Z\",\"domain\":\"unlockyourlifehere.com\",\"raw_record\":{\"first_request_timestamp\":\"2025-06-12T20:34:24Z\",\"requests\":[{\"data\":\"{\\\"objectClassName\\\":\\\"domain\\\",\\\"handle\\\":\\\"2894681047_DOMAIN_COM-VRSN\\\",\\\"ldhName\\\":\\\"UNLOCKYOURLIFEHERE.COM\\\",\\\"links\\\":[{\\\"value\\\":\\\"https:\\\\/\\\\/rdap.verisign.com\\\\/com\\\\/v1\\\\/domain\\\\/UNLOCKYOURLIFEHERE.COM\\\",\\\"rel\\\":\\\"self\\\",\\\"href\\\":\\\"https:\\\\/\\\\/rdap.verisign.com\\\\/com\\\\/v1\\\\/domain\\\\/UNLOCKYOURLIFEHERE.COM\\\",\\\"type\\\":\\\"application\\\\/rdap+json\\\"},{\\\"value\\\":\\\"https:\\\\/\\\\/rdap.godaddy.com\\\\/v1\\\\/domain\\\\/UNLOCKYOURLIFEHERE.COM\\\",\\\"rel\\\":\\\"related\\\",\\\"href\\\":\\\"https:\\\\/\\\\/rdap.godaddy.com\\\\/v1\\\\/domain\\\\/UNLOCKYOURLIFEHERE.COM\\\",\\\"type\\\":\\\"application\\\\/rdap+json\\\"}],\\\"status\\\":[\\\"redemption period\\\"],\\\"entities\\\":[{\\\"objectClassName\\\":\\\"entity\\\",\\\"handle\\\":\\\"146\\\",\\\"roles\\\":[\\\"registrar\\\"],\\\"publicIds\\\":[{\\\"type\\\":\\\"IANA Registrar ID\\\",\\\"identifier\\\":\\\"146\\\"}],\\\"vcardArray\\\":[\\\"vcard\\\",[[\\\"version\\\",{},\\\"text\\\",\\\"4.0\\\"],[\\\"fn\\\",{},\\\"text\\\",\\\"GoDaddy.com, LLC\\\"]]],\\\"entities\\\":[{\\\"objectClassName\\\":\\\"entity\\\",\\\"roles\\\":[\\\"abuse\\\"],\\\"vcardArray\\\":[\\\"vcard\\\",[[\\\"version\\\",{},\\\"text\\\",\\\"4.0\\\"],[\\\"fn\\\",{},\\\"text\\\",\\\"\\\"],[\\\"tel\\\",{\\\"type\\\":\\\"voice\\\"},\\\"uri\\\",\\\"tel:480-624-2505\\\"],[\\\"email\\\",{},\\\"text\\\",\\\"abuse@godaddy.com\\\"]]]}]}],\\\"events\\\":[{\\\"eventAction\\\":\\\"registration\\\",\\\"eventDate\\\":\\\"2024-06-28T11:49:19Z\\\"},{\\\"eventAction\\\":\\\"expiration\\\",\\\"eventDate\\\":\\\"2025-06-28T11:49:19Z\\\"},{\\\"eventAction\\\":\\\"last changed\\\",\\\"eventDate\\\":\\\"2025-05-20T02:44:33Z\\\"},{\\\"eventAction\\\":\\\"last update of RDAP database\\\",\\\"eventDate\\\":\\\"2025-06-12T20:34:16Z\\\"}],\\\"secureDNS\\\":{\\\"delegationSigned\\\":false},\\\"rdapConformance\\\":[\\\"rdap_level_0\\\",\\\"icann_rdap_technical_implementation_guide_0\\\",\\\"icann_rdap_response_profile_0\\\"],\\\"notices\\\":[{\\\"title\\\":\\\"Terms of Use\\\",\\\"description\\\":[\\\"Service subject to Terms of Use.\\\"],\\\"links\\\":[{\\\"href\\\":\\\"https:\\\\/\\\\/www.verisign.com\\\\/domain-names\\\\/registration-data-access-protocol\\\\/terms-service\\\\/index.xhtml\\\",\\\"type\\\":\\\"text\\\\/html\\\"}]},{\\\"title\\\":\\\"Status Codes\\\",\\\"description\\\":[\\\"For more information on domain status codes, please visit https:\\\\/\\\\/icann.org\\\\/epp\\\"],\\\"links\\\":[{\\\"href\\\":\\\"https:\\\\/\\\\/icann.org\\\\/epp\\\",\\\"type\\\":\\\"text\\\\/html\\\"}]},{\\\"title\\\":\\\"RDDS Inaccuracy Complaint Form\\\",\\\"description\\\":[\\\"URL of the ICANN RDDS Inaccuracy Complaint Form: https:\\\\/\\\\/icann.org\\\\/wicf\\\"],\\\"links\\\":[{\\\"href\\\":\\\"https:\\\\/\\\\/icann.org\\\\/wicf\\\",\\\"type\\\":\\\"text\\\\/html\\\"}]}]}\",\"source_type\":\"registry\",\"timestamp\":\"2025-06-12T20:34:24Z\",\"url\":\"https://rdap.verisign.com/com/v1/domain/unlockyourlifehere.com\"}]},\"parsed_record\":{\"parsed_fields\":{\"conformance\":[\"rdap_level_0\",\"icann_rdap_technical_implementation_guide_0\",\"icann_rdap_response_profile_0\"],\"contacts\":[],\"creation_date\":\"2024-06-28T11: 49: 19+00: 00\",\"dnssec\":{\"signed\":false},\"domain\":\"UNLOCKYOURLIFEHERE.COM\",\"domain_statuses\":[\"redemption period\"],\"email_domains\":[\"godaddy.com\"],\"emails\":[\"abuse@godaddy.com\"],\"expiration_date\":\"2025-06-28T11: 49: 19+00: 00\",\"handle\":\"2894681047_DOMAIN_COM-VRSN\",\"last_changed_date\":\"2025-05-20T02: 44: 33+00: 00\",\"links\":[{\"href\":\"https://rdap.verisign.com/com/v1/domain/UNLOCKYOURLIFEHERE.COM\",\"rel\":\"self\"},{\"href\":\"https://rdap.godaddy.com/v1/domain/UNLOCKYOURLIFEHERE.COM\",\"rel\":\"related\"}],\"registrar\":{\"contacts\":[{\"email\":\"abuse@godaddy.com\",\"name\":\"\",\"phone\":\"tel:480-624-2505\",\"roles\":[\"abuse\"]}],\"iana_id\":\"146\",\"name\":\"GoDaddy.com, LLC\"},\"unclassified_emails\":[]},\"registrar_request_url\":null,\"registry_request_url\":\"https://rdap.verisign.com/com/v1/domain/unlockyourlifehere.com\"}}", + "threat": { + "feed": { + "description": "Changes to global domain registration information, populated by the Registration Data Access Protocol (RDAP). Compliments the 5-Minute WHOIS Feed as registries and registrars switch from Whois to RDAP.", + "name": "DomainTools domain RDAP", + "reference": "https://docs.techdocs.ci.domaintools.cloud/feeds/realtime/userguide/" + }, + "indicator": { + "name": "unlockyourlifehere.com", + "type": "domain-name" + } + } + } + ] +} \ No newline at end of file diff --git a/packages/ti_domaintools/data_stream/domainrdap/_dev/test/system/test-default-config.yml b/packages/ti_domaintools/data_stream/domainrdap/_dev/test/system/test-default-config.yml new file mode 100644 index 00000000000..31a0f441817 --- /dev/null +++ b/packages/ti_domaintools/data_stream/domainrdap/_dev/test/system/test-default-config.yml @@ -0,0 +1,11 @@ +input: cel +service: ti_domaintools +vars: +data_stream: + vars: + api_url: http://{{Hostname}}:{{Port}}/v1 + interval: 10m + api_username: xxx + api_key: xxx +assert: + hit_count: 2 diff --git a/packages/ti_domaintools/data_stream/domainrdap/agent/stream/cel.yml.hbs b/packages/ti_domaintools/data_stream/domainrdap/agent/stream/cel.yml.hbs new file mode 100644 index 00000000000..78410be48bb --- /dev/null +++ b/packages/ti_domaintools/data_stream/domainrdap/agent/stream/cel.yml.hbs @@ -0,0 +1,64 @@ +config_version: "2" +interval: {{interval}} +resource.tracer: + enabled: {{enable_request_tracer}} + filename: "../../logs/cel/http-request-trace-*.ndjson" + maxbackups: 5 +resource.url: {{api_url}} +state: + api_username: {{api_username}} + api_key: {{api_key}} + session_id: {{session_id}} + app_name: elastic_feeds + app_partner: elastic + app_version: 0.1.0 + top: {{top}} +redact: + fields: + - api_key +program: | + state.with( + request( + "GET", + state.url.trim_right("/") + "/feed/domainrdap/?" + { + "api_username": [state.api_username], + "api_key": [state.api_key], + "sessionID": [state.session_id], + "app_name": [state.app_name], + "app_partner": [state.app_partner], + "app_version": [state.app_version], + }.format_query() + ).with( + { + "Header": { + "Accept": ["application/x-ndjson"], + }, + } + ).do_request().as(resp, (resp.StatusCode == 200 || resp.StatusCode == 206) ? + { + "events": string(resp.Body).split("\n").map(e, e!="", + { + "message": e, + } + ), + "want_more": resp.StatusCode == 206 + } + : + { + "events": { + "error": { + "code": string(resp.StatusCode), + "id": string(resp.Status), + "message": "GET: " + + ( + (size(resp.Body) != 0) ? + string(resp.Body) + : + string(resp.Status) + " (" + string(resp.StatusCode) + ")" + ), + }, + }, + "want_more": false, + } + ) + ) diff --git a/packages/ti_domaintools/data_stream/domainrdap/elasticsearch/ilm/default_policy.json b/packages/ti_domaintools/data_stream/domainrdap/elasticsearch/ilm/default_policy.json new file mode 100644 index 00000000000..68d2c5e57a6 --- /dev/null +++ b/packages/ti_domaintools/data_stream/domainrdap/elasticsearch/ilm/default_policy.json @@ -0,0 +1,23 @@ +{ + "policy": { + "phases": { + "hot": { + "actions": { + "rollover": { + "max_age": "2d", + "max_size": "50gb" + }, + "set_priority": { + "priority": 100 + } + } + }, + "delete": { + "min_age": "3d", + "actions": { + "delete": {} + } + } + } + } +} \ No newline at end of file diff --git a/packages/ti_domaintools/data_stream/domainrdap/elasticsearch/ingest_pipeline/default.yml b/packages/ti_domaintools/data_stream/domainrdap/elasticsearch/ingest_pipeline/default.yml new file mode 100644 index 00000000000..7193f31c3b5 --- /dev/null +++ b/packages/ti_domaintools/data_stream/domainrdap/elasticsearch/ingest_pipeline/default.yml @@ -0,0 +1,95 @@ +--- +description: Pipeline for processing domain rdap feed +processors: +- json: + field: message + target_field: domaintools + on_failure: + - set: + field: error.message + value: "Failed to parse JSON from message: {{ ctx.message }} {{ error.message }}" + +- set: + if: ctx.domaintools?.raw_record?.first_request_timestamp != null + field: domaintools.first_request_timestamp + copy_from: domaintools.raw_record.first_request_timestamp + +- foreach: + field: domaintools.raw_record.requests + processor: + append: + field: domaintools.requests_url + value: "{{ _ingest._value.url }}" + + + ############################ + # Generic indicator fields # + ############################ + +- set: + field: threat.indicator.type + value: domain-name +- set: + if: ctx.domaintools?.domain != null + field: threat.indicator.name + copy_from: domaintools.domain + + ###################### + # Threat feed fields # + ###################### + +- set: + field: threat.feed.description + value: 'Changes to global domain registration information, populated by the Registration Data Access Protocol (RDAP). Compliments the 5-Minute WHOIS Feed as registries and registrars switch from Whois to RDAP.' +- set: + field: threat.feed.name + value: 'DomainTools domain RDAP' +- set: + field: threat.feed.reference + value: https://docs.techdocs.ci.domaintools.cloud/feeds/realtime/userguide/ + + #################### + # Event ECS fields # + #################### + +- set: + field: ecs.version + value: '8.11.0' +- set: + field: event.kind + value: enrichment +- set: + field: event.category + value: ['threat'] +- set: + field: event.type + value: ['indicator'] + +- remove: + field: + - domaintools.raw_record + - domaintools.parsed_record.parsed_fields.conformance + - domaintools.parsed_record.parsed_fields.dnssec + - domaintools.parsed_record.parsed_fields.domain + - domaintools.parsed_record.parsed_fields.domain_statuses + - domaintools.parsed_record.parsed_fields.links + - domaintools.parsed_record.parsed_fields.unclassified_emails + - domaintools.parsed_record.registrar_request_url + - domaintools.parsed_record.registry_request_url + ignore_missing: true + + +on_failure: +- set: + field: event.kind + value: pipeline_error +- append: + field: tags + value: preserve_original_event + allow_duplicates: false +- append: + field: error.message + value: >- + Processor '{{{ _ingest.on_failure_processor_type }}}' + {{#_ingest.on_failure_processor_tag}}with tag '{{{ _ingest.on_failure_processor_tag }}}' + {{/_ingest.on_failure_processor_tag}}failed with message '{{{ _ingest.on_failure_message }}}' diff --git a/packages/ti_domaintools/data_stream/domainrdap/fields/base-fields.yml b/packages/ti_domaintools/data_stream/domainrdap/fields/base-fields.yml new file mode 100644 index 00000000000..0f3844069d4 --- /dev/null +++ b/packages/ti_domaintools/data_stream/domainrdap/fields/base-fields.yml @@ -0,0 +1,8 @@ +- name: data_stream.type + external: ecs +- name: data_stream.dataset + external: ecs +- name: data_stream.namespace + external: ecs +- name: "@timestamp" + external: ecs diff --git a/packages/ti_domaintools/data_stream/domainrdap/fields/ecs.yml b/packages/ti_domaintools/data_stream/domainrdap/fields/ecs.yml new file mode 100644 index 00000000000..664ff10ce6a --- /dev/null +++ b/packages/ti_domaintools/data_stream/domainrdap/fields/ecs.yml @@ -0,0 +1,24 @@ +- name: ecs.version + external: ecs +- name: error.message + external: ecs +- name: event.category + external: ecs +- name: event.id + external: ecs +- name: event.ingested + external: ecs +- name: event.kind + external: ecs +- name: event.type + external: ecs +- name: threat.indicator.type + external: ecs +- name: threat.indicator.name + external: ecs +- name: threat.feed.description + external: ecs +- name: threat.feed.name + external: ecs +- name: threat.feed.reference + external: ecs diff --git a/packages/ti_domaintools/data_stream/domainrdap/fields/fields.yml b/packages/ti_domaintools/data_stream/domainrdap/fields/fields.yml new file mode 100644 index 00000000000..594851d4ccc --- /dev/null +++ b/packages/ti_domaintools/data_stream/domainrdap/fields/fields.yml @@ -0,0 +1,107 @@ +- name: domaintools + type: group + fields: + - name: domain + type: keyword + description: > + The Domain. + + - name: feed + type: constant_keyword + value: "domainrdap" + description: > + The feed type. + + - name: timestamp + type: date + description: > + Timestamp when the domain was added to the DomainTools feed, in ISO 8601 UTC form. + + - name: first_request_timestamp + type: date + description: > + The first request timestamp. + + - name: requests_url + type: keyword + description: > + List of extracted rdap request urls used. + + - name: parsed_record + type: group + fields: + - name: parsed_fields + type: group + fields: + - name: contacts + type: group + description: > + List of contact objects extracted from RDAP response. + fields: + - name: name + type: keyword + - name: email + type: keyword + - name: country + type: keyword + - name: email_domains + type: keyword + description: > + List of email domains. + - name: emails + type: keyword + description: > + List of emails. + - name: creation_date + type: keyword + description: > + The domain creation date. + - name: expiration_date + type: keyword + description: > + The domain expiraton date. + - name: last_changed_date + type: keyword + description: > + The domain last changed date. + - name: handle + type: keyword + description: > + The domain handle. + - name: nameservers + type: keyword + description: > + The domain nameservers. + - name: registrar + type: group + description: > + The registrar contact from RDAP response. + fields: + - name: contacts + type: group + description: > + List of registrar contact objects extracted from RDAP response. + fields: + - name: name + type: keyword + - name: email + type: keyword + - name: phone + type: keyword + - name: roles + type: keyword + - name: iana_id + type: keyword + - name: name + type: keyword + description: > + The registrar name. + +- name: message + external: ecs + description: > + The feed from DomainTools Feed API. + +- name: input.type + type: keyword + description: Type of filebeat input. diff --git a/packages/ti_domaintools/data_stream/domainrdap/fields/is-ioc-transform-source-true.yml b/packages/ti_domaintools/data_stream/domainrdap/fields/is-ioc-transform-source-true.yml new file mode 100644 index 00000000000..b1d45027981 --- /dev/null +++ b/packages/ti_domaintools/data_stream/domainrdap/fields/is-ioc-transform-source-true.yml @@ -0,0 +1,4 @@ +- name: labels.is_ioc_transform_source + type: constant_keyword + value: "true" + description: Indicates whether an IOC is in the raw source data stream, or the in latest destination index. diff --git a/packages/ti_domaintools/data_stream/domainrdap/lifecycle.yml b/packages/ti_domaintools/data_stream/domainrdap/lifecycle.yml new file mode 100644 index 00000000000..5a4af9095b7 --- /dev/null +++ b/packages/ti_domaintools/data_stream/domainrdap/lifecycle.yml @@ -0,0 +1 @@ +data_retention: "5d" diff --git a/packages/ti_domaintools/data_stream/domainrdap/manifest.yml b/packages/ti_domaintools/data_stream/domainrdap/manifest.yml new file mode 100644 index 00000000000..1c471f4c0fd --- /dev/null +++ b/packages/ti_domaintools/data_stream/domainrdap/manifest.yml @@ -0,0 +1,66 @@ +title: "DomainTools Domain RDAP" +type: logs +ilm_policy: logs-ti_domaintools.domainrdap-default_policy +streams: + - input: cel + vars: + - name: api_url + type: text + title: DomainTools API URL + multi: false + required: true + show_user: true + default: https://api.domaintools.com/v1 + description: The URL of the DomainTools API. + - name: interval + type: text + title: Interval + multi: false + required: true + show_user: true + default: 10m + description: Interval at which the feed will be pulled. Supported units for this parameter are h/m/s. + - name: api_username + type: text + title: DomainTools API Username + multi: false + required: true + show_user: true + default: DomainTools API Username + description: DomainTools API Username + - name: api_key + type: password + title: DomainTools API Key + multi: false + required: true + show_user: true + secret: true + description: DomainTools API Key + - name: session_id + type: text + title: Session ID + multi: false + required: true + show_user: true + default: DomainToolsElasticSID + description: The Session ID to use in requesting feed. + - name: top + type: text + title: Top + multi: false + required: true + show_user: true + default: "100" + description: Limits the number of results in the response payload. + - name: enable_request_tracer + type: bool + title: Enable request tracing + description: >- + The request tracer logs requests and responses to the agent's local file-system for debugging configurations. Enabling this request tracing compromises security and should only be used for debugging. Disabling the request tracer will delete any stored traces. See [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-cel.html#_resource_tracer_enable) for details. + default: false + multi: false + required: false + show_user: false + title: DomainTools Domain RDAP + description: Subscribe to DomainTools Domain RDAP + template_path: cel.yml.hbs diff --git a/packages/ti_domaintools/data_stream/domainrdap/sample_event.json b/packages/ti_domaintools/data_stream/domainrdap/sample_event.json new file mode 100644 index 00000000000..78212cc0ca1 --- /dev/null +++ b/packages/ti_domaintools/data_stream/domainrdap/sample_event.json @@ -0,0 +1,110 @@ +{ + "input": { + "type": "cel" + }, + "agent": { + "name": "docker-fleet-agent", + "id": "da8d0a37-2d46-4788-96bd-e9ee19e332ec", + "ephemeral_id": "d1cbe648-0a1d-48e8-a161-cd82403e623e", + "type": "filebeat", + "version": "8.15.3" + }, + "@timestamp": "2025-01-30T20:15:25.396Z", + "ecs": { + "version": "8.11.0" + }, + "data_stream": { + "namespace": "default", + "type": "logs", + "dataset": "ti_domaintools.domainrdap" + }, + "host": { + "hostname": "docker-fleet-agent", + "os": { + "kernel": "6.10.11-linuxkit", + "codename": "focal", + "name": "Ubuntu", + "type": "linux", + "family": "debian", + "version": "20.04.6 LTS (Focal Fossa)", + "platform": "ubuntu" + }, + "containerized": false, + "ip": [ + "172.19.0.10" + ], + "name": "docker-fleet-agent", + "id": "cfae1e7244ae479b9b0968259c91b13a", + "mac": [ + "02-42-AC-13-00-0A" + ], + "architecture": "aarch64" + }, + "elastic_agent": { + "id": "da8d0a37-2d46-4788-96bd-e9ee19e332ec", + "version": "8.15.3", + "snapshot": false + }, + "domaintools": { + "domain": "unlockyourlifehere.com", + "timestamp": "2025-06-12T20:34:31Z", + "feed": "domainrdap", + "first_request_timestamp": "2025-06-12T20:34:24Z", + "requests_url": [ + "https://rdap.verisign.com/com/v1/domain/unlockyourlifehere.com" + ], + "parsed_record": { + "parsed_fields": { + "emails": [ + "abuse@godaddy.com" + ], + "last_changed_date": "2025-05-20T02: 44: 33+00: 00", + "registrar": { + "name": "GoDaddy.com, LLC", + "contacts": [ + { + "name": "", + "phone": "tel:480-624-2505", + "email": "abuse@godaddy.com", + "roles": [ + "abuse" + ] + } + ], + "iana_id": "146" + }, + "handle": "2894681047_DOMAIN_COM-VRSN", + "creation_date": "2024-06-28T11: 49: 19+00: 00", + "expiration_date": "2025-06-28T11: 49: 19+00: 00", + "email_domains": [ + "godaddy.com" + ], + "contacts": [] + } + } + }, + "threat": { + "indicator": { + "name": "unlockyourlifehere.com", + "type": "domain-name" + }, + "feed": { + "reference": "https://docs.techdocs.ci.domaintools.cloud/feeds/realtime/userguide/", + "name": "DomainTools domain RDAP", + "description": "Changes to global domain registration information, populated by the Registration Data Access Protocol (RDAP). Compliments the 5-Minute WHOIS Feed as registries and registrars switch from Whois to RDAP." + } + }, + "message": "{\"timestamp\":\"2025-06-12T20:34:31Z\",\"domain\":\"unlockyourlifehere.com\",\"raw_record\":{\"first_request_timestamp\":\"2025-06-12T20:34:24Z\",\"requests\":[{\"data\":\"{\\\"objectClassName\\\":\\\"domain\\\",\\\"handle\\\":\\\"2894681047_DOMAIN_COM-VRSN\\\",\\\"ldhName\\\":\\\"UNLOCKYOURLIFEHERE.COM\\\",\\\"links\\\":[{\\\"value\\\":\\\"https:\\\\/\\\\/rdap.verisign.com\\\\/com\\\\/v1\\\\/domain\\\\/UNLOCKYOURLIFEHERE.COM\\\",\\\"rel\\\":\\\"self\\\",\\\"href\\\":\\\"https:\\\\/\\\\/rdap.verisign.com\\\\/com\\\\/v1\\\\/domain\\\\/UNLOCKYOURLIFEHERE.COM\\\",\\\"type\\\":\\\"application\\\\/rdap+json\\\"},{\\\"value\\\":\\\"https:\\\\/\\\\/rdap.godaddy.com\\\\/v1\\\\/domain\\\\/UNLOCKYOURLIFEHERE.COM\\\",\\\"rel\\\":\\\"related\\\",\\\"href\\\":\\\"https:\\\\/\\\\/rdap.godaddy.com\\\\/v1\\\\/domain\\\\/UNLOCKYOURLIFEHERE.COM\\\",\\\"type\\\":\\\"application\\\\/rdap+json\\\"}],\\\"status\\\":[\\\"redemption period\\\"],\\\"entities\\\":[{\\\"objectClassName\\\":\\\"entity\\\",\\\"handle\\\":\\\"146\\\",\\\"roles\\\":[\\\"registrar\\\"],\\\"publicIds\\\":[{\\\"type\\\":\\\"IANA Registrar ID\\\",\\\"identifier\\\":\\\"146\\\"}],\\\"vcardArray\\\":[\\\"vcard\\\",[[\\\"version\\\",{},\\\"text\\\",\\\"4.0\\\"],[\\\"fn\\\",{},\\\"text\\\",\\\"GoDaddy.com, LLC\\\"]]],\\\"entities\\\":[{\\\"objectClassName\\\":\\\"entity\\\",\\\"roles\\\":[\\\"abuse\\\"],\\\"vcardArray\\\":[\\\"vcard\\\",[[\\\"version\\\",{},\\\"text\\\",\\\"4.0\\\"],[\\\"fn\\\",{},\\\"text\\\",\\\"\\\"],[\\\"tel\\\",{\\\"type\\\":\\\"voice\\\"},\\\"uri\\\",\\\"tel:480-624-2505\\\"],[\\\"email\\\",{},\\\"text\\\",\\\"abuse@godaddy.com\\\"]]]}]}],\\\"events\\\":[{\\\"eventAction\\\":\\\"registration\\\",\\\"eventDate\\\":\\\"2024-06-28T11:49:19Z\\\"},{\\\"eventAction\\\":\\\"expiration\\\",\\\"eventDate\\\":\\\"2025-06-28T11:49:19Z\\\"},{\\\"eventAction\\\":\\\"last changed\\\",\\\"eventDate\\\":\\\"2025-05-20T02:44:33Z\\\"},{\\\"eventAction\\\":\\\"last update of RDAP database\\\",\\\"eventDate\\\":\\\"2025-06-12T20:34:16Z\\\"}],\\\"secureDNS\\\":{\\\"delegationSigned\\\":false},\\\"rdapConformance\\\":[\\\"rdap_level_0\\\",\\\"icann_rdap_technical_implementation_guide_0\\\",\\\"icann_rdap_response_profile_0\\\"],\\\"notices\\\":[{\\\"title\\\":\\\"Terms of Use\\\",\\\"description\\\":[\\\"Service subject to Terms of Use.\\\"],\\\"links\\\":[{\\\"href\\\":\\\"https:\\\\/\\\\/www.verisign.com\\\\/domain-names\\\\/registration-data-access-protocol\\\\/terms-service\\\\/index.xhtml\\\",\\\"type\\\":\\\"text\\\\/html\\\"}]},{\\\"title\\\":\\\"Status Codes\\\",\\\"description\\\":[\\\"For more information on domain status codes, please visit https:\\\\/\\\\/icann.org\\\\/epp\\\"],\\\"links\\\":[{\\\"href\\\":\\\"https:\\\\/\\\\/icann.org\\\\/epp\\\",\\\"type\\\":\\\"text\\\\/html\\\"}]},{\\\"title\\\":\\\"RDDS Inaccuracy Complaint Form\\\",\\\"description\\\":[\\\"URL of the ICANN RDDS Inaccuracy Complaint Form: https:\\\\/\\\\/icann.org\\\\/wicf\\\"],\\\"links\\\":[{\\\"href\\\":\\\"https:\\\\/\\\\/icann.org\\\\/wicf\\\",\\\"type\\\":\\\"text\\\\/html\\\"}]}]}\",\"source_type\":\"registry\",\"timestamp\":\"2025-06-12T20:34:24Z\",\"url\":\"https://rdap.verisign.com/com/v1/domain/unlockyourlifehere.com\"}]},\"parsed_record\":{\"parsed_fields\":{\"conformance\":[\"rdap_level_0\",\"icann_rdap_technical_implementation_guide_0\",\"icann_rdap_response_profile_0\"],\"contacts\":[],\"creation_date\":\"2024-06-28T11: 49: 19+00: 00\",\"dnssec\":{\"signed\":false},\"domain\":\"UNLOCKYOURLIFEHERE.COM\",\"domain_statuses\":[\"redemption period\"],\"email_domains\":[\"godaddy.com\"],\"emails\":[\"abuse@godaddy.com\"],\"expiration_date\":\"2025-06-28T11: 49: 19+00: 00\",\"handle\":\"2894681047_DOMAIN_COM-VRSN\",\"last_changed_date\":\"2025-05-20T02: 44: 33+00: 00\",\"links\":[{\"href\":\"https://rdap.verisign.com/com/v1/domain/UNLOCKYOURLIFEHERE.COM\",\"rel\":\"self\"},{\"href\":\"https://rdap.godaddy.com/v1/domain/UNLOCKYOURLIFEHERE.COM\",\"rel\":\"related\"}],\"registrar\":{\"contacts\":[{\"email\":\"abuse@godaddy.com\",\"name\":\"\",\"phone\":\"tel:480-624-2505\",\"roles\":[\"abuse\"]}],\"iana_id\":\"146\",\"name\":\"GoDaddy.com, LLC\"},\"unclassified_emails\":[]},\"registrar_request_url\":null,\"registry_request_url\":\"https://rdap.verisign.com/com/v1/domain/unlockyourlifehere.com\"}}", + "event": { + "agent_id_status": "verified", + "ingested": "2025-06-12T20:34:31Z", + "kind": "enrichment", + "category": [ + "threat" + ], + "type": [ + "indicator" + ], + "dataset": "ti_domaintools.domainrdap" + } +} \ No newline at end of file diff --git a/packages/ti_domaintools/data_stream/nad_feed/_dev/test/pipeline/test-event.json b/packages/ti_domaintools/data_stream/nad_feed/_dev/test/pipeline/test-event.json new file mode 100644 index 00000000000..75451670f59 --- /dev/null +++ b/packages/ti_domaintools/data_stream/nad_feed/_dev/test/pipeline/test-event.json @@ -0,0 +1,13 @@ +{ + "events": [ + { + "message": "{\"timestamp\":\"2025-01-11T08:42:46Z\",\"domain\":\"ccnitsolution.com\"}" + }, + { + "message": "{\"timestamp\":\"2025-01-11T08:42:46Z\",\"domain\":\"ccnitsolution2.com\"}" + }, + { + "message": "{\"timestamp\":\"2025-01-11T08:42:46Z\",\"domain\":\"ccnitsolution3.com\"}" + } + ] +} \ No newline at end of file diff --git a/packages/ti_domaintools/data_stream/nad_feed/_dev/test/pipeline/test-event.json-expected.json b/packages/ti_domaintools/data_stream/nad_feed/_dev/test/pipeline/test-event.json-expected.json new file mode 100644 index 00000000000..86ea0b52b63 --- /dev/null +++ b/packages/ti_domaintools/data_stream/nad_feed/_dev/test/pipeline/test-event.json-expected.json @@ -0,0 +1,94 @@ +{ + "expected": [ + { + "domaintools": { + "domain": "ccnitsolution.com", + "timestamp": "2025-01-11T08:42:46Z" + }, + "ecs": { + "version": "8.11.0" + }, + "event": { + "category": [ + "threat" + ], + "kind": "enrichment", + "type": [ + "indicator" + ] + }, + "message": "{\"timestamp\":\"2025-01-11T08:42:46Z\",\"domain\":\"ccnitsolution.com\"}", + "threat": { + "feed": { + "description": "Apex-level domains (e.g. example.com but not www.example.com) that we observe for the first time, and have not observed previously with our global DNS sensor network.", + "name": "DomainTools NAD", + "reference": "https://docs.techdocs.ci.domaintools.cloud/feeds/realtime/userguide/" + }, + "indicator": { + "name": "ccnitsolution.com", + "type": "domain-name" + } + } + }, + { + "domaintools": { + "domain": "ccnitsolution2.com", + "timestamp": "2025-01-11T08:42:46Z" + }, + "ecs": { + "version": "8.11.0" + }, + "event": { + "category": [ + "threat" + ], + "kind": "enrichment", + "type": [ + "indicator" + ] + }, + "message": "{\"timestamp\":\"2025-01-11T08:42:46Z\",\"domain\":\"ccnitsolution2.com\"}", + "threat": { + "feed": { + "description": "Apex-level domains (e.g. example.com but not www.example.com) that we observe for the first time, and have not observed previously with our global DNS sensor network.", + "name": "DomainTools NAD", + "reference": "https://docs.techdocs.ci.domaintools.cloud/feeds/realtime/userguide/" + }, + "indicator": { + "name": "ccnitsolution2.com", + "type": "domain-name" + } + } + }, + { + "domaintools": { + "domain": "ccnitsolution3.com", + "timestamp": "2025-01-11T08:42:46Z" + }, + "ecs": { + "version": "8.11.0" + }, + "event": { + "category": [ + "threat" + ], + "kind": "enrichment", + "type": [ + "indicator" + ] + }, + "message": "{\"timestamp\":\"2025-01-11T08:42:46Z\",\"domain\":\"ccnitsolution3.com\"}", + "threat": { + "feed": { + "description": "Apex-level domains (e.g. example.com but not www.example.com) that we observe for the first time, and have not observed previously with our global DNS sensor network.", + "name": "DomainTools NAD", + "reference": "https://docs.techdocs.ci.domaintools.cloud/feeds/realtime/userguide/" + }, + "indicator": { + "name": "ccnitsolution3.com", + "type": "domain-name" + } + } + } + ] +} \ No newline at end of file diff --git a/packages/ti_domaintools/data_stream/nad_feed/_dev/test/system/test-default-config.yml b/packages/ti_domaintools/data_stream/nad_feed/_dev/test/system/test-default-config.yml new file mode 100644 index 00000000000..31a0f441817 --- /dev/null +++ b/packages/ti_domaintools/data_stream/nad_feed/_dev/test/system/test-default-config.yml @@ -0,0 +1,11 @@ +input: cel +service: ti_domaintools +vars: +data_stream: + vars: + api_url: http://{{Hostname}}:{{Port}}/v1 + interval: 10m + api_username: xxx + api_key: xxx +assert: + hit_count: 2 diff --git a/packages/ti_domaintools/data_stream/nad_feed/agent/stream/cel.yml.hbs b/packages/ti_domaintools/data_stream/nad_feed/agent/stream/cel.yml.hbs new file mode 100644 index 00000000000..4c1c7216118 --- /dev/null +++ b/packages/ti_domaintools/data_stream/nad_feed/agent/stream/cel.yml.hbs @@ -0,0 +1,63 @@ +config_version: "2" +interval: {{interval}} +resource.tracer: + enabled: {{enable_request_tracer}} + filename: "../../logs/cel/http-request-trace-*.ndjson" + maxbackups: 5 +resource.url: {{api_url}} +state: + api_username: {{api_username}} + api_key: {{api_key}} + session_id: {{session_id}} + app_name: elastic_feeds + app_partner: elastic + app_version: 0.1.0 +redact: + fields: + - api_key +program: | + state.with( + request( + "GET", + state.url.trim_right("/") + "/feed/nad/?" + { + "api_username": [state.api_username], + "api_key": [state.api_key], + "sessionID": [state.session_id], + "app_name": [state.app_name], + "app_partner": [state.app_partner], + "app_version": [state.app_version], + }.format_query() + ).with( + { + "Header": { + "Accept": ["application/x-ndjson"], + }, + } + ).do_request().as(resp, (resp.StatusCode == 200 || resp.StatusCode == 206) ? + { + "events": string(resp.Body).split("\n").map(e, e!="", + { + "message": e, + } + ), + "want_more": resp.StatusCode == 206 + } + : + { + "events": { + "error": { + "code": string(resp.StatusCode), + "id": string(resp.Status), + "message": "GET: " + + ( + (size(resp.Body) != 0) ? + string(resp.Body) + : + string(resp.Status) + " (" + string(resp.StatusCode) + ")" + ), + }, + }, + "want_more": false, + } + ) + ) diff --git a/packages/ti_domaintools/data_stream/nad_feed/elasticsearch/ilm/default_policy.json b/packages/ti_domaintools/data_stream/nad_feed/elasticsearch/ilm/default_policy.json new file mode 100644 index 00000000000..68d2c5e57a6 --- /dev/null +++ b/packages/ti_domaintools/data_stream/nad_feed/elasticsearch/ilm/default_policy.json @@ -0,0 +1,23 @@ +{ + "policy": { + "phases": { + "hot": { + "actions": { + "rollover": { + "max_age": "2d", + "max_size": "50gb" + }, + "set_priority": { + "priority": 100 + } + } + }, + "delete": { + "min_age": "3d", + "actions": { + "delete": {} + } + } + } + } +} \ No newline at end of file diff --git a/packages/ti_domaintools/data_stream/nad_feed/elasticsearch/ingest_pipeline/default.yml b/packages/ti_domaintools/data_stream/nad_feed/elasticsearch/ingest_pipeline/default.yml new file mode 100644 index 00000000000..54d3e33bfb1 --- /dev/null +++ b/packages/ti_domaintools/data_stream/nad_feed/elasticsearch/ingest_pipeline/default.yml @@ -0,0 +1,65 @@ +--- +description: Pipeline for processing nad feed +processors: +- json: + if: ctx?.message != null + field: message + target_field: domaintools + + ############################ + # Generic indicator fields # + ############################ + +- set: + field: threat.indicator.type + value: domain-name +- set: + if: ctx.domaintools?.domain != null + field: threat.indicator.name + copy_from: domaintools.domain + + ###################### + # Threat feed fields # + ###################### + +- set: + field: threat.feed.description + value: 'Apex-level domains (e.g. example.com but not www.example.com) that we observe for the first time, and have not observed previously with our global DNS sensor network.' +- set: + field: threat.feed.name + value: 'DomainTools NAD' +- set: + field: threat.feed.reference + value: https://docs.techdocs.ci.domaintools.cloud/feeds/realtime/userguide/ + + #################### + # Event ECS fields # + #################### + +- set: + field: ecs.version + value: '8.11.0' +- set: + field: event.kind + value: enrichment +- set: + field: event.category + value: ['threat'] +- set: + field: event.type + value: ['indicator'] + +on_failure: +- set: + field: event.kind + value: pipeline_error +- append: + field: tags + value: preserve_original_event + allow_duplicates: false +- append: + field: error.message + value: >- + Processor '{{{ _ingest.on_failure_processor_type }}}' + {{#_ingest.on_failure_processor_tag}}with tag '{{{ _ingest.on_failure_processor_tag }}}' + {{/_ingest.on_failure_processor_tag}}failed with message '{{{ _ingest.on_failure_message }}}' diff --git a/packages/ti_domaintools/data_stream/nad_feed/fields/base-fields.yml b/packages/ti_domaintools/data_stream/nad_feed/fields/base-fields.yml new file mode 100644 index 00000000000..0f3844069d4 --- /dev/null +++ b/packages/ti_domaintools/data_stream/nad_feed/fields/base-fields.yml @@ -0,0 +1,8 @@ +- name: data_stream.type + external: ecs +- name: data_stream.dataset + external: ecs +- name: data_stream.namespace + external: ecs +- name: "@timestamp" + external: ecs diff --git a/packages/ti_domaintools/data_stream/nad_feed/fields/ecs.yml b/packages/ti_domaintools/data_stream/nad_feed/fields/ecs.yml new file mode 100644 index 00000000000..664ff10ce6a --- /dev/null +++ b/packages/ti_domaintools/data_stream/nad_feed/fields/ecs.yml @@ -0,0 +1,24 @@ +- name: ecs.version + external: ecs +- name: error.message + external: ecs +- name: event.category + external: ecs +- name: event.id + external: ecs +- name: event.ingested + external: ecs +- name: event.kind + external: ecs +- name: event.type + external: ecs +- name: threat.indicator.type + external: ecs +- name: threat.indicator.name + external: ecs +- name: threat.feed.description + external: ecs +- name: threat.feed.name + external: ecs +- name: threat.feed.reference + external: ecs diff --git a/packages/ti_domaintools/data_stream/nad_feed/fields/fields.yml b/packages/ti_domaintools/data_stream/nad_feed/fields/fields.yml new file mode 100644 index 00000000000..c4b73865a48 --- /dev/null +++ b/packages/ti_domaintools/data_stream/nad_feed/fields/fields.yml @@ -0,0 +1,27 @@ +- name: domaintools + type: group + fields: + - name: domain + type: keyword + description: > + The Domain. Apex-level domains (e.g. example.com but not www.example.com) that we observe for the first time, and have not observed previously with our global DNS sensor network. + + - name: feed + type: constant_keyword + value: "nad123" + description: > + The feed. + + - name: timestamp + type: date + description: > + Timestamp when the domain was added to the DomainTools feed, in ISO 8601 UTC form. + +- name: message + external: ecs + description: > + The feed. + +- name: input.type + type: keyword + description: Type of filebeat input. diff --git a/packages/ti_domaintools/data_stream/nad_feed/fields/is-ioc-transform-source-true.yml b/packages/ti_domaintools/data_stream/nad_feed/fields/is-ioc-transform-source-true.yml new file mode 100644 index 00000000000..b1d45027981 --- /dev/null +++ b/packages/ti_domaintools/data_stream/nad_feed/fields/is-ioc-transform-source-true.yml @@ -0,0 +1,4 @@ +- name: labels.is_ioc_transform_source + type: constant_keyword + value: "true" + description: Indicates whether an IOC is in the raw source data stream, or the in latest destination index. diff --git a/packages/ti_domaintools/data_stream/nad_feed/lifecycle.yml b/packages/ti_domaintools/data_stream/nad_feed/lifecycle.yml new file mode 100644 index 00000000000..5a4af9095b7 --- /dev/null +++ b/packages/ti_domaintools/data_stream/nad_feed/lifecycle.yml @@ -0,0 +1 @@ +data_retention: "5d" diff --git a/packages/ti_domaintools/data_stream/nad_feed/manifest.yml b/packages/ti_domaintools/data_stream/nad_feed/manifest.yml new file mode 100644 index 00000000000..87e138c543e --- /dev/null +++ b/packages/ti_domaintools/data_stream/nad_feed/manifest.yml @@ -0,0 +1,62 @@ +title: "DomainTools Newly Active Domains Feed" +type: logs +ilm_policy: logs-ti_domaintools.nad_feed-default_policy +streams: + - input: cel + vars: + - name: api_url + type: text + title: DomainTools API URL + multi: false + required: true + show_user: true + default: https://api.domaintools.com/v1 + description: The URL of the DomainTools API. + - name: interval + type: text + title: Interval + multi: false + required: true + show_user: true + default: 10m + description: Interval at which the feed will be pulled. Supported units for this parameter are h/m/s. + - name: api_username + type: text + title: DomainTools API Username + multi: false + required: true + show_user: true + default: DomainTools API Username + description: DomainTools API Username + - name: api_key + type: password + title: DomainTools API Key + multi: false + required: true + show_user: true + secret: true + description: DomainTools API Key + - name: session_id + type: text + title: Session ID + multi: false + required: true + show_user: true + default: DomainToolsElasticSID + description: The Session ID to use in requesting feed. + - name: enable_request_tracer + type: bool + title: Enable request tracing + description: >- + The request tracer logs requests and responses to the agent's local file-system for debugging configurations. + Enabling this request tracing compromises security and should only be used for debugging. Disabling the request + tracer will delete any stored traces. + See [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-cel.html#_resource_tracer_enable) + for details. + default: false + multi: false + required: false + show_user: false + title: DomainTools Newly Active Domains Feed + description: Subscribe to DomainTools Newly Active Domains Feed + template_path: cel.yml.hbs diff --git a/packages/ti_domaintools/data_stream/nad_feed/sample_event.json b/packages/ti_domaintools/data_stream/nad_feed/sample_event.json new file mode 100644 index 00000000000..0fcdd858928 --- /dev/null +++ b/packages/ti_domaintools/data_stream/nad_feed/sample_event.json @@ -0,0 +1,77 @@ +{ + "input": { + "type": "cel" + }, + "agent": { + "name": "docker-fleet-agent", + "id": "da8d0a37-2d46-4788-96bd-e9ee19e332ec", + "ephemeral_id": "d1cbe648-0a1d-48e8-a161-cd82403e623e", + "type": "filebeat", + "version": "8.15.3" + }, + "@timestamp": "2025-01-30T20:15:25.396Z", + "ecs": { + "version": "8.11.0" + }, + "data_stream": { + "namespace": "default", + "type": "logs", + "dataset": "ti_domaintools.nad_feed" + }, + "host": { + "hostname": "docker-fleet-agent", + "os": { + "kernel": "6.10.11-linuxkit", + "codename": "focal", + "name": "Ubuntu", + "type": "linux", + "family": "debian", + "version": "20.04.6 LTS (Focal Fossa)", + "platform": "ubuntu" + }, + "containerized": false, + "ip": [ + "172.19.0.10" + ], + "name": "docker-fleet-agent", + "id": "cfae1e7244ae479b9b0968259c91b13a", + "mac": [ + "02-42-AC-13-00-0A" + ], + "architecture": "aarch64" + }, + "elastic_agent": { + "id": "da8d0a37-2d46-4788-96bd-e9ee19e332ec", + "version": "8.15.3", + "snapshot": false + }, + "domaintools": { + "domain": "tractorpoweredcoreaerator.com", + "feed": "nad", + "timestamp": "2025-01-30T20:14:48Z" + }, + "threat": { + "indicator": { + "name": "tractorpoweredcoreaerator.com", + "type": "domain-name" + }, + "feed": { + "reference": "https://docs.techdocs.ci.domaintools.cloud/feeds/realtime/userguide/", + "name": "DomainTools NAD", + "description": "Apex-level domains (e.g. example.com but not www.example.com) that we observe for the first time, and have not observed previously with our global DNS sensor network." + } + }, + "message": "{\"timestamp\":\"2025-01-30T20:14:48Z\",\"domain\":\"tractorpoweredcoreaerator.com\"}", + "event": { + "agent_id_status": "verified", + "ingested": "2025-01-30T20:15:26Z", + "kind": "enrichment", + "category": [ + "threat" + ], + "type": [ + "indicator" + ], + "dataset": "ti_domaintools.nad_feed" + } +} \ No newline at end of file diff --git a/packages/ti_domaintools/data_stream/nod_feed/_dev/test/pipeline/test-event.json-expected.json b/packages/ti_domaintools/data_stream/nod_feed/_dev/test/pipeline/test-event.json-expected.json index ed1f112b114..0021dce75af 100644 --- a/packages/ti_domaintools/data_stream/nod_feed/_dev/test/pipeline/test-event.json-expected.json +++ b/packages/ti_domaintools/data_stream/nod_feed/_dev/test/pipeline/test-event.json-expected.json @@ -91,4 +91,4 @@ } } ] -} +} \ No newline at end of file diff --git a/packages/ti_domaintools/data_stream/nod_feed/_dev/test/system/test-default-config.yml b/packages/ti_domaintools/data_stream/nod_feed/_dev/test/system/test-default-config.yml index 84dfca9d63b..31a0f441817 100644 --- a/packages/ti_domaintools/data_stream/nod_feed/_dev/test/system/test-default-config.yml +++ b/packages/ti_domaintools/data_stream/nod_feed/_dev/test/system/test-default-config.yml @@ -1,5 +1,5 @@ input: cel -service: domaintools +service: ti_domaintools vars: data_stream: vars: diff --git a/packages/ti_domaintools/data_stream/nod_feed/agent/stream/cel.yml.hbs b/packages/ti_domaintools/data_stream/nod_feed/agent/stream/cel.yml.hbs index caadb3970d2..6cbda681970 100644 --- a/packages/ti_domaintools/data_stream/nod_feed/agent/stream/cel.yml.hbs +++ b/packages/ti_domaintools/data_stream/nod_feed/agent/stream/cel.yml.hbs @@ -8,7 +8,7 @@ resource.url: {{api_url}} state: api_username: {{api_username}} api_key: {{api_key}} - session_id: DomainToolsElasticSID + session_id: {{session_id}} app_name: elastic_feeds app_partner: elastic app_version: 0.1.0 @@ -16,47 +16,48 @@ redact: fields: - api_key program: | - state.with( - request( - "GET", - state.url.trim_right("/") + "/feed/nod/?" + { - "api_username": [state.api_username], - "api_key": [state.api_key], - "sessionID": [state.session_id], - "app_name": [state.app_name], - "app_partner": [state.app_partner], - "app_version": [state.app_version], - }.format_query() - ).with( - { - "Header": { - "Accept": ["application/x-ndjson"], - }, - } - ).do_request().as(resp, (resp.StatusCode == 200) ? - { - "events": string(resp.Body).split("\n").filter(x,x!="").map(e, - { - "message": e, - } - ), - } - : - { - "events": { - "error": { - "code": string(resp.StatusCode), - "id": string(resp.Status), - "message": "GET:" + - ( - (size(resp.Body) != 0) ? - string(resp.Body) - : - string(resp.Status) + " (" + string(resp.StatusCode) + ")" - ), - }, - }, - "want_more": false, - } - ) - ) \ No newline at end of file + state.with( + request( + "GET", + state.url.trim_right("/") + "/feed/nod/?" + { + "api_username": [state.api_username], + "api_key": [state.api_key], + "sessionID": [state.session_id], + "app_name": [state.app_name], + "app_partner": [state.app_partner], + "app_version": [state.app_version], + }.format_query() + ).with( + { + "Header": { + "Accept": ["application/x-ndjson"], + }, + } + ).do_request().as(resp, (resp.StatusCode == 200 || resp.StatusCode == 206) ? + { + "events": string(resp.Body).split("\n").map(e, e!="", + { + "message": e, + } + ), + "want_more": resp.StatusCode == 206 + } + : + { + "events": { + "error": { + "code": string(resp.StatusCode), + "id": string(resp.Status), + "message": "GET: " + + ( + (size(resp.Body) != 0) ? + string(resp.Body) + : + string(resp.Status) + " (" + string(resp.StatusCode) + ")" + ), + }, + }, + "want_more": false, + } + ) + ) diff --git a/packages/ti_domaintools/data_stream/nod_feed/elasticsearch/ingest_pipeline/default.yml b/packages/ti_domaintools/data_stream/nod_feed/elasticsearch/ingest_pipeline/default.yml index 360d8391a77..9c41eb598cd 100644 --- a/packages/ti_domaintools/data_stream/nod_feed/elasticsearch/ingest_pipeline/default.yml +++ b/packages/ti_domaintools/data_stream/nod_feed/elasticsearch/ingest_pipeline/default.yml @@ -1,10 +1,8 @@ --- description: Pipeline for processing nod feed processors: -- set: - field: domaintools.feed - value: 'nod' - json: + if: ctx?.message != null field: message target_field: domaintools @@ -16,6 +14,7 @@ processors: field: threat.indicator.type value: domain-name - set: + if: ctx.domaintools?.domain != null field: threat.indicator.name copy_from: domaintools.domain @@ -52,6 +51,15 @@ processors: on_failure: - set: - field: error.message - value: '{{ _ingest.on_failure_message }}' - + field: event.kind + value: pipeline_error +- append: + field: tags + value: preserve_original_event + allow_duplicates: false +- append: + field: error.message + value: >- + Processor '{{{ _ingest.on_failure_processor_type }}}' + {{#_ingest.on_failure_processor_tag}}with tag '{{{ _ingest.on_failure_processor_tag }}}' + {{/_ingest.on_failure_processor_tag}}failed with message '{{{ _ingest.on_failure_message }}}' diff --git a/packages/ti_domaintools/data_stream/nod_feed/fields/ecs.yml b/packages/ti_domaintools/data_stream/nod_feed/fields/ecs.yml new file mode 100644 index 00000000000..664ff10ce6a --- /dev/null +++ b/packages/ti_domaintools/data_stream/nod_feed/fields/ecs.yml @@ -0,0 +1,24 @@ +- name: ecs.version + external: ecs +- name: error.message + external: ecs +- name: event.category + external: ecs +- name: event.id + external: ecs +- name: event.ingested + external: ecs +- name: event.kind + external: ecs +- name: event.type + external: ecs +- name: threat.indicator.type + external: ecs +- name: threat.indicator.name + external: ecs +- name: threat.feed.description + external: ecs +- name: threat.feed.name + external: ecs +- name: threat.feed.reference + external: ecs diff --git a/packages/ti_domaintools/data_stream/nod_feed/fields/fields.yml b/packages/ti_domaintools/data_stream/nod_feed/fields/fields.yml index 494a628d94a..ed8930060db 100644 --- a/packages/ti_domaintools/data_stream/nod_feed/fields/fields.yml +++ b/packages/ti_domaintools/data_stream/nod_feed/fields/fields.yml @@ -1,6 +1,5 @@ - name: domaintools type: group - fields: - name: domain type: keyword @@ -8,7 +7,8 @@ The Domain. Apex-level domains (e.g. example.com but not www.example.com) that we observe for the first time, and have not observed previously with our global DNS sensor network. - name: feed - type: keyword + type: constant_keyword + value: "nod" description: > The feed. diff --git a/packages/ti_domaintools/data_stream/nod_feed/manifest.yml b/packages/ti_domaintools/data_stream/nod_feed/manifest.yml index 36f3c56a7cd..f31204818bf 100644 --- a/packages/ti_domaintools/data_stream/nod_feed/manifest.yml +++ b/packages/ti_domaintools/data_stream/nod_feed/manifest.yml @@ -1,4 +1,4 @@ -title: "DomainTools Real Time Unified Feeds" +title: "DomainTools Newly Observed Domains Feed" type: logs ilm_policy: logs-ti_domaintools.nod_feed-default_policy streams: @@ -36,6 +36,14 @@ streams: show_user: true secret: true description: DomainTools API Key + - name: session_id + type: text + title: Session ID + multi: false + required: true + show_user: true + default: DomainToolsElasticSID + description: The Session ID to use in requesting feed. - name: enable_request_tracer type: bool title: Enable request tracing @@ -49,6 +57,6 @@ streams: multi: false required: false show_user: false - title: DomainTools Real Time Unified Feeds - description: Subscribe to DomainTools Real Time Unified Feeds + title: DomainTools Newly Observed Domains Feed + description: Subscribe to DomainTools Newly Observed Domains Feed template_path: cel.yml.hbs diff --git a/packages/ti_domaintools/data_stream/nod_feed/sample_event.json b/packages/ti_domaintools/data_stream/nod_feed/sample_event.json index e9d96fcd7f2..29cd66ffb06 100644 --- a/packages/ti_domaintools/data_stream/nod_feed/sample_event.json +++ b/packages/ti_domaintools/data_stream/nod_feed/sample_event.json @@ -20,10 +20,27 @@ "version": "8.17.0" }, "elastic_agent": { - "id": "df6cda61-c87d-40c3-92d1-6eb4f18f3a79", - "snapshot": false, - "version": "8.15.3" + "id": "da8d0a37-2d46-4788-96bd-e9ee19e332ec", + "version": "8.15.3", + "snapshot": false + }, + "domaintools": { + "domain": "tractorpoweredcoreaerator.com", + "feed": "nod", + "timestamp": "2025-01-30T20:14:48Z" + }, + "threat": { + "indicator": { + "name": "tractorpoweredcoreaerator.com", + "type": "domain-name" + }, + "feed": { + "reference": "https://docs.techdocs.ci.domaintools.cloud/feeds/realtime/userguide/", + "name": "DomainTools NOD", + "description": "Apex-level domains (e.g. example.com but not www.example.com) that we observe for the first time, and have not observed previously with our global DNS sensor network." + } }, + "message": "{\"timestamp\":\"2025-01-30T20:14:48Z\",\"domain\":\"tractorpoweredcoreaerator.com\"}", "event": { "agent_id_status": "verified", "category": [ @@ -75,4 +92,4 @@ "type": "domain-name" } } -} +} \ No newline at end of file diff --git a/packages/ti_domaintools/docs/README.md b/packages/ti_domaintools/docs/README.md index fb28f747b1c..151ecddae29 100644 --- a/packages/ti_domaintools/docs/README.md +++ b/packages/ti_domaintools/docs/README.md @@ -1,40 +1,50 @@ -# DomainTools Real Time Unified Feeds +# DomainTools Feeds -The DomainTools Real Time Unified Feeds integration allows you to monitor DomainTools Newly Observed Domains. -The DomainTools NOD Feed provides real-time access to newly registered and observed domains, enabling proactive threat detection and defense. +DomainTools Feeds provide data on the different stages of the domain lifecycle: from first-observed in the wild, to newly re-activated after a period of quiet. Access current feed data in real-time or retrieve historical feed data through separate APIs. Some feeds also offer data for DNS firewalls in Response Policy Zone (RPZ) format. -With over 300,000 new domains observed daily, the feed empowers security teams to identify and block potentially malicious domains before they can be weaponized. -Ideal for threat hunting, phishing prevention, and brand protection, the NOD Feed delivers unparalleled visibility into emerging domain activity to stay ahead of evolving threats. +Summary of Available Feeds: -For example, if you wanted to monitor Newly Observed Domains (NOD) feed, you could ingest the DomainTools NOD feed. -Then you can reference domaintools.nod_feed when using visualizations or alerts. +- `Newly Active Domains (NAD)`: Apex-level domains (e.g. example.com but not ) that we observe based on the latest lifecycle of the domain. A domain may be seen either for the first time ever, or again after at least 10 days of inactivity (no observed resolutions in DNS). Populated with our global passive DNS (pDNS) sensor network. +- `Newly Observed Domains (NOD)`: Apex-level domains (e.g. example.com but not ) that we observe for the first time, and have not observed previously with our global DNS sensor network. +- `Domain Discovery`: New domains as they are either discovered in domain registration information, observed by our global sensor network, or reported by trusted third parties. +- `Domain RDAP`: Changes to global domain registration information, populated by the Registration Data Access Protocol (RDAP). Compliments the 5-Minute WHOIS Feed as registries and registrars switch from Whois to RDAP. + +With over 300,000 new domains observed daily, the feed empowers security teams to identify and block potentially malicious domains before they can be weaponized. +Ideal for threat hunting, phishing prevention, and brand protection. + +For example, if you wanted to monitor Newly Observed Domains (NOD) feed, you could ingest the DomainTools NOD feed. +Then you can reference ti_domaintools.nod_feed when using visualizations or alerts. ## Data streams -The DomainTools Real Time Unified Feeds integration collects one type of data streams: logs +The DomainTools Feeds integration collects one type of data streams: **logs** -Log data streams collected by the DomainTools integration include the Newly Observed Domains (NOD) feed: Apex-level domains (e.g. Example Domain but not www.example.com) that we observe for the first time, and have not observed previously. -Populated with our global DNS sensor network. +Log data streams collected by the DomainTools integration include the following feeds: + +- `Newly Observed Domains (NOD)` +- `Newly Active Domains (NAD)` +- `Domain Discovery` +- `Domain RDAP` ## Requirements -You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. +You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware. -You will require a license to one or more DomainTools feeds, and API credentials. -Your required API credentials will vary with your authentication method, detailed below. +You will require a license to one or more DomainTools feeds, and API credentials. +Your required API credentials will vary with your authentication method, detailed below. -Obtain your API credentials from your group’s API administrator. +Obtain your API credentials from your group’s API administrator. API administrators can manage their API keys at research.domaintools.com, selecting the drop-down account menu and choosing API admin. ## Setup For step-by-step instructions on how to set up an integration, see the Getting started guide. -### Newly Observed Domains (NOD) Feed +### Newly Observed Domains (NOD) Feed The `nod_feed` data stream provides events from [DomainTools Newly Observed Domains Feed](https://www.domaintools.com/products/threat-intelligence-feeds/). -This data is collected via the [DomainTools Real Time Feeds API](https://docs.domaintools.com/feeds/realtime/). +This data is collected via the [DomainTools Feeds API](https://docs.domaintools.com/feeds/realtime/). #### Example @@ -63,10 +73,27 @@ An example event for `nod_feed` looks as following: "version": "8.17.0" }, "elastic_agent": { - "id": "df6cda61-c87d-40c3-92d1-6eb4f18f3a79", - "snapshot": false, - "version": "8.15.3" + "id": "da8d0a37-2d46-4788-96bd-e9ee19e332ec", + "version": "8.15.3", + "snapshot": false + }, + "domaintools": { + "domain": "tractorpoweredcoreaerator.com", + "feed": "nod", + "timestamp": "2025-01-30T20:14:48Z" + }, + "threat": { + "indicator": { + "name": "tractorpoweredcoreaerator.com", + "type": "domain-name" + }, + "feed": { + "reference": "https://docs.techdocs.ci.domaintools.cloud/feeds/realtime/userguide/", + "name": "DomainTools NOD", + "description": "Apex-level domains (e.g. example.com but not www.example.com) that we observe for the first time, and have not observed previously with our global DNS sensor network." + } }, + "message": "{\"timestamp\":\"2025-01-30T20:14:48Z\",\"domain\":\"tractorpoweredcoreaerator.com\"}", "event": { "agent_id_status": "verified", "category": [ @@ -130,10 +157,423 @@ An example event for `nod_feed` looks as following: | data_stream.namespace | A user defined namespace. Namespaces are useful to allow grouping of data. Many users already organize their indices this way, and the data stream naming scheme now provides this best practice as a default. Many users will populate this field with `default`. If no value is used, it falls back to `default`. Beyond the Elasticsearch index naming criteria noted above, `namespace` value has the additional restrictions: \* Must not contain `-` \* No longer than 100 characters | constant_keyword | | data_stream.type | An overarching type for the data stream. Currently allowed values are "logs" and "metrics". We expect to also add "traces" and "synthetics" in the near future. | constant_keyword | | domaintools.domain | The Domain. Apex-level domains (e.g. example.com but not www.example.com) that we observe for the first time, and have not observed previously with our global DNS sensor network. | keyword | -| domaintools.feed | The feed. | keyword | +| domaintools.feed | The feed. | constant_keyword | +| domaintools.timestamp | Timestamp when the domain was added to the DomainTools feed, in ISO 8601 UTC form. | date | +| ecs.version | ECS version this event conforms to. `ecs.version` is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. | keyword | +| error.message | Error message. | match_only_text | +| event.category | This is one of four ECS Categorization Fields, and indicates the second level in the ECS category hierarchy. `event.category` represents the "big buckets" of ECS categories. For example, filtering on `event.category:process` yields all events relating to process activity. This field is closely related to `event.type`, which is used as a subcategory. This field is an array. This will allow proper categorization of some events that fall in multiple categories. | keyword | +| event.id | Unique ID to describe the event. | keyword | +| event.ingested | Timestamp when an event arrived in the central data store. This is different from `@timestamp`, which is when the event originally occurred. It's also different from `event.created`, which is meant to capture the first time an agent saw the event. In normal conditions, assuming no tampering, the timestamps should chronologically look like this: `@timestamp` \< `event.created` \< `event.ingested`. | date | +| event.kind | This is one of four ECS Categorization Fields, and indicates the highest level in the ECS category hierarchy. `event.kind` gives high-level information about what type of information the event contains, without being specific to the contents of the event. For example, values of this field distinguish alert events from metric events. The value of this field can be used to inform how these kinds of events should be handled. They may warrant different retention, different access control, it may also help understand whether the data is coming in at a regular interval or not. | keyword | +| event.type | This is one of four ECS Categorization Fields, and indicates the third level in the ECS category hierarchy. `event.type` represents a categorization "sub-bucket" that, when used along with the `event.category` field values, enables filtering events down to a level appropriate for single visualization. This field is an array. This will allow proper categorization of some events that fall in multiple event types. | keyword | +| input.type | Type of filebeat input. | keyword | +| labels.is_ioc_transform_source | Indicates whether an IOC is in the raw source data stream, or the in latest destination index. | constant_keyword | +| message | The feed. | match_only_text | +| threat.feed.description | Description of the threat feed in a UI friendly format. | keyword | +| threat.feed.name | The name of the threat feed in UI friendly format. | keyword | +| threat.feed.reference | Reference information for the threat feed in a UI friendly format. | keyword | +| threat.indicator.name | The display name indicator in an UI friendly format URL, IP address, email address, registry key, port number, hash value, or other relevant name can serve as the display name. | keyword | +| threat.indicator.type | Type of indicator as represented by Cyber Observable in STIX 2.0. | keyword | + + +### Newly Active Domains (NAD) Feed + +The `nod_feed` data stream provides events from [DomainTools Newly Active Domains Feed](https://www.domaintools.com/products/threat-intelligence-feeds/). +This data is collected via the [DomainTools Feeds API](https://docs.domaintools.com/feeds/realtime/). + +#### Example + +An example event for `nad_feed` looks as following: + +```json +{ + "input": { + "type": "cel" + }, + "agent": { + "name": "docker-fleet-agent", + "id": "da8d0a37-2d46-4788-96bd-e9ee19e332ec", + "ephemeral_id": "d1cbe648-0a1d-48e8-a161-cd82403e623e", + "type": "filebeat", + "version": "8.15.3" + }, + "@timestamp": "2025-01-30T20:15:25.396Z", + "ecs": { + "version": "8.11.0" + }, + "data_stream": { + "namespace": "default", + "type": "logs", + "dataset": "ti_domaintools.nad_feed" + }, + "host": { + "hostname": "docker-fleet-agent", + "os": { + "kernel": "6.10.11-linuxkit", + "codename": "focal", + "name": "Ubuntu", + "type": "linux", + "family": "debian", + "version": "20.04.6 LTS (Focal Fossa)", + "platform": "ubuntu" + }, + "containerized": false, + "ip": [ + "172.19.0.10" + ], + "name": "docker-fleet-agent", + "id": "cfae1e7244ae479b9b0968259c91b13a", + "mac": [ + "02-42-AC-13-00-0A" + ], + "architecture": "aarch64" + }, + "elastic_agent": { + "id": "da8d0a37-2d46-4788-96bd-e9ee19e332ec", + "version": "8.15.3", + "snapshot": false + }, + "domaintools": { + "domain": "tractorpoweredcoreaerator.com", + "feed": "nad", + "timestamp": "2025-01-30T20:14:48Z" + }, + "threat": { + "indicator": { + "name": "tractorpoweredcoreaerator.com", + "type": "domain-name" + }, + "feed": { + "reference": "https://docs.techdocs.ci.domaintools.cloud/feeds/realtime/userguide/", + "name": "DomainTools NAD", + "description": "Apex-level domains (e.g. example.com but not www.example.com) that we observe for the first time, and have not observed previously with our global DNS sensor network." + } + }, + "message": "{\"timestamp\":\"2025-01-30T20:14:48Z\",\"domain\":\"tractorpoweredcoreaerator.com\"}", + "event": { + "agent_id_status": "verified", + "ingested": "2025-01-30T20:15:26Z", + "kind": "enrichment", + "category": [ + "threat" + ], + "type": [ + "indicator" + ], + "dataset": "ti_domaintools.nad_feed" + } +} +``` + +**Exported fields** + +| Field | Description | Type | +|---|---|---| +| @timestamp | Date/time when the event originated. This is the date/time extracted from the event, typically representing when the event was generated by the source. If the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline. Required field for all events. | date | +| data_stream.dataset | The field can contain anything that makes sense to signify the source of the data. Examples include `nginx.access`, `prometheus`, `endpoint` etc. For data streams that otherwise fit, but that do not have dataset set we use the value "generic" for the dataset value. `event.dataset` should have the same value as `data_stream.dataset`. Beyond the Elasticsearch data stream naming criteria noted above, the `dataset` value has additional restrictions: \* Must not contain `-` \* No longer than 100 characters | constant_keyword | +| data_stream.namespace | A user defined namespace. Namespaces are useful to allow grouping of data. Many users already organize their indices this way, and the data stream naming scheme now provides this best practice as a default. Many users will populate this field with `default`. If no value is used, it falls back to `default`. Beyond the Elasticsearch index naming criteria noted above, `namespace` value has the additional restrictions: \* Must not contain `-` \* No longer than 100 characters | constant_keyword | +| data_stream.type | An overarching type for the data stream. Currently allowed values are "logs" and "metrics". We expect to also add "traces" and "synthetics" in the near future. | constant_keyword | +| domaintools.domain | The Domain. Apex-level domains (e.g. example.com but not www.example.com) that we observe for the first time, and have not observed previously with our global DNS sensor network. | keyword | +| domaintools.feed | The feed. | constant_keyword | | domaintools.timestamp | Timestamp when the domain was added to the DomainTools feed, in ISO 8601 UTC form. | date | +| ecs.version | ECS version this event conforms to. `ecs.version` is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. | keyword | +| error.message | Error message. | match_only_text | +| event.category | This is one of four ECS Categorization Fields, and indicates the second level in the ECS category hierarchy. `event.category` represents the "big buckets" of ECS categories. For example, filtering on `event.category:process` yields all events relating to process activity. This field is closely related to `event.type`, which is used as a subcategory. This field is an array. This will allow proper categorization of some events that fall in multiple categories. | keyword | +| event.id | Unique ID to describe the event. | keyword | +| event.ingested | Timestamp when an event arrived in the central data store. This is different from `@timestamp`, which is when the event originally occurred. It's also different from `event.created`, which is meant to capture the first time an agent saw the event. In normal conditions, assuming no tampering, the timestamps should chronologically look like this: `@timestamp` \< `event.created` \< `event.ingested`. | date | +| event.kind | This is one of four ECS Categorization Fields, and indicates the highest level in the ECS category hierarchy. `event.kind` gives high-level information about what type of information the event contains, without being specific to the contents of the event. For example, values of this field distinguish alert events from metric events. The value of this field can be used to inform how these kinds of events should be handled. They may warrant different retention, different access control, it may also help understand whether the data is coming in at a regular interval or not. | keyword | +| event.type | This is one of four ECS Categorization Fields, and indicates the third level in the ECS category hierarchy. `event.type` represents a categorization "sub-bucket" that, when used along with the `event.category` field values, enables filtering events down to a level appropriate for single visualization. This field is an array. This will allow proper categorization of some events that fall in multiple event types. | keyword | | input.type | Type of filebeat input. | keyword | | labels.is_ioc_transform_source | Indicates whether an IOC is in the raw source data stream, or the in latest destination index. | constant_keyword | | message | The feed. | match_only_text | +| threat.feed.description | Description of the threat feed in a UI friendly format. | keyword | +| threat.feed.name | The name of the threat feed in UI friendly format. | keyword | +| threat.feed.reference | Reference information for the threat feed in a UI friendly format. | keyword | +| threat.indicator.name | The display name indicator in an UI friendly format URL, IP address, email address, registry key, port number, hash value, or other relevant name can serve as the display name. | keyword | +| threat.indicator.type | Type of indicator as represented by Cyber Observable in STIX 2.0. | keyword | + + +### Domain Discovery Feed + +The `domaindiscovery feed` data stream provides events from [DomainTools Domain Discovery Feed](https://www.domaintools.com/products/threat-intelligence-feeds/). +This data is collected via the [DomainTools Feeds API](https://docs.domaintools.com/feeds/realtime/). + +#### Example + +An example event for `domaindiscovery` looks as following: +```json +{ + "input": { + "type": "cel" + }, + "agent": { + "name": "docker-fleet-agent", + "id": "da8d0a37-2d46-4788-96bd-e9ee19e332ec", + "ephemeral_id": "d1cbe648-0a1d-48e8-a161-cd82403e623e", + "type": "filebeat", + "version": "8.15.3" + }, + "@timestamp": "2025-01-30T20:15:25.396Z", + "ecs": { + "version": "8.11.0" + }, + "data_stream": { + "namespace": "default", + "type": "logs", + "dataset": "ti_domaintools.domaindiscovery" + }, + "host": { + "hostname": "docker-fleet-agent", + "os": { + "kernel": "6.10.11-linuxkit", + "codename": "focal", + "name": "Ubuntu", + "type": "linux", + "family": "debian", + "version": "20.04.6 LTS (Focal Fossa)", + "platform": "ubuntu" + }, + "containerized": false, + "ip": [ + "172.19.0.10" + ], + "name": "docker-fleet-agent", + "id": "cfae1e7244ae479b9b0968259c91b13a", + "mac": [ + "02-42-AC-13-00-0A" + ], + "architecture": "aarch64" + }, + "elastic_agent": { + "id": "da8d0a37-2d46-4788-96bd-e9ee19e332ec", + "version": "8.15.3", + "snapshot": false + }, + "domaintools": { + "domain": "tractorpoweredcoreaerator.com", + "timestamp": "2025-01-30T20:14:48Z", + "feed": "domaindiscovery" + }, + "threat": { + "indicator": { + "name": "tractorpoweredcoreaerator.com", + "type": "domain-name" + }, + "feed": { + "reference": "https://docs.techdocs.ci.domaintools.cloud/feeds/realtime/userguide/", + "name": "DomainTools domaindiscovery", + "description": "New domains as they are either discovered in domain registration information, observed by our global sensor network, or reported by trusted third parties." + } + }, + "message": "{\"timestamp\":\"2025-01-30T20:14:48Z\",\"domain\":\"tractorpoweredcoreaerator.com\"}", + "event": { + "agent_id_status": "verified", + "ingested": "2025-01-30T20:15:26Z", + "kind": "enrichment", + "category": [ + "threat" + ], + "type": [ + "indicator" + ], + "dataset": "ti_domaintools.domaindiscovery" + } +} +``` + +**Exported fields** + +| Field | Description | Type | +|---|---|---| +| @timestamp | Date/time when the event originated. This is the date/time extracted from the event, typically representing when the event was generated by the source. If the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline. Required field for all events. | date | +| data_stream.dataset | The field can contain anything that makes sense to signify the source of the data. Examples include `nginx.access`, `prometheus`, `endpoint` etc. For data streams that otherwise fit, but that do not have dataset set we use the value "generic" for the dataset value. `event.dataset` should have the same value as `data_stream.dataset`. Beyond the Elasticsearch data stream naming criteria noted above, the `dataset` value has additional restrictions: \* Must not contain `-` \* No longer than 100 characters | constant_keyword | +| data_stream.namespace | A user defined namespace. Namespaces are useful to allow grouping of data. Many users already organize their indices this way, and the data stream naming scheme now provides this best practice as a default. Many users will populate this field with `default`. If no value is used, it falls back to `default`. Beyond the Elasticsearch index naming criteria noted above, `namespace` value has the additional restrictions: \* Must not contain `-` \* No longer than 100 characters | constant_keyword | +| data_stream.type | An overarching type for the data stream. Currently allowed values are "logs" and "metrics". We expect to also add "traces" and "synthetics" in the near future. | constant_keyword | +| domaintools.domain | The Domain. | keyword | +| domaintools.feed | The feed type. | constant_keyword | +| domaintools.timestamp | Timestamp when the domain was added to the DomainTools feed, in ISO 8601 UTC form. | date | +| ecs.version | ECS version this event conforms to. `ecs.version` is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. | keyword | +| error.message | Error message. | match_only_text | +| event.category | This is one of four ECS Categorization Fields, and indicates the second level in the ECS category hierarchy. `event.category` represents the "big buckets" of ECS categories. For example, filtering on `event.category:process` yields all events relating to process activity. This field is closely related to `event.type`, which is used as a subcategory. This field is an array. This will allow proper categorization of some events that fall in multiple categories. | keyword | +| event.id | Unique ID to describe the event. | keyword | +| event.ingested | Timestamp when an event arrived in the central data store. This is different from `@timestamp`, which is when the event originally occurred. It's also different from `event.created`, which is meant to capture the first time an agent saw the event. In normal conditions, assuming no tampering, the timestamps should chronologically look like this: `@timestamp` \< `event.created` \< `event.ingested`. | date | +| event.kind | This is one of four ECS Categorization Fields, and indicates the highest level in the ECS category hierarchy. `event.kind` gives high-level information about what type of information the event contains, without being specific to the contents of the event. For example, values of this field distinguish alert events from metric events. The value of this field can be used to inform how these kinds of events should be handled. They may warrant different retention, different access control, it may also help understand whether the data is coming in at a regular interval or not. | keyword | +| event.type | This is one of four ECS Categorization Fields, and indicates the third level in the ECS category hierarchy. `event.type` represents a categorization "sub-bucket" that, when used along with the `event.category` field values, enables filtering events down to a level appropriate for single visualization. This field is an array. This will allow proper categorization of some events that fall in multiple event types. | keyword | +| input.type | Type of filebeat input. | keyword | +| labels.is_ioc_transform_source | Indicates whether an IOC is in the raw source data stream, or the in latest destination index. | constant_keyword | +| message | The feed from DomainTools Feed API. | match_only_text | +| threat.feed.description | Description of the threat feed in a UI friendly format. | keyword | +| threat.feed.name | The name of the threat feed in UI friendly format. | keyword | +| threat.feed.reference | Reference information for the threat feed in a UI friendly format. | keyword | +| threat.indicator.name | The display name indicator in an UI friendly format URL, IP address, email address, registry key, port number, hash value, or other relevant name can serve as the display name. | keyword | +| threat.indicator.type | Type of indicator as represented by Cyber Observable in STIX 2.0. | keyword | + + +### Domain RDAP + +The `domainrdap feed` data stream provides events from [DomainTools Domain RDAP](https://www.domaintools.com/products/threat-intelligence-feeds/). +This data is collected via the [DomainTools Feeds API](https://docs.domaintools.com/feeds/realtime/). + +#### Example + +An example event for `domainrdap` looks as following: + +```json +{ + "input": { + "type": "cel" + }, + "agent": { + "name": "docker-fleet-agent", + "id": "da8d0a37-2d46-4788-96bd-e9ee19e332ec", + "ephemeral_id": "d1cbe648-0a1d-48e8-a161-cd82403e623e", + "type": "filebeat", + "version": "8.15.3" + }, + "@timestamp": "2025-01-30T20:15:25.396Z", + "ecs": { + "version": "8.11.0" + }, + "data_stream": { + "namespace": "default", + "type": "logs", + "dataset": "ti_domaintools.domainrdap" + }, + "host": { + "hostname": "docker-fleet-agent", + "os": { + "kernel": "6.10.11-linuxkit", + "codename": "focal", + "name": "Ubuntu", + "type": "linux", + "family": "debian", + "version": "20.04.6 LTS (Focal Fossa)", + "platform": "ubuntu" + }, + "containerized": false, + "ip": [ + "172.19.0.10" + ], + "name": "docker-fleet-agent", + "id": "cfae1e7244ae479b9b0968259c91b13a", + "mac": [ + "02-42-AC-13-00-0A" + ], + "architecture": "aarch64" + }, + "elastic_agent": { + "id": "da8d0a37-2d46-4788-96bd-e9ee19e332ec", + "version": "8.15.3", + "snapshot": false + }, + "domaintools": { + "domain": "unlockyourlifehere.com", + "timestamp": "2025-06-12T20:34:31Z", + "feed": "domainrdap", + "first_request_timestamp": "2025-06-12T20:34:24Z", + "requests_url": [ + "https://rdap.verisign.com/com/v1/domain/unlockyourlifehere.com" + ], + "parsed_record": { + "parsed_fields": { + "emails": [ + "abuse@godaddy.com" + ], + "last_changed_date": "2025-05-20T02: 44: 33+00: 00", + "registrar": { + "name": "GoDaddy.com, LLC", + "contacts": [ + { + "name": "", + "phone": "tel:480-624-2505", + "email": "abuse@godaddy.com", + "roles": [ + "abuse" + ] + } + ], + "iana_id": "146" + }, + "handle": "2894681047_DOMAIN_COM-VRSN", + "creation_date": "2024-06-28T11: 49: 19+00: 00", + "expiration_date": "2025-06-28T11: 49: 19+00: 00", + "email_domains": [ + "godaddy.com" + ], + "contacts": [] + } + } + }, + "threat": { + "indicator": { + "name": "unlockyourlifehere.com", + "type": "domain-name" + }, + "feed": { + "reference": "https://docs.techdocs.ci.domaintools.cloud/feeds/realtime/userguide/", + "name": "DomainTools domain RDAP", + "description": "Changes to global domain registration information, populated by the Registration Data Access Protocol (RDAP). Compliments the 5-Minute WHOIS Feed as registries and registrars switch from Whois to RDAP." + } + }, + "message": "{\"timestamp\":\"2025-06-12T20:34:31Z\",\"domain\":\"unlockyourlifehere.com\",\"raw_record\":{\"first_request_timestamp\":\"2025-06-12T20:34:24Z\",\"requests\":[{\"data\":\"{\\\"objectClassName\\\":\\\"domain\\\",\\\"handle\\\":\\\"2894681047_DOMAIN_COM-VRSN\\\",\\\"ldhName\\\":\\\"UNLOCKYOURLIFEHERE.COM\\\",\\\"links\\\":[{\\\"value\\\":\\\"https:\\\\/\\\\/rdap.verisign.com\\\\/com\\\\/v1\\\\/domain\\\\/UNLOCKYOURLIFEHERE.COM\\\",\\\"rel\\\":\\\"self\\\",\\\"href\\\":\\\"https:\\\\/\\\\/rdap.verisign.com\\\\/com\\\\/v1\\\\/domain\\\\/UNLOCKYOURLIFEHERE.COM\\\",\\\"type\\\":\\\"application\\\\/rdap+json\\\"},{\\\"value\\\":\\\"https:\\\\/\\\\/rdap.godaddy.com\\\\/v1\\\\/domain\\\\/UNLOCKYOURLIFEHERE.COM\\\",\\\"rel\\\":\\\"related\\\",\\\"href\\\":\\\"https:\\\\/\\\\/rdap.godaddy.com\\\\/v1\\\\/domain\\\\/UNLOCKYOURLIFEHERE.COM\\\",\\\"type\\\":\\\"application\\\\/rdap+json\\\"}],\\\"status\\\":[\\\"redemption period\\\"],\\\"entities\\\":[{\\\"objectClassName\\\":\\\"entity\\\",\\\"handle\\\":\\\"146\\\",\\\"roles\\\":[\\\"registrar\\\"],\\\"publicIds\\\":[{\\\"type\\\":\\\"IANA Registrar ID\\\",\\\"identifier\\\":\\\"146\\\"}],\\\"vcardArray\\\":[\\\"vcard\\\",[[\\\"version\\\",{},\\\"text\\\",\\\"4.0\\\"],[\\\"fn\\\",{},\\\"text\\\",\\\"GoDaddy.com, LLC\\\"]]],\\\"entities\\\":[{\\\"objectClassName\\\":\\\"entity\\\",\\\"roles\\\":[\\\"abuse\\\"],\\\"vcardArray\\\":[\\\"vcard\\\",[[\\\"version\\\",{},\\\"text\\\",\\\"4.0\\\"],[\\\"fn\\\",{},\\\"text\\\",\\\"\\\"],[\\\"tel\\\",{\\\"type\\\":\\\"voice\\\"},\\\"uri\\\",\\\"tel:480-624-2505\\\"],[\\\"email\\\",{},\\\"text\\\",\\\"abuse@godaddy.com\\\"]]]}]}],\\\"events\\\":[{\\\"eventAction\\\":\\\"registration\\\",\\\"eventDate\\\":\\\"2024-06-28T11:49:19Z\\\"},{\\\"eventAction\\\":\\\"expiration\\\",\\\"eventDate\\\":\\\"2025-06-28T11:49:19Z\\\"},{\\\"eventAction\\\":\\\"last changed\\\",\\\"eventDate\\\":\\\"2025-05-20T02:44:33Z\\\"},{\\\"eventAction\\\":\\\"last update of RDAP database\\\",\\\"eventDate\\\":\\\"2025-06-12T20:34:16Z\\\"}],\\\"secureDNS\\\":{\\\"delegationSigned\\\":false},\\\"rdapConformance\\\":[\\\"rdap_level_0\\\",\\\"icann_rdap_technical_implementation_guide_0\\\",\\\"icann_rdap_response_profile_0\\\"],\\\"notices\\\":[{\\\"title\\\":\\\"Terms of Use\\\",\\\"description\\\":[\\\"Service subject to Terms of Use.\\\"],\\\"links\\\":[{\\\"href\\\":\\\"https:\\\\/\\\\/www.verisign.com\\\\/domain-names\\\\/registration-data-access-protocol\\\\/terms-service\\\\/index.xhtml\\\",\\\"type\\\":\\\"text\\\\/html\\\"}]},{\\\"title\\\":\\\"Status Codes\\\",\\\"description\\\":[\\\"For more information on domain status codes, please visit https:\\\\/\\\\/icann.org\\\\/epp\\\"],\\\"links\\\":[{\\\"href\\\":\\\"https:\\\\/\\\\/icann.org\\\\/epp\\\",\\\"type\\\":\\\"text\\\\/html\\\"}]},{\\\"title\\\":\\\"RDDS Inaccuracy Complaint Form\\\",\\\"description\\\":[\\\"URL of the ICANN RDDS Inaccuracy Complaint Form: https:\\\\/\\\\/icann.org\\\\/wicf\\\"],\\\"links\\\":[{\\\"href\\\":\\\"https:\\\\/\\\\/icann.org\\\\/wicf\\\",\\\"type\\\":\\\"text\\\\/html\\\"}]}]}\",\"source_type\":\"registry\",\"timestamp\":\"2025-06-12T20:34:24Z\",\"url\":\"https://rdap.verisign.com/com/v1/domain/unlockyourlifehere.com\"}]},\"parsed_record\":{\"parsed_fields\":{\"conformance\":[\"rdap_level_0\",\"icann_rdap_technical_implementation_guide_0\",\"icann_rdap_response_profile_0\"],\"contacts\":[],\"creation_date\":\"2024-06-28T11: 49: 19+00: 00\",\"dnssec\":{\"signed\":false},\"domain\":\"UNLOCKYOURLIFEHERE.COM\",\"domain_statuses\":[\"redemption period\"],\"email_domains\":[\"godaddy.com\"],\"emails\":[\"abuse@godaddy.com\"],\"expiration_date\":\"2025-06-28T11: 49: 19+00: 00\",\"handle\":\"2894681047_DOMAIN_COM-VRSN\",\"last_changed_date\":\"2025-05-20T02: 44: 33+00: 00\",\"links\":[{\"href\":\"https://rdap.verisign.com/com/v1/domain/UNLOCKYOURLIFEHERE.COM\",\"rel\":\"self\"},{\"href\":\"https://rdap.godaddy.com/v1/domain/UNLOCKYOURLIFEHERE.COM\",\"rel\":\"related\"}],\"registrar\":{\"contacts\":[{\"email\":\"abuse@godaddy.com\",\"name\":\"\",\"phone\":\"tel:480-624-2505\",\"roles\":[\"abuse\"]}],\"iana_id\":\"146\",\"name\":\"GoDaddy.com, LLC\"},\"unclassified_emails\":[]},\"registrar_request_url\":null,\"registry_request_url\":\"https://rdap.verisign.com/com/v1/domain/unlockyourlifehere.com\"}}", + "event": { + "agent_id_status": "verified", + "ingested": "2025-06-12T20:34:31Z", + "kind": "enrichment", + "category": [ + "threat" + ], + "type": [ + "indicator" + ], + "dataset": "ti_domaintools.domainrdap" + } +} +``` + +**Exported fields** + +| Field | Description | Type | +|---|---|---| +| @timestamp | Date/time when the event originated. This is the date/time extracted from the event, typically representing when the event was generated by the source. If the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline. Required field for all events. | date | +| data_stream.dataset | The field can contain anything that makes sense to signify the source of the data. Examples include `nginx.access`, `prometheus`, `endpoint` etc. For data streams that otherwise fit, but that do not have dataset set we use the value "generic" for the dataset value. `event.dataset` should have the same value as `data_stream.dataset`. Beyond the Elasticsearch data stream naming criteria noted above, the `dataset` value has additional restrictions: \* Must not contain `-` \* No longer than 100 characters | constant_keyword | +| data_stream.namespace | A user defined namespace. Namespaces are useful to allow grouping of data. Many users already organize their indices this way, and the data stream naming scheme now provides this best practice as a default. Many users will populate this field with `default`. If no value is used, it falls back to `default`. Beyond the Elasticsearch index naming criteria noted above, `namespace` value has the additional restrictions: \* Must not contain `-` \* No longer than 100 characters | constant_keyword | +| data_stream.type | An overarching type for the data stream. Currently allowed values are "logs" and "metrics". We expect to also add "traces" and "synthetics" in the near future. | constant_keyword | +| domaintools.domain | The Domain. | keyword | +| domaintools.feed | The feed type. | constant_keyword | +| domaintools.first_request_timestamp | The first request timestamp. | date | +| domaintools.parsed_record.parsed_fields.contacts.country | | keyword | +| domaintools.parsed_record.parsed_fields.contacts.email | | keyword | +| domaintools.parsed_record.parsed_fields.contacts.name | | keyword | +| domaintools.parsed_record.parsed_fields.creation_date | The domain creation date. | keyword | +| domaintools.parsed_record.parsed_fields.email_domains | List of email domains. | keyword | +| domaintools.parsed_record.parsed_fields.emails | List of emails. | keyword | +| domaintools.parsed_record.parsed_fields.expiration_date | The domain expiraton date. | keyword | +| domaintools.parsed_record.parsed_fields.handle | The domain handle. | keyword | +| domaintools.parsed_record.parsed_fields.last_changed_date | The domain last changed date. | keyword | +| domaintools.parsed_record.parsed_fields.nameservers | The domain nameservers. | keyword | +| domaintools.parsed_record.parsed_fields.registrar.contacts.email | | keyword | +| domaintools.parsed_record.parsed_fields.registrar.contacts.name | | keyword | +| domaintools.parsed_record.parsed_fields.registrar.contacts.phone | | keyword | +| domaintools.parsed_record.parsed_fields.registrar.contacts.roles | | keyword | +| domaintools.parsed_record.parsed_fields.registrar.iana_id | | keyword | +| domaintools.parsed_record.parsed_fields.registrar.name | The registrar name. | keyword | +| domaintools.requests_url | List of extracted rdap request urls used. | keyword | +| domaintools.timestamp | Timestamp when the domain was added to the DomainTools feed, in ISO 8601 UTC form. | date | +| ecs.version | ECS version this event conforms to. `ecs.version` is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. | keyword | +| error.message | Error message. | match_only_text | +| event.category | This is one of four ECS Categorization Fields, and indicates the second level in the ECS category hierarchy. `event.category` represents the "big buckets" of ECS categories. For example, filtering on `event.category:process` yields all events relating to process activity. This field is closely related to `event.type`, which is used as a subcategory. This field is an array. This will allow proper categorization of some events that fall in multiple categories. | keyword | +| event.id | Unique ID to describe the event. | keyword | +| event.ingested | Timestamp when an event arrived in the central data store. This is different from `@timestamp`, which is when the event originally occurred. It's also different from `event.created`, which is meant to capture the first time an agent saw the event. In normal conditions, assuming no tampering, the timestamps should chronologically look like this: `@timestamp` \< `event.created` \< `event.ingested`. | date | +| event.kind | This is one of four ECS Categorization Fields, and indicates the highest level in the ECS category hierarchy. `event.kind` gives high-level information about what type of information the event contains, without being specific to the contents of the event. For example, values of this field distinguish alert events from metric events. The value of this field can be used to inform how these kinds of events should be handled. They may warrant different retention, different access control, it may also help understand whether the data is coming in at a regular interval or not. | keyword | +| event.type | This is one of four ECS Categorization Fields, and indicates the third level in the ECS category hierarchy. `event.type` represents a categorization "sub-bucket" that, when used along with the `event.category` field values, enables filtering events down to a level appropriate for single visualization. This field is an array. This will allow proper categorization of some events that fall in multiple event types. | keyword | +| input.type | Type of filebeat input. | keyword | +| labels.is_ioc_transform_source | Indicates whether an IOC is in the raw source data stream, or the in latest destination index. | constant_keyword | +| message | The feed from DomainTools Feed API. | match_only_text | +| threat.feed.description | Description of the threat feed in a UI friendly format. | keyword | +| threat.feed.name | The name of the threat feed in UI friendly format. | keyword | +| threat.feed.reference | Reference information for the threat feed in a UI friendly format. | keyword | +| threat.indicator.name | The display name indicator in an UI friendly format URL, IP address, email address, registry key, port number, hash value, or other relevant name can serve as the display name. | keyword | +| threat.indicator.type | Type of indicator as represented by Cyber Observable in STIX 2.0. | keyword | diff --git a/packages/ti_domaintools/elasticsearch/transform/latest_domaindiscovery/fields/base-fields.yml b/packages/ti_domaintools/elasticsearch/transform/latest_domaindiscovery/fields/base-fields.yml new file mode 100644 index 00000000000..be7bccb662c --- /dev/null +++ b/packages/ti_domaintools/elasticsearch/transform/latest_domaindiscovery/fields/base-fields.yml @@ -0,0 +1,16 @@ +- name: data_stream.type + external: ecs +- name: data_stream.dataset + external: ecs +- name: data_stream.namespace + external: ecs +- name: event.module + type: constant_keyword + external: ecs + value: ti_domaintools +- name: event.dataset + type: constant_keyword + external: ecs + value: ti_domaintools.domaindiscovery +- name: "@timestamp" + external: ecs diff --git a/packages/ti_domaintools/elasticsearch/transform/latest_domaindiscovery/fields/beats.yml b/packages/ti_domaintools/elasticsearch/transform/latest_domaindiscovery/fields/beats.yml new file mode 100644 index 00000000000..b13d5cc96f4 --- /dev/null +++ b/packages/ti_domaintools/elasticsearch/transform/latest_domaindiscovery/fields/beats.yml @@ -0,0 +1,11 @@ +- name: input.type + type: keyword + description: Type of Filebeat input. +- name: log.flags + type: keyword + description: Flags for the log file. +- name: log.offset + type: long + description: Offset of the entry in the log file. +- name: log.file.path + external: ecs diff --git a/packages/ti_domaintools/elasticsearch/transform/latest_domaindiscovery/fields/ecs.yml b/packages/ti_domaintools/elasticsearch/transform/latest_domaindiscovery/fields/ecs.yml new file mode 100644 index 00000000000..664ff10ce6a --- /dev/null +++ b/packages/ti_domaintools/elasticsearch/transform/latest_domaindiscovery/fields/ecs.yml @@ -0,0 +1,24 @@ +- name: ecs.version + external: ecs +- name: error.message + external: ecs +- name: event.category + external: ecs +- name: event.id + external: ecs +- name: event.ingested + external: ecs +- name: event.kind + external: ecs +- name: event.type + external: ecs +- name: threat.indicator.type + external: ecs +- name: threat.indicator.name + external: ecs +- name: threat.feed.description + external: ecs +- name: threat.feed.name + external: ecs +- name: threat.feed.reference + external: ecs diff --git a/packages/ti_domaintools/elasticsearch/transform/latest_domaindiscovery/fields/fields.yml b/packages/ti_domaintools/elasticsearch/transform/latest_domaindiscovery/fields/fields.yml new file mode 100644 index 00000000000..934284a92b7 --- /dev/null +++ b/packages/ti_domaintools/elasticsearch/transform/latest_domaindiscovery/fields/fields.yml @@ -0,0 +1,23 @@ +- name: domaintools + type: group + fields: + - name: domain + type: keyword + description: > + The Domain. + + - name: feed + type: constant_keyword + value: "domaindiscovery_feed" + description: > + The feed. + + - name: timestamp + type: date + description: > + Timestamp when the domain was added to the DomainTools feed, in ISO 8601 UTC form. + +- name: message + external: ecs + description: >- + The feed. diff --git a/packages/ti_domaintools/elasticsearch/transform/latest_domaindiscovery/fields/is-ioc-transform-source-false.yml b/packages/ti_domaintools/elasticsearch/transform/latest_domaindiscovery/fields/is-ioc-transform-source-false.yml new file mode 100644 index 00000000000..15524f7a6cc --- /dev/null +++ b/packages/ti_domaintools/elasticsearch/transform/latest_domaindiscovery/fields/is-ioc-transform-source-false.yml @@ -0,0 +1,4 @@ +- name: labels.is_ioc_transform_source + type: constant_keyword + value: "false" + description: Indicates whether an IOC is in the raw source data stream, or the in latest destination index. diff --git a/packages/ti_domaintools/elasticsearch/transform/latest_domaindiscovery/manifest.yml b/packages/ti_domaintools/elasticsearch/transform/latest_domaindiscovery/manifest.yml new file mode 100644 index 00000000000..f5296fd0c0a --- /dev/null +++ b/packages/ti_domaintools/elasticsearch/transform/latest_domaindiscovery/manifest.yml @@ -0,0 +1,18 @@ +start: true +destination_index_template: + settings: + index: + sort: + field: + - "@timestamp" + order: + - desc + mappings: + dynamic: true + dynamic_templates: + - strings_as_keyword: + match_mapping_type: string + mapping: + ignore_above: 1024 + type: keyword + date_detection: false diff --git a/packages/ti_domaintools/elasticsearch/transform/latest_domaindiscovery/transform.yml b/packages/ti_domaintools/elasticsearch/transform/latest_domaindiscovery/transform.yml new file mode 100644 index 00000000000..d764fa49dd4 --- /dev/null +++ b/packages/ti_domaintools/elasticsearch/transform/latest_domaindiscovery/transform.yml @@ -0,0 +1,35 @@ +# Use of "*" to use all namespaces defined. +source: + index: + - "logs-ti_domaintools.domaindiscovery-*" +# The version suffix on the dest.index should be incremented if a breaking change +# is made to the index mapping. You must also bump the fleet_transform_version +# for any change to this transform configuration to take effect. The old destination +# index is not automatically removed. We are dependent on https://github.com/elastic/package-spec/issues/523 to give +# us that ability in order to prevent having duplicate IoC data and prevent query +# time field type conflicts. +dest: + index: "logs-ti_domaintools_latest.domaindiscovery-2" + aliases: + - alias: "logs-ti_domaintools_latest.domaindiscovery" + move_on_creation: true +latest: + unique_key: + - domaintools.domain + sort: event.ingested +description: Latest Newly Domain discovery data +frequency: 30s +sync: + time: + field: event.ingested + # Updated to 120s because of refresh delay in Serverless. With default 60s, sometimes transform wouldn't process all documents. + delay: 120s +retention_policy: + time: + field: event.ingested + max_age: 7d +_meta: + managed: true + # Bump this version to delete, reinstall, and restart the transform during package. + # Version bump is needed if there is any code change in transform. + fleet_transform_version: 0.2.0 diff --git a/packages/ti_domaintools/elasticsearch/transform/latest_domainrdap/fields/base-fields.yml b/packages/ti_domaintools/elasticsearch/transform/latest_domainrdap/fields/base-fields.yml new file mode 100644 index 00000000000..4d382b5eac4 --- /dev/null +++ b/packages/ti_domaintools/elasticsearch/transform/latest_domainrdap/fields/base-fields.yml @@ -0,0 +1,16 @@ +- name: data_stream.type + external: ecs +- name: data_stream.dataset + external: ecs +- name: data_stream.namespace + external: ecs +- name: event.module + type: constant_keyword + external: ecs + value: ti_domaintools +- name: event.dataset + type: constant_keyword + external: ecs + value: ti_domaintools.domainrdap +- name: "@timestamp" + external: ecs diff --git a/packages/ti_domaintools/elasticsearch/transform/latest_domainrdap/fields/beats.yml b/packages/ti_domaintools/elasticsearch/transform/latest_domainrdap/fields/beats.yml new file mode 100644 index 00000000000..b13d5cc96f4 --- /dev/null +++ b/packages/ti_domaintools/elasticsearch/transform/latest_domainrdap/fields/beats.yml @@ -0,0 +1,11 @@ +- name: input.type + type: keyword + description: Type of Filebeat input. +- name: log.flags + type: keyword + description: Flags for the log file. +- name: log.offset + type: long + description: Offset of the entry in the log file. +- name: log.file.path + external: ecs diff --git a/packages/ti_domaintools/elasticsearch/transform/latest_domainrdap/fields/ecs.yml b/packages/ti_domaintools/elasticsearch/transform/latest_domainrdap/fields/ecs.yml new file mode 100644 index 00000000000..664ff10ce6a --- /dev/null +++ b/packages/ti_domaintools/elasticsearch/transform/latest_domainrdap/fields/ecs.yml @@ -0,0 +1,24 @@ +- name: ecs.version + external: ecs +- name: error.message + external: ecs +- name: event.category + external: ecs +- name: event.id + external: ecs +- name: event.ingested + external: ecs +- name: event.kind + external: ecs +- name: event.type + external: ecs +- name: threat.indicator.type + external: ecs +- name: threat.indicator.name + external: ecs +- name: threat.feed.description + external: ecs +- name: threat.feed.name + external: ecs +- name: threat.feed.reference + external: ecs diff --git a/packages/ti_domaintools/elasticsearch/transform/latest_domainrdap/fields/fields.yml b/packages/ti_domaintools/elasticsearch/transform/latest_domainrdap/fields/fields.yml new file mode 100644 index 00000000000..a0479a4c0e1 --- /dev/null +++ b/packages/ti_domaintools/elasticsearch/transform/latest_domainrdap/fields/fields.yml @@ -0,0 +1,101 @@ +- name: domaintools + type: group + fields: + - name: domain + type: keyword + description: > + The Domain. + + - name: feed + type: constant_keyword + value: "domainrdap" + description: > + The feed. + + - name: timestamp + type: date + description: > + Timestamp when the domain was added to the DomainTools feed, in ISO 8601 UTC form. + + - name: first_request_timestamp + type: date + description: > + The first request timestamp. + + - name: requests_url + type: keyword + description: > + List of extracted rdap request urls used. + + - name: parsed_record + type: group + fields: + - name: parsed_fields + type: group + fields: + - name: contact + type: group + description: > + List of contact objects extracted from RDAP response. + fields: + - name: name + type: keyword + - name: email + type: keyword + - name: country + type: keyword + - name: email_domains + type: keyword + description: > + List of email domains. + - name: emails + type: keyword + description: > + List of emails. + - name: creation_date + type: keyword + description: > + The domain creation date. + - name: expiration_date + type: keyword + description: > + The domain expiraton date. + - name: last_changed_date + type: keyword + description: > + The domain last changed date. + - name: handle + type: keyword + description: > + The domain handle. + - name: nameservers + type: keyword + description: > + The domain nameservers. + - name: registrar + type: group + description: > + The registrar contact from RDAP response. + fields: + - name: contact + type: group + description: > + List of registrar contact objects extracted from RDAP response. + fields: + - name: name + type: keyword + - name: email + type: keyword + - name: phone + type: keyword + - name: iana_id + type: keyword + - name: name + type: keyword + description: > + The registrar name. + +- name: message + external: ecs + description: >- + The feed. diff --git a/packages/ti_domaintools/elasticsearch/transform/latest_domainrdap/fields/is-ioc-transform-source-false.yml b/packages/ti_domaintools/elasticsearch/transform/latest_domainrdap/fields/is-ioc-transform-source-false.yml new file mode 100644 index 00000000000..15524f7a6cc --- /dev/null +++ b/packages/ti_domaintools/elasticsearch/transform/latest_domainrdap/fields/is-ioc-transform-source-false.yml @@ -0,0 +1,4 @@ +- name: labels.is_ioc_transform_source + type: constant_keyword + value: "false" + description: Indicates whether an IOC is in the raw source data stream, or the in latest destination index. diff --git a/packages/ti_domaintools/elasticsearch/transform/latest_domainrdap/manifest.yml b/packages/ti_domaintools/elasticsearch/transform/latest_domainrdap/manifest.yml new file mode 100644 index 00000000000..13f6dec7adc --- /dev/null +++ b/packages/ti_domaintools/elasticsearch/transform/latest_domainrdap/manifest.yml @@ -0,0 +1,18 @@ +start: true +destination_index_template: + settings: + index: + sort: + field: + - "@timestamp" + order: + - desc + mappings: + dynamic: true + dynamic_templates: + - strings_as_keyword: + match_mapping_type: string + mapping: + ignore_above: 2048 + type: keyword + date_detection: false diff --git a/packages/ti_domaintools/elasticsearch/transform/latest_domainrdap/transform.yml b/packages/ti_domaintools/elasticsearch/transform/latest_domainrdap/transform.yml new file mode 100644 index 00000000000..57d0a505123 --- /dev/null +++ b/packages/ti_domaintools/elasticsearch/transform/latest_domainrdap/transform.yml @@ -0,0 +1,35 @@ +# Use of "*" to use all namespaces defined. +source: + index: + - "logs-ti_domaintools.domainrdap-*" +# The version suffix on the dest.index should be incremented if a breaking change +# is made to the index mapping. You must also bump the fleet_transform_version +# for any change to this transform configuration to take effect. The old destination +# index is not automatically removed. We are dependent on https://github.com/elastic/package-spec/issues/523 to give +# us that ability in order to prevent having duplicate IoC data and prevent query +# time field type conflicts. +dest: + index: "logs-ti_domaintools_latest.domainrdap-2" + aliases: + - alias: "logs-ti_domaintools_latest.domainrdap" + move_on_creation: true +latest: + unique_key: + - domaintools.domain + sort: event.ingested +description: Latest Newly Domain RDAP +frequency: 30s +sync: + time: + field: event.ingested + # Updated to 120s because of refresh delay in Serverless. With default 60s, sometimes transform wouldn't process all documents. + delay: 120s +retention_policy: + time: + field: event.ingested + max_age: 7d +_meta: + managed: true + # Bump this version to delete, reinstall, and restart the transform during package. + # Version bump is needed if there is any code change in transform. + fleet_transform_version: 0.2.0 diff --git a/packages/ti_domaintools/elasticsearch/transform/latest_nad/fields/base-fields.yml b/packages/ti_domaintools/elasticsearch/transform/latest_nad/fields/base-fields.yml new file mode 100644 index 00000000000..a2bcd004393 --- /dev/null +++ b/packages/ti_domaintools/elasticsearch/transform/latest_nad/fields/base-fields.yml @@ -0,0 +1,16 @@ +- name: data_stream.type + external: ecs +- name: data_stream.dataset + external: ecs +- name: data_stream.namespace + external: ecs +- name: event.module + type: constant_keyword + external: ecs + value: ti_domaintools +- name: event.dataset + type: constant_keyword + external: ecs + value: ti_domaintools.nad_feed +- name: "@timestamp" + external: ecs diff --git a/packages/ti_domaintools/elasticsearch/transform/latest_nad/fields/beats.yml b/packages/ti_domaintools/elasticsearch/transform/latest_nad/fields/beats.yml new file mode 100644 index 00000000000..b13d5cc96f4 --- /dev/null +++ b/packages/ti_domaintools/elasticsearch/transform/latest_nad/fields/beats.yml @@ -0,0 +1,11 @@ +- name: input.type + type: keyword + description: Type of Filebeat input. +- name: log.flags + type: keyword + description: Flags for the log file. +- name: log.offset + type: long + description: Offset of the entry in the log file. +- name: log.file.path + external: ecs diff --git a/packages/ti_domaintools/elasticsearch/transform/latest_nad/fields/ecs.yml b/packages/ti_domaintools/elasticsearch/transform/latest_nad/fields/ecs.yml new file mode 100644 index 00000000000..664ff10ce6a --- /dev/null +++ b/packages/ti_domaintools/elasticsearch/transform/latest_nad/fields/ecs.yml @@ -0,0 +1,24 @@ +- name: ecs.version + external: ecs +- name: error.message + external: ecs +- name: event.category + external: ecs +- name: event.id + external: ecs +- name: event.ingested + external: ecs +- name: event.kind + external: ecs +- name: event.type + external: ecs +- name: threat.indicator.type + external: ecs +- name: threat.indicator.name + external: ecs +- name: threat.feed.description + external: ecs +- name: threat.feed.name + external: ecs +- name: threat.feed.reference + external: ecs diff --git a/packages/ti_domaintools/elasticsearch/transform/latest_nad/fields/fields.yml b/packages/ti_domaintools/elasticsearch/transform/latest_nad/fields/fields.yml new file mode 100644 index 00000000000..b4ba350838a --- /dev/null +++ b/packages/ti_domaintools/elasticsearch/transform/latest_nad/fields/fields.yml @@ -0,0 +1,23 @@ +- name: domaintools + type: group + fields: + - name: domain + type: keyword + description: > + The Domain. Apex-level domains (e.g. example.com but not www.example.com) that we observe for the first time, and have not observed previously with our global DNS sensor network. + + - name: feed + type: constant_keyword + value: "nad" + description: > + The feed. + + - name: timestamp + type: date + description: > + Timestamp when the domain was added to the DomainTools feed, in ISO 8601 UTC form. + +- name: message + external: ecs + description: >- + The feed. diff --git a/packages/ti_domaintools/elasticsearch/transform/latest_nad/fields/is-ioc-transform-source-false.yml b/packages/ti_domaintools/elasticsearch/transform/latest_nad/fields/is-ioc-transform-source-false.yml new file mode 100644 index 00000000000..15524f7a6cc --- /dev/null +++ b/packages/ti_domaintools/elasticsearch/transform/latest_nad/fields/is-ioc-transform-source-false.yml @@ -0,0 +1,4 @@ +- name: labels.is_ioc_transform_source + type: constant_keyword + value: "false" + description: Indicates whether an IOC is in the raw source data stream, or the in latest destination index. diff --git a/packages/ti_domaintools/elasticsearch/transform/latest_nad/manifest.yml b/packages/ti_domaintools/elasticsearch/transform/latest_nad/manifest.yml new file mode 100644 index 00000000000..f5296fd0c0a --- /dev/null +++ b/packages/ti_domaintools/elasticsearch/transform/latest_nad/manifest.yml @@ -0,0 +1,18 @@ +start: true +destination_index_template: + settings: + index: + sort: + field: + - "@timestamp" + order: + - desc + mappings: + dynamic: true + dynamic_templates: + - strings_as_keyword: + match_mapping_type: string + mapping: + ignore_above: 1024 + type: keyword + date_detection: false diff --git a/packages/ti_domaintools/elasticsearch/transform/latest_nad/transform.yml b/packages/ti_domaintools/elasticsearch/transform/latest_nad/transform.yml new file mode 100644 index 00000000000..e8b8d382bac --- /dev/null +++ b/packages/ti_domaintools/elasticsearch/transform/latest_nad/transform.yml @@ -0,0 +1,35 @@ +# Use of "*" to use all namespaces defined. +source: + index: + - "logs-ti_domaintools.nad_feed-*" +# The version suffix on the dest.index should be incremented if a breaking change +# is made to the index mapping. You must also bump the fleet_transform_version +# for any change to this transform configuration to take effect. The old destination +# index is not automatically removed. We are dependent on https://github.com/elastic/package-spec/issues/523 to give +# us that ability in order to prevent having duplicate IoC data and prevent query +# time field type conflicts. +dest: + index: "logs-ti_domaintools_latest.nad_feed-2" + aliases: + - alias: "logs-ti_domaintools_latest.nad_feed" + move_on_creation: true +latest: + unique_key: + - domaintools.domain + sort: event.ingested +description: Latest Newly Active Domains data +frequency: 30s +sync: + time: + field: event.ingested + # Updated to 120s because of refresh delay in Serverless. With default 60s, sometimes transform wouldn't process all documents. + delay: 120s +retention_policy: + time: + field: event.ingested + max_age: 7d +_meta: + managed: true + # Bump this version to delete, reinstall, and restart the transform during package. + # Version bump is needed if there is any code change in transform. + fleet_transform_version: 0.2.0 diff --git a/packages/ti_domaintools/elasticsearch/transform/latest_nod/fields/fields.yml b/packages/ti_domaintools/elasticsearch/transform/latest_nod/fields/fields.yml index 515775b2f2c..bd8678310bb 100644 --- a/packages/ti_domaintools/elasticsearch/transform/latest_nod/fields/fields.yml +++ b/packages/ti_domaintools/elasticsearch/transform/latest_nod/fields/fields.yml @@ -1,6 +1,5 @@ - name: domaintools type: group - fields: - name: domain type: keyword @@ -8,7 +7,8 @@ The Domain. Apex-level domains (e.g. example.com but not www.example.com) that we observe for the first time, and have not observed previously with our global DNS sensor network. - name: feed - type: keyword + type: constant_keyword + value: "nod" description: > The feed. @@ -19,5 +19,5 @@ - name: message external: ecs - description: > - The feed. \ No newline at end of file + description: >- + The feed. diff --git a/packages/ti_domaintools/img/ti_domaintools_overview-dashboard.png b/packages/ti_domaintools/img/ti_domaintools_overview-dashboard.png index c08593a2f51..51dcc28172b 100644 Binary files a/packages/ti_domaintools/img/ti_domaintools_overview-dashboard.png and b/packages/ti_domaintools/img/ti_domaintools_overview-dashboard.png differ diff --git a/packages/ti_domaintools/kibana/dashboard/ti_domaintools-7ad9a714-58db-45e3-ba84-1e2dff1eb9a5.json b/packages/ti_domaintools/kibana/dashboard/ti_domaintools-7ad9a714-58db-45e3-ba84-1e2dff1eb9a5.json new file mode 100644 index 00000000000..1e9864357d8 --- /dev/null +++ b/packages/ti_domaintools/kibana/dashboard/ti_domaintools-7ad9a714-58db-45e3-ba84-1e2dff1eb9a5.json @@ -0,0 +1,1347 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", + "key": "domaintools.domain", + "negate": false, + "type": "exists" + }, + "query": { + "exists": { + "field": "domaintools.domain" + } + } + } + ], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "optionsJSON": { + "hidePanelTitles": false, + "syncColors": true, + "syncCursor": true, + "syncTooltips": true, + "useMargins": true + }, + "panelsJSON": [ + { + "embeddableConfig": { + "enhancements": {}, + "hidePanelTitles": false, + "savedVis": { + "data": { + "aggs": [], + "searchSource": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "description": "", + "id": "", + "params": { + "fontSize": 11, + "markdown": "**Overview** \nThis dashboard provides an overview of data collected through the DomainTools Feeds integration. It displays the total number of unique count of domains in different feeds to give a snapshot of current activity, visualizes domain observations over time to highlight trends or spikes, and includes a view of recently domaintools feeds for quick access to the latest entries.", + "openLinksInNewTab": false + }, + "title": "", + "type": "markdown", + "uiState": {} + } + }, + "gridData": { + "h": 8, + "i": "5123b88a-ad6a-4c30-8d4c-576fff3361e1", + "w": 24, + "x": 0, + "y": 0 + }, + "panelIndex": "5123b88a-ad6a-4c30-8d4c-576fff3361e1", + "title": "Contents", + "type": "visualization" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-b29120e5-51f6-4e7e-a0d0-b22bb945816e", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "currentIndexPatternId": "logs-*", + "layers": { + "b29120e5-51f6-4e7e-a0d0-b22bb945816e": { + "columnOrder": [ + "e3e689ce-3b28-4a0e-86bb-73fbc38dedd1" + ], + "columns": { + "e3e689ce-3b28-4a0e-86bb-73fbc38dedd1": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Unique count of Newly Observed Domain Feeds", + "operationType": "unique_count", + "params": { + "emptyAsNull": true + }, + "scale": "ratio", + "sourceField": "domaintools.domain" + } + }, + "incompleteColumns": {}, + "indexPatternId": "logs-*" + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "index": "2c733840-66cc-45b9-8fd0-9fb17d2b9be9", + "negate": false, + "params": [ + { + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "logs-*", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "ti_domaintools.nod_feed" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "ti_domaintools.nod_feed" + } + } + }, + { + "meta": { + "alias": null, + "disabled": false, + "field": "domaintools.domain", + "index": "logs-*", + "key": "domaintools.domain", + "negate": false, + "type": "exists", + "value": "exists" + }, + "query": { + "exists": { + "field": "domaintools.domain" + } + } + } + ], + "relation": "AND", + "type": "combined" + }, + "query": {} + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "color": "#54B399", + "icon": "empty", + "layerId": "b29120e5-51f6-4e7e-a0d0-b22bb945816e", + "layerType": "data", + "metricAccessor": "e3e689ce-3b28-4a0e-86bb-73fbc38dedd1" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsMetric" + }, + "enhancements": {}, + "hidePanelTitles": true + }, + "gridData": { + "h": 8, + "i": "6edf718a-aedf-4659-8661-be76e5935dde", + "w": 6, + "x": 24, + "y": 0 + }, + "panelIndex": "6edf718a-aedf-4659-8661-be76e5935dde", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-b29120e5-51f6-4e7e-a0d0-b22bb945816e", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "currentIndexPatternId": "logs-*", + "layers": { + "b29120e5-51f6-4e7e-a0d0-b22bb945816e": { + "columnOrder": [ + "e3e689ce-3b28-4a0e-86bb-73fbc38dedd1" + ], + "columns": { + "e3e689ce-3b28-4a0e-86bb-73fbc38dedd1": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Unique count of Newly Active Domain Feeds", + "operationType": "unique_count", + "params": { + "emptyAsNull": true + }, + "scale": "ratio", + "sourceField": "domaintools.domain" + } + }, + "incompleteColumns": {}, + "indexPatternId": "logs-*" + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "index": "bf265c5c-76e6-4da9-be04-398938462272", + "negate": false, + "params": [ + { + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "logs-*", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "ti_domaintools.nad_feed" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "ti_domaintools.nad_feed" + } + } + }, + { + "meta": { + "alias": null, + "disabled": false, + "field": "domaintools.domain", + "index": "logs-*", + "key": "domaintools.domain", + "negate": false, + "type": "exists", + "value": "exists" + }, + "query": { + "exists": { + "field": "domaintools.domain" + } + } + } + ], + "relation": "AND", + "type": "combined" + }, + "query": {} + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "color": "#1BB7DE", + "icon": "empty", + "layerId": "b29120e5-51f6-4e7e-a0d0-b22bb945816e", + "layerType": "data", + "metricAccessor": "e3e689ce-3b28-4a0e-86bb-73fbc38dedd1" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsMetric" + }, + "enhancements": {}, + "hidePanelTitles": true + }, + "gridData": { + "h": 8, + "i": "b142ba38-d785-416e-ae2f-24729d7a3907", + "w": 6, + "x": 30, + "y": 0 + }, + "panelIndex": "b142ba38-d785-416e-ae2f-24729d7a3907", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-b29120e5-51f6-4e7e-a0d0-b22bb945816e", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "currentIndexPatternId": "logs-*", + "layers": { + "b29120e5-51f6-4e7e-a0d0-b22bb945816e": { + "columnOrder": [ + "e3e689ce-3b28-4a0e-86bb-73fbc38dedd1" + ], + "columns": { + "e3e689ce-3b28-4a0e-86bb-73fbc38dedd1": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Unique count of Domain Discovery Feeds", + "operationType": "unique_count", + "params": { + "emptyAsNull": true + }, + "scale": "ratio", + "sourceField": "domaintools.domain" + } + }, + "incompleteColumns": {}, + "indexPatternId": "logs-*" + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "index": "303418fd-ef7f-46f7-bfc2-f7f8a276fd6d", + "negate": false, + "params": [ + { + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "logs-*", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "ti_domaintools.domaindiscovery" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "ti_domaintools.domaindiscovery" + } + } + }, + { + "meta": { + "alias": null, + "disabled": false, + "field": "domaintools.domain", + "index": "logs-*", + "key": "domaintools.domain", + "negate": false, + "type": "exists", + "value": "exists" + }, + "query": { + "exists": { + "field": "domaintools.domain" + } + } + } + ], + "relation": "AND", + "type": "combined" + }, + "query": {} + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "color": "#6092C0", + "icon": "empty", + "layerId": "b29120e5-51f6-4e7e-a0d0-b22bb945816e", + "layerType": "data", + "metricAccessor": "e3e689ce-3b28-4a0e-86bb-73fbc38dedd1" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsMetric" + }, + "enhancements": {}, + "hidePanelTitles": true + }, + "gridData": { + "h": 8, + "i": "4a28114a-934a-439d-bda3-397545a27d2d", + "w": 6, + "x": 36, + "y": 0 + }, + "panelIndex": "4a28114a-934a-439d-bda3-397545a27d2d", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-b29120e5-51f6-4e7e-a0d0-b22bb945816e", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "currentIndexPatternId": "logs-*", + "layers": { + "b29120e5-51f6-4e7e-a0d0-b22bb945816e": { + "columnOrder": [ + "e3e689ce-3b28-4a0e-86bb-73fbc38dedd1" + ], + "columns": { + "e3e689ce-3b28-4a0e-86bb-73fbc38dedd1": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Unique count of Domain RDAP Feeds", + "operationType": "unique_count", + "params": { + "emptyAsNull": true + }, + "scale": "ratio", + "sourceField": "domaintools.domain" + } + }, + "incompleteColumns": {}, + "indexPatternId": "logs-*" + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "index": "95a2738a-5aec-4694-9e6d-ab182becde24", + "negate": false, + "params": [ + { + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "logs-*", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "ti_domaintools.domainrdap" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "ti_domaintools.domainrdap" + } + } + }, + { + "meta": { + "alias": null, + "disabled": false, + "field": "domaintools.domain", + "index": "logs-*", + "key": "domaintools.domain", + "negate": false, + "type": "exists", + "value": "exists" + }, + "query": { + "exists": { + "field": "domaintools.domain" + } + } + } + ], + "relation": "AND", + "type": "combined" + }, + "query": {} + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "color": "#C6C9CB", + "icon": "empty", + "layerId": "b29120e5-51f6-4e7e-a0d0-b22bb945816e", + "layerType": "data", + "metricAccessor": "e3e689ce-3b28-4a0e-86bb-73fbc38dedd1" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsMetric" + }, + "enhancements": {}, + "hidePanelTitles": true + }, + "gridData": { + "h": 8, + "i": "f4f68fae-c8bf-4f4b-8da2-f8ef7d134172", + "w": 6, + "x": 42, + "y": 0 + }, + "panelIndex": "f4f68fae-c8bf-4f4b-8da2-f8ef7d134172", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-6c2ccc16-2ef8-4a91-800a-85ec8e128e91", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "788c583b-a0fe-4174-8d6f-c8282247f3bd", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "layers": { + "6c2ccc16-2ef8-4a91-800a-85ec8e128e91": { + "columnOrder": [ + "d41ba4f5-06fd-4d9a-950c-bb002713f2f7", + "c56d13b4-7805-466d-b717-ddd4f9fca449" + ], + "columns": { + "c56d13b4-7805-466d-b717-ddd4f9fca449": { + "dataType": "number", + "isBucketed": false, + "label": "Count of records", + "operationType": "count", + "params": { + "emptyAsNull": true + }, + "scale": "ratio", + "sourceField": "___records___" + }, + "d41ba4f5-06fd-4d9a-950c-bb002713f2f7": { + "dataType": "date", + "isBucketed": true, + "label": "domaintools.timestamp", + "operationType": "date_histogram", + "params": { + "dropPartials": false, + "includeEmptyRows": true, + "interval": "auto" + }, + "scale": "interval", + "sourceField": "domaintools.timestamp" + } + }, + "incompleteColumns": {}, + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "788c583b-a0fe-4174-8d6f-c8282247f3bd", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "ti_domaintools.nod_feed" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "ti_domaintools.nod_feed" + } + } + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "None", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "c56d13b4-7805-466d-b717-ddd4f9fca449" + ], + "layerId": "6c2ccc16-2ef8-4a91-800a-85ec8e128e91", + "layerType": "data", + "position": "top", + "seriesType": "line", + "showGridlines": false, + "xAccessor": "d41ba4f5-06fd-4d9a-950c-bb002713f2f7" + } + ], + "legend": { + "isVisible": true, + "position": "right" + }, + "preferredSeriesType": "line", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsXY" + }, + "enhancements": {}, + "hidePanelTitles": false + }, + "gridData": { + "h": 14, + "i": "aef56e82-dba8-48fc-ad1e-1633e295d7ba", + "w": 21, + "x": 0, + "y": 8 + }, + "panelIndex": "aef56e82-dba8-48fc-ad1e-1633e295d7ba", + "title": "Newly Observed Domains over Time [Logs DomainTools Feeds]", + "type": "lens" + }, + { + "embeddableConfig": { + "enhancements": {} + }, + "gridData": { + "h": 14, + "i": "020517f8-2f8d-41af-8c63-c7079f5fcb6a", + "w": 27, + "x": 21, + "y": 8 + }, + "panelIndex": "020517f8-2f8d-41af-8c63-c7079f5fcb6a", + "panelRefName": "panel_020517f8-2f8d-41af-8c63-c7079f5fcb6a", + "type": "search" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-6c2ccc16-2ef8-4a91-800a-85ec8e128e91", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "currentIndexPatternId": "logs-*", + "layers": { + "6c2ccc16-2ef8-4a91-800a-85ec8e128e91": { + "columnOrder": [ + "d41ba4f5-06fd-4d9a-950c-bb002713f2f7", + "c56d13b4-7805-466d-b717-ddd4f9fca449" + ], + "columns": { + "c56d13b4-7805-466d-b717-ddd4f9fca449": { + "dataType": "number", + "isBucketed": false, + "label": "Count of records", + "operationType": "count", + "params": { + "emptyAsNull": true + }, + "scale": "ratio", + "sourceField": "___records___" + }, + "d41ba4f5-06fd-4d9a-950c-bb002713f2f7": { + "dataType": "date", + "isBucketed": true, + "label": "domaintools.timestamp", + "operationType": "date_histogram", + "params": { + "dropPartials": false, + "includeEmptyRows": true, + "interval": "auto" + }, + "scale": "interval", + "sourceField": "domaintools.timestamp" + } + }, + "incompleteColumns": {}, + "indexPatternId": "logs-*", + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "a297da4b-3add-4b07-a74d-4192baf47a11", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "ti_domaintools.nad_feed" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "ti_domaintools.nad_feed" + } + } + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "None", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "c56d13b4-7805-466d-b717-ddd4f9fca449" + ], + "layerId": "6c2ccc16-2ef8-4a91-800a-85ec8e128e91", + "layerType": "data", + "position": "top", + "seriesType": "line", + "showGridlines": false, + "xAccessor": "d41ba4f5-06fd-4d9a-950c-bb002713f2f7", + "yConfig": [ + { + "color": "#1bb7de", + "forAccessor": "c56d13b4-7805-466d-b717-ddd4f9fca449" + } + ] + } + ], + "legend": { + "isVisible": true, + "position": "right" + }, + "preferredSeriesType": "line", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsXY" + }, + "enhancements": {}, + "hidePanelTitles": false + }, + "gridData": { + "h": 13, + "i": "982c1274-e635-47ce-9afe-8f7f2efb5b28", + "w": 21, + "x": 0, + "y": 22 + }, + "panelIndex": "982c1274-e635-47ce-9afe-8f7f2efb5b28", + "title": "Newly Active Domains over Time [Logs DomainTools Feeds]", + "type": "lens" + }, + { + "embeddableConfig": { + "enhancements": {} + }, + "gridData": { + "h": 13, + "i": "fc9c237d-9c02-4391-85e5-af1aa8a76e42", + "w": 27, + "x": 21, + "y": 22 + }, + "panelIndex": "fc9c237d-9c02-4391-85e5-af1aa8a76e42", + "panelRefName": "panel_fc9c237d-9c02-4391-85e5-af1aa8a76e42", + "type": "search" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-0c69a811-9cb5-4aef-85a4-176619ccf7d7", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "0b1b74d6-22f6-43ee-adc0-cfe3700b2cdc", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "layers": { + "0c69a811-9cb5-4aef-85a4-176619ccf7d7": { + "columnOrder": [ + "c228f756-385d-4efe-942b-edb16a8a3827", + "1dcef8c5-9814-4c38-814b-cc15efd97b40" + ], + "columns": { + "1dcef8c5-9814-4c38-814b-cc15efd97b40": { + "dataType": "number", + "isBucketed": false, + "label": "Count of records", + "operationType": "count", + "params": { + "emptyAsNull": true + }, + "scale": "ratio", + "sourceField": "___records___" + }, + "c228f756-385d-4efe-942b-edb16a8a3827": { + "dataType": "date", + "isBucketed": true, + "label": "domaintools.timestamp", + "operationType": "date_histogram", + "params": { + "dropPartials": false, + "includeEmptyRows": true, + "interval": "auto" + }, + "scale": "interval", + "sourceField": "domaintools.timestamp" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "0b1b74d6-22f6-43ee-adc0-cfe3700b2cdc", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "ti_domaintools.domaindiscovery" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "ti_domaintools.domaindiscovery" + } + } + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "layers": [ + { + "accessors": [ + "1dcef8c5-9814-4c38-814b-cc15efd97b40" + ], + "layerId": "0c69a811-9cb5-4aef-85a4-176619ccf7d7", + "layerType": "data", + "position": "top", + "seriesType": "line", + "showGridlines": false, + "xAccessor": "c228f756-385d-4efe-942b-edb16a8a3827", + "yConfig": [ + { + "color": "#6092c0", + "forAccessor": "1dcef8c5-9814-4c38-814b-cc15efd97b40" + } + ] + } + ], + "legend": { + "isVisible": true, + "position": "right" + }, + "preferredSeriesType": "line", + "title": "Empty XY chart", + "valueLabels": "hide" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsXY" + }, + "enhancements": {}, + "hidePanelTitles": false + }, + "gridData": { + "h": 13, + "i": "1efcc97b-cd5d-4e24-a7b8-718baeb2ed77", + "w": 21, + "x": 0, + "y": 35 + }, + "panelIndex": "1efcc97b-cd5d-4e24-a7b8-718baeb2ed77", + "title": "Domain Discovery over Time [Logs DomainTools Feeds]", + "type": "lens" + }, + { + "embeddableConfig": { + "enhancements": {} + }, + "gridData": { + "h": 13, + "i": "2561b9a0-b035-48a5-9d24-906188dc270f", + "w": 27, + "x": 21, + "y": 35 + }, + "panelIndex": "2561b9a0-b035-48a5-9d24-906188dc270f", + "panelRefName": "panel_2561b9a0-b035-48a5-9d24-906188dc270f", + "type": "search" + }, + { + "embeddableConfig": { + "enhancements": {} + }, + "gridData": { + "h": 13, + "i": "e69be1e3-3922-4306-826f-e06f19455a41", + "w": 27, + "x": 21, + "y": 48 + }, + "panelIndex": "e69be1e3-3922-4306-826f-e06f19455a41", + "panelRefName": "panel_e69be1e3-3922-4306-826f-e06f19455a41", + "type": "search" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-76906f91-d089-4353-920a-49f0201ece31", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "currentIndexPatternId": "logs-*", + "layers": { + "76906f91-d089-4353-920a-49f0201ece31": { + "columnOrder": [ + "740d5a07-22e8-429e-aa3b-90b480acd291", + "05b4f99d-2dae-4339-8eef-c789308fb2cb" + ], + "columns": { + "05b4f99d-2dae-4339-8eef-c789308fb2cb": { + "dataType": "number", + "isBucketed": false, + "label": "Count of records", + "operationType": "count", + "params": { + "emptyAsNull": true + }, + "scale": "ratio", + "sourceField": "___records___" + }, + "740d5a07-22e8-429e-aa3b-90b480acd291": { + "dataType": "date", + "isBucketed": true, + "label": "domaintools.timestamp", + "operationType": "date_histogram", + "params": { + "dropPartials": false, + "includeEmptyRows": true, + "interval": "auto" + }, + "scale": "interval", + "sourceField": "domaintools.timestamp" + } + }, + "incompleteColumns": {}, + "indexPatternId": "logs-*", + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "9a18bb8d-1556-4b88-9e41-548cd5ff48d9", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "ti_domaintools.domainrdap" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "ti_domaintools.domainrdap" + } + } + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "None", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "05b4f99d-2dae-4339-8eef-c789308fb2cb" + ], + "layerId": "76906f91-d089-4353-920a-49f0201ece31", + "layerType": "data", + "seriesType": "line", + "xAccessor": "740d5a07-22e8-429e-aa3b-90b480acd291", + "yConfig": [ + { + "color": "#c6c9cb", + "forAccessor": "05b4f99d-2dae-4339-8eef-c789308fb2cb" + } + ] + } + ], + "legend": { + "isVisible": true, + "position": "right" + }, + "preferredSeriesType": "line", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsXY" + }, + "enhancements": {} + }, + "gridData": { + "h": 13, + "i": "ec5325e1-573b-441a-b52c-6e7d7259afde", + "w": 21, + "x": 0, + "y": 48 + }, + "panelIndex": "ec5325e1-573b-441a-b52c-6e7d7259afde", + "type": "lens" + } + ], + "timeRestore": false, + "title": "[Logs DomainTools Feeds] Overview", + "version": 1 + }, + "coreMigrationVersion": "8.8.0", + "created_at": "2025-07-02T16:12:12.499Z", + "id": "ti_domaintools-7ad9a714-58db-45e3-ba84-1e2dff1eb9a5", + "managed": false, + "references": [ + { + "id": "logs-*", + "name": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "6edf718a-aedf-4659-8661-be76e5935dde:indexpattern-datasource-layer-b29120e5-51f6-4e7e-a0d0-b22bb945816e", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "b142ba38-d785-416e-ae2f-24729d7a3907:indexpattern-datasource-layer-b29120e5-51f6-4e7e-a0d0-b22bb945816e", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "4a28114a-934a-439d-bda3-397545a27d2d:indexpattern-datasource-layer-b29120e5-51f6-4e7e-a0d0-b22bb945816e", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "f4f68fae-c8bf-4f4b-8da2-f8ef7d134172:indexpattern-datasource-layer-b29120e5-51f6-4e7e-a0d0-b22bb945816e", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "aef56e82-dba8-48fc-ad1e-1633e295d7ba:indexpattern-datasource-layer-6c2ccc16-2ef8-4a91-800a-85ec8e128e91", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "aef56e82-dba8-48fc-ad1e-1633e295d7ba:788c583b-a0fe-4174-8d6f-c8282247f3bd", + "type": "index-pattern" + }, + { + "id": "ti_domaintools-7f96fd28-25f1-4e44-9825-2617faa05217", + "name": "020517f8-2f8d-41af-8c63-c7079f5fcb6a:panel_020517f8-2f8d-41af-8c63-c7079f5fcb6a", + "type": "search" + }, + { + "id": "logs-*", + "name": "982c1274-e635-47ce-9afe-8f7f2efb5b28:indexpattern-datasource-layer-6c2ccc16-2ef8-4a91-800a-85ec8e128e91", + "type": "index-pattern" + }, + { + "id": "ti_domaintools-0003128e-b815-468e-913d-b091a156a805", + "name": "fc9c237d-9c02-4391-85e5-af1aa8a76e42:panel_fc9c237d-9c02-4391-85e5-af1aa8a76e42", + "type": "search" + }, + { + "id": "logs-*", + "name": "1efcc97b-cd5d-4e24-a7b8-718baeb2ed77:indexpattern-datasource-layer-0c69a811-9cb5-4aef-85a4-176619ccf7d7", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "1efcc97b-cd5d-4e24-a7b8-718baeb2ed77:0b1b74d6-22f6-43ee-adc0-cfe3700b2cdc", + "type": "index-pattern" + }, + { + "id": "ti_domaintools-653c19ae-f37b-4414-8fc9-ebc1b3abe29b", + "name": "2561b9a0-b035-48a5-9d24-906188dc270f:panel_2561b9a0-b035-48a5-9d24-906188dc270f", + "type": "search" + }, + { + "id": "ti_domaintools-8ac2e86f-4c5b-4bbf-b3af-a39cc710f84e", + "name": "e69be1e3-3922-4306-826f-e06f19455a41:panel_e69be1e3-3922-4306-826f-e06f19455a41", + "type": "search" + }, + { + "id": "logs-*", + "name": "ec5325e1-573b-441a-b52c-6e7d7259afde:indexpattern-datasource-layer-76906f91-d089-4353-920a-49f0201ece31", + "type": "index-pattern" + } + ], + "type": "dashboard", + "typeMigrationVersion": "8.9.0" +} \ No newline at end of file diff --git a/packages/ti_domaintools/kibana/dashboard/ti_domaintools-e403767d-161a-4fa5-9c10-a8dac12a3f67.json b/packages/ti_domaintools/kibana/dashboard/ti_domaintools-e403767d-161a-4fa5-9c10-a8dac12a3f67.json deleted file mode 100644 index 0f25f805f03..00000000000 --- a/packages/ti_domaintools/kibana/dashboard/ti_domaintools-e403767d-161a-4fa5-9c10-a8dac12a3f67.json +++ /dev/null @@ -1,396 +0,0 @@ -{ - "attributes": { - "controlGroupInput": { - "chainingSystem": "HIERARCHICAL", - "controlStyle": "oneLine", - "ignoreParentSettingsJSON": { - "ignoreFilters": false, - "ignoreQuery": false, - "ignoreTimerange": false, - "ignoreValidations": false - }, - "panelsJSON": { - "69930059-92ea-4c51-84ec-443e72bf9dde": { - "explicitInput": { - "enhancements": {}, - "fieldName": "threat.indicator.name", - "grow": true, - "id": "69930059-92ea-4c51-84ec-443e72bf9dde", - "searchTechnique": "prefix", - "title": "Domain", - "width": "medium" - }, - "grow": true, - "order": 0, - "type": "optionsListControl", - "width": "medium" - } - }, - "showApplySelections": false - }, - "description": "This dashboard provides an overview of data collected through the DomainTools Real Time Unified Feeds integration.", - "kibanaSavedObjectMeta": { - "searchSourceJSON": { - "filter": [ - { - "$state": { - "store": "appState" - }, - "meta": { - "alias": null, - "disabled": false, - "field": "data_stream.dataset", - "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", - "key": "data_stream.dataset", - "negate": false, - "params": { - "query": "ti_domaintools.nod_feed" - }, - "type": "phrase" - }, - "query": { - "match_phrase": { - "data_stream.dataset": "ti_domaintools.nod_feed" - } - } - }, - { - "$state": { - "store": "appState" - }, - "meta": { - "alias": null, - "disabled": false, - "field": "labels.is_ioc_transform_source", - "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[1].meta.index", - "key": "labels.is_ioc_transform_source", - "negate": false, - "params": { - "query": "false" - }, - "type": "phrase" - }, - "query": { - "match_phrase": { - "labels.is_ioc_transform_source": "false" - } - } - } - ], - "query": { - "language": "kuery", - "query": "" - } - } - }, - "optionsJSON": { - "hidePanelTitles": false, - "syncColors": false, - "syncCursor": true, - "syncTooltips": false, - "useMargins": true - }, - "panelsJSON": [ - { - "embeddableConfig": { - "description": "", - "enhancements": {}, - "savedVis": { - "data": { - "aggs": [], - "searchSource": { - "filter": [], - "query": { - "language": "kuery", - "query": "" - } - } - }, - "description": "", - "id": "", - "params": { - "fontSize": 12, - "markdown": "**Overview** \nThis dashboard provides an overview of data collected through the DomainTools Real Time Unified Feeds integration. It displays the total number of unique observed domains to give a snapshot of current activity, visualizes domain observations over time to highlight trends or spikes, and includes a view of recently observed domains for quick access to the latest entries.", - "openLinksInNewTab": false - }, - "title": "", - "type": "markdown", - "uiState": {} - } - }, - "gridData": { - "h": 6, - "i": "21b53b57-e166-4aa3-b0d8-b546011f0fd2", - "w": 33, - "x": 0, - "y": 0 - }, - "panelIndex": "21b53b57-e166-4aa3-b0d8-b546011f0fd2", - "title": "Table of Contents", - "type": "visualization" - }, - { - "embeddableConfig": { - "attributes": { - "references": [ - { - "id": "logs-*", - "name": "indexpattern-datasource-layer-5874cfad-c1d1-4d09-99f0-c6cb94fb7dee", - "type": "index-pattern" - } - ], - "state": { - "adHocDataViews": {}, - "datasourceStates": { - "formBased": { - "currentIndexPatternId": "logs-*", - "layers": { - "5874cfad-c1d1-4d09-99f0-c6cb94fb7dee": { - "columnOrder": [ - "55fe47ed-c084-49bb-acaf-31ee72309064" - ], - "columns": { - "55fe47ed-c084-49bb-acaf-31ee72309064": { - "customLabel": true, - "dataType": "number", - "isBucketed": false, - "label": "Unique Observed Domains", - "operationType": "unique_count", - "params": { - "emptyAsNull": false - }, - "scale": "ratio", - "sourceField": "domaintools.domain" - } - }, - "ignoreGlobalFilters": false, - "incompleteColumns": {}, - "indexPatternId": "logs-*", - "sampling": 1 - } - } - }, - "indexpattern": { - "layers": {} - }, - "textBased": { - "layers": {} - } - }, - "filters": [], - "internalReferences": [], - "query": { - "language": "kuery", - "query": "" - }, - "visualization": { - "color": "#6092C0", - "layerId": "5874cfad-c1d1-4d09-99f0-c6cb94fb7dee", - "layerType": "data", - "metricAccessor": "55fe47ed-c084-49bb-acaf-31ee72309064" - } - }, - "title": "", - "type": "lens", - "visualizationType": "lnsMetric" - }, - "enhancements": {}, - "hidePanelTitles": true - }, - "gridData": { - "h": 6, - "i": "2c5bc5c6-cdac-48ca-a10e-c2d9a028315c", - "w": 15, - "x": 33, - "y": 0 - }, - "panelIndex": "2c5bc5c6-cdac-48ca-a10e-c2d9a028315c", - "title": "Unique Observed Domains [Logs DomainTools Real Time Unified Feeds]", - "type": "lens" - }, - { - "embeddableConfig": { - "attributes": { - "references": [ - { - "id": "logs-*", - "name": "indexpattern-datasource-layer-d1a413b2-183d-4646-98f9-17f531556c9c", - "type": "index-pattern" - } - ], - "state": { - "adHocDataViews": {}, - "datasourceStates": { - "formBased": { - "currentIndexPatternId": "logs-*", - "layers": { - "d1a413b2-183d-4646-98f9-17f531556c9c": { - "columnOrder": [ - "1f79320c-8432-443e-a05a-b98bc3086ce6", - "fcb1cf22-78f3-4ca5-83dc-4fef1c6b1a81" - ], - "columns": { - "1f79320c-8432-443e-a05a-b98bc3086ce6": { - "customLabel": true, - "dataType": "date", - "isBucketed": true, - "label": "Timestamp", - "operationType": "date_histogram", - "params": { - "dropPartials": false, - "includeEmptyRows": true, - "interval": "auto" - }, - "scale": "interval", - "sourceField": "domaintools.timestamp" - }, - "fcb1cf22-78f3-4ca5-83dc-4fef1c6b1a81": { - "customLabel": true, - "dataType": "number", - "isBucketed": false, - "label": "Count", - "operationType": "count", - "params": { - "emptyAsNull": false - }, - "scale": "ratio", - "sourceField": "___records___" - } - }, - "ignoreGlobalFilters": false, - "incompleteColumns": {}, - "indexPatternId": "logs-*", - "sampling": 1 - } - } - }, - "indexpattern": { - "layers": {} - }, - "textBased": { - "layers": {} - } - }, - "filters": [], - "internalReferences": [], - "query": { - "language": "kuery", - "query": "" - }, - "visualization": { - "layers": [ - { - "accessors": [ - "fcb1cf22-78f3-4ca5-83dc-4fef1c6b1a81" - ], - "colorMapping": { - "assignments": [], - "colorMode": { - "type": "categorical" - }, - "paletteId": "eui_amsterdam_color_blind", - "specialAssignments": [ - { - "color": { - "type": "loop" - }, - "rule": { - "type": "other" - }, - "touched": false - } - ] - }, - "layerId": "d1a413b2-183d-4646-98f9-17f531556c9c", - "layerType": "data", - "position": "top", - "seriesType": "line", - "showGridlines": false, - "xAccessor": "1f79320c-8432-443e-a05a-b98bc3086ce6" - } - ], - "legend": { - "isVisible": true, - "position": "right" - }, - "preferredSeriesType": "line", - "title": "Empty XY chart", - "valueLabels": "hide" - } - }, - "title": "", - "type": "lens", - "visualizationType": "lnsXY" - }, - "enhancements": {} - }, - "gridData": { - "h": 14, - "i": "9adc6a53-0fbd-42e6-babf-809748e0cfc7", - "w": 29, - "x": 0, - "y": 6 - }, - "panelIndex": "9adc6a53-0fbd-42e6-babf-809748e0cfc7", - "title": "Observed Domains over Time [Logs DomainTools Real Time Unified Feeds]", - "type": "lens" - }, - { - "embeddableConfig": { - "enhancements": {} - }, - "gridData": { - "h": 14, - "i": "f9f47276-8ce0-4c5c-84d0-984cbaff85f4", - "w": 19, - "x": 29, - "y": 6 - }, - "panelIndex": "f9f47276-8ce0-4c5c-84d0-984cbaff85f4", - "panelRefName": "panel_f9f47276-8ce0-4c5c-84d0-984cbaff85f4", - "type": "search" - } - ], - "timeRestore": false, - "title": "[Logs DomainTools Real Time Unified Feeds] Overview", - "version": 2 - }, - "coreMigrationVersion": "8.8.0", - "created_at": "2025-06-18T07:05:45.070Z", - "id": "ti_domaintools-e403767d-161a-4fa5-9c10-a8dac12a3f67", - "references": [ - { - "id": "logs-*", - "name": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", - "type": "index-pattern" - }, - { - "id": "logs-*", - "name": "kibanaSavedObjectMeta.searchSourceJSON.filter[1].meta.index", - "type": "index-pattern" - }, - { - "id": "logs-*", - "name": "2c5bc5c6-cdac-48ca-a10e-c2d9a028315c:indexpattern-datasource-layer-5874cfad-c1d1-4d09-99f0-c6cb94fb7dee", - "type": "index-pattern" - }, - { - "id": "logs-*", - "name": "9adc6a53-0fbd-42e6-babf-809748e0cfc7:indexpattern-datasource-layer-d1a413b2-183d-4646-98f9-17f531556c9c", - "type": "index-pattern" - }, - { - "id": "ti_domaintools-c258b26f-c167-4075-87e3-79a8a130ab24", - "name": "f9f47276-8ce0-4c5c-84d0-984cbaff85f4:panel_f9f47276-8ce0-4c5c-84d0-984cbaff85f4", - "type": "search" - }, - { - "id": "logs-*", - "name": "controlGroup_69930059-92ea-4c51-84ec-443e72bf9dde:optionsListDataView", - "type": "index-pattern" - } - ], - "type": "dashboard", - "typeMigrationVersion": "10.2.0", - "updated_by": "u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0" -} \ No newline at end of file diff --git a/packages/ti_domaintools/kibana/search/ti_domaintools-0003128e-b815-468e-913d-b091a156a805.json b/packages/ti_domaintools/kibana/search/ti_domaintools-0003128e-b815-468e-913d-b091a156a805.json new file mode 100644 index 00000000000..c8d7fe48a35 --- /dev/null +++ b/packages/ti_domaintools/kibana/search/ti_domaintools-0003128e-b815-468e-913d-b091a156a805.json @@ -0,0 +1,55 @@ +{ + "attributes": { + "columns": [ + "domaintools.domain", + "domaintools.feed", + "domaintools.timestamp", + "threat.feed.name", + "threat.feed.description" + ], + "description": "", + "grid": {}, + "hideChart": false, + "isTextBasedQuery": false, + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.index", + "query": { + "language": "kuery", + "query": "data_stream.dataset : \"ti_domaintools.nad_feed\" and labels.is_ioc_transform_source : \"false\" " + } + } + }, + "refreshInterval": { + "pause": true, + "value": 60000 + }, + "sort": [ + [ + "@timestamp", + "desc" + ] + ], + "timeRange": { + "from": "now-15m", + "to": "now" + }, + "timeRestore": false, + "title": "Recently Newly Active Domains [Logs DomainTools Feeds]", + "usesAdHocDataView": false + }, + "coreMigrationVersion": "8.8.0", + "created_at": "2025-07-02T15:57:44.851Z", + "id": "ti_domaintools-0003128e-b815-468e-913d-b091a156a805", + "managed": false, + "references": [ + { + "id": "logs-*", + "name": "kibanaSavedObjectMeta.searchSourceJSON.index", + "type": "index-pattern" + } + ], + "type": "search", + "typeMigrationVersion": "8.0.0" +} \ No newline at end of file diff --git a/packages/ti_domaintools/kibana/search/ti_domaintools-653c19ae-f37b-4414-8fc9-ebc1b3abe29b.json b/packages/ti_domaintools/kibana/search/ti_domaintools-653c19ae-f37b-4414-8fc9-ebc1b3abe29b.json new file mode 100644 index 00000000000..3d46837a002 --- /dev/null +++ b/packages/ti_domaintools/kibana/search/ti_domaintools-653c19ae-f37b-4414-8fc9-ebc1b3abe29b.json @@ -0,0 +1,55 @@ +{ + "attributes": { + "columns": [ + "domaintools.domain", + "domaintools.feed", + "domaintools.timestamp", + "threat.feed.name", + "threat.feed.description" + ], + "description": "", + "grid": {}, + "hideChart": false, + "isTextBasedQuery": false, + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.index", + "query": { + "language": "kuery", + "query": "data_stream.dataset : \"ti_domaintools.domaindiscovery\" and labels.is_ioc_transform_source : \"false\" " + } + } + }, + "refreshInterval": { + "pause": true, + "value": 60000 + }, + "sort": [ + [ + "@timestamp", + "desc" + ] + ], + "timeRange": { + "from": "now-15m", + "to": "now" + }, + "timeRestore": false, + "title": "Recently Domain Discovery [Logs DomainTools Feeds]", + "usesAdHocDataView": false + }, + "coreMigrationVersion": "8.8.0", + "created_at": "2025-07-02T15:58:28.355Z", + "id": "ti_domaintools-653c19ae-f37b-4414-8fc9-ebc1b3abe29b", + "managed": false, + "references": [ + { + "id": "logs-*", + "name": "kibanaSavedObjectMeta.searchSourceJSON.index", + "type": "index-pattern" + } + ], + "type": "search", + "typeMigrationVersion": "8.0.0" +} \ No newline at end of file diff --git a/packages/ti_domaintools/kibana/search/ti_domaintools-7f96fd28-25f1-4e44-9825-2617faa05217.json b/packages/ti_domaintools/kibana/search/ti_domaintools-7f96fd28-25f1-4e44-9825-2617faa05217.json new file mode 100644 index 00000000000..7297302a3c6 --- /dev/null +++ b/packages/ti_domaintools/kibana/search/ti_domaintools-7f96fd28-25f1-4e44-9825-2617faa05217.json @@ -0,0 +1,55 @@ +{ + "attributes": { + "columns": [ + "domaintools.domain", + "domaintools.feed", + "domaintools.timestamp", + "threat.feed.name", + "threat.feed.description" + ], + "description": "", + "grid": {}, + "hideChart": false, + "isTextBasedQuery": false, + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.index", + "query": { + "language": "kuery", + "query": "data_stream.dataset : \"ti_domaintools.nod_feed\" and labels.is_ioc_transform_source : \"false\" " + } + } + }, + "refreshInterval": { + "pause": true, + "value": 60000 + }, + "sort": [ + [ + "@timestamp", + "desc" + ] + ], + "timeRange": { + "from": "now-60m", + "to": "now" + }, + "timeRestore": false, + "title": "Recently Newly Observed Domains [Logs DomainTools Feeds]", + "usesAdHocDataView": false + }, + "coreMigrationVersion": "8.8.0", + "created_at": "2025-07-02T15:48:36.827Z", + "id": "ti_domaintools-7f96fd28-25f1-4e44-9825-2617faa05217", + "managed": false, + "references": [ + { + "id": "logs-*", + "name": "kibanaSavedObjectMeta.searchSourceJSON.index", + "type": "index-pattern" + } + ], + "type": "search", + "typeMigrationVersion": "8.0.0" +} \ No newline at end of file diff --git a/packages/ti_domaintools/kibana/search/ti_domaintools-8ac2e86f-4c5b-4bbf-b3af-a39cc710f84e.json b/packages/ti_domaintools/kibana/search/ti_domaintools-8ac2e86f-4c5b-4bbf-b3af-a39cc710f84e.json new file mode 100644 index 00000000000..44f397d27e4 --- /dev/null +++ b/packages/ti_domaintools/kibana/search/ti_domaintools-8ac2e86f-4c5b-4bbf-b3af-a39cc710f84e.json @@ -0,0 +1,58 @@ +{ + "attributes": { + "columns": [ + "domaintools.domain", + "domaintools.feed", + "domaintools.timestamp", + "threat.feed.name", + "threat.feed.description", + "domaintools.parsed_record.parsed_fields.creation_date", + "domaintools.parsed_record.parsed_fields.registrar.name", + "domaintools.parsed_record.parsed_fields.expiration_date" + ], + "description": "", + "grid": {}, + "hideChart": false, + "isTextBasedQuery": false, + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.index", + "query": { + "language": "kuery", + "query": "data_stream.dataset : \"ti_domaintools.domainrdap\" and labels.is_ioc_transform_source : \"false\" " + } + } + }, + "refreshInterval": { + "pause": true, + "value": 60000 + }, + "sort": [ + [ + "@timestamp", + "desc" + ] + ], + "timeRange": { + "from": "now-15m", + "to": "now" + }, + "timeRestore": false, + "title": "Recently Domain RDAP [Logs DomainTools Feeds]", + "usesAdHocDataView": false + }, + "coreMigrationVersion": "8.8.0", + "created_at": "2025-07-02T15:59:46.264Z", + "id": "ti_domaintools-8ac2e86f-4c5b-4bbf-b3af-a39cc710f84e", + "managed": false, + "references": [ + { + "id": "logs-*", + "name": "kibanaSavedObjectMeta.searchSourceJSON.index", + "type": "index-pattern" + } + ], + "type": "search", + "typeMigrationVersion": "8.0.0" +} \ No newline at end of file diff --git a/packages/ti_domaintools/kibana/search/ti_domaintools-c258b26f-c167-4075-87e3-79a8a130ab24.json b/packages/ti_domaintools/kibana/search/ti_domaintools-c258b26f-c167-4075-87e3-79a8a130ab24.json deleted file mode 100644 index 7f2942abd17..00000000000 --- a/packages/ti_domaintools/kibana/search/ti_domaintools-c258b26f-c167-4075-87e3-79a8a130ab24.json +++ /dev/null @@ -1,104 +0,0 @@ -{ - "attributes": { - "columns": [ - "domaintools.domain" - ], - "description": "", - "grid": { - "columns": { - "@timestamp": { - "width": 212 - } - } - }, - "hideChart": false, - "isTextBasedQuery": false, - "kibanaSavedObjectMeta": { - "searchSourceJSON": { - "filter": [ - { - "$state": { - "store": "appState" - }, - "meta": { - "alias": null, - "disabled": false, - "field": "data_stream.dataset", - "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", - "key": "data_stream.dataset", - "negate": false, - "params": { - "query": "ti_domaintools.nod_feed" - }, - "type": "phrase" - }, - "query": { - "match_phrase": { - "data_stream.dataset": "ti_domaintools.nod_feed" - } - } - }, - { - "$state": { - "store": "appState" - }, - "meta": { - "alias": null, - "disabled": false, - "field": "labels.is_ioc_transform_source", - "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[1].meta.index", - "key": "labels.is_ioc_transform_source", - "negate": false, - "params": { - "query": "false" - }, - "type": "phrase" - }, - "query": { - "match_phrase": { - "labels.is_ioc_transform_source": "false" - } - } - } - ], - "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.index", - "query": { - "language": "kuery", - "query": "" - } - } - }, - "sort": [ - [ - "@timestamp", - "desc" - ] - ], - "timeRestore": false, - "title": "Recently Observed Domains [Logs DomainTools Real Time Unified Feeds]" - }, - "coreMigrationVersion": "8.8.0", - "created_at": "2025-06-18T07:08:25.643Z", - "created_by": "u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0", - "id": "ti_domaintools-c258b26f-c167-4075-87e3-79a8a130ab24", - "references": [ - { - "id": "logs-*", - "name": "kibanaSavedObjectMeta.searchSourceJSON.index", - "type": "index-pattern" - }, - { - "id": "logs-*", - "name": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", - "type": "index-pattern" - }, - { - "id": "logs-*", - "name": "kibanaSavedObjectMeta.searchSourceJSON.filter[1].meta.index", - "type": "index-pattern" - } - ], - "type": "search", - "typeMigrationVersion": "10.4.0", - "updated_by": "u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0" -} \ No newline at end of file diff --git a/packages/ti_domaintools/kibana/search/ti_domaintools-eb599614-25b6-48b2-9225-1e0789619305.json b/packages/ti_domaintools/kibana/search/ti_domaintools-eb599614-25b6-48b2-9225-1e0789619305.json deleted file mode 100644 index a513812dfdb..00000000000 --- a/packages/ti_domaintools/kibana/search/ti_domaintools-eb599614-25b6-48b2-9225-1e0789619305.json +++ /dev/null @@ -1,78 +0,0 @@ -{ - "attributes": { - "columns": [ - "domaintools.domain" - ], - "description": "", - "grid": { - "columns": { - "@timestamp": { - "width": 212 - } - } - }, - "hideChart": false, - "isTextBasedQuery": false, - "kibanaSavedObjectMeta": { - "searchSourceJSON": { - "filter": [ - { - "$state": { - "store": "appState" - }, - "meta": { - "alias": null, - "disabled": false, - "field": "_index", - "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", - "key": "_index", - "negate": false, - "params": { - "query": "logs-ti_domaintools.nod_feed-*" - }, - "type": "phrase" - }, - "query": { - "match_phrase": { - "_index": "logs-ti_domaintools.nod_feed-*" - } - } - } - ], - "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.index", - "query": { - "language": "kuery", - "query": "_index : logs-ti_domaintools.nod_feed-*" - } - } - }, - "sort": [ - [ - "@timestamp", - "desc" - ] - ], - "timeRestore": false, - "title": "Recently Observed Domains [Logs DomainTools Real Time Unified Feeds]" - }, - "coreMigrationVersion": "8.8.0", - "created_at": "2025-06-02T11:19:33.761Z", - "created_by": "u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0", - "id": "ti_domaintools-eb599614-25b6-48b2-9225-1e0789619305", - "managed": false, - "references": [ - { - "id": "logs-*", - "name": "kibanaSavedObjectMeta.searchSourceJSON.index", - "type": "index-pattern" - }, - { - "id": "logs-*", - "name": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", - "type": "index-pattern" - } - ], - "type": "search", - "typeMigrationVersion": "10.4.0", - "updated_by": "u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0" -} \ No newline at end of file diff --git a/packages/ti_domaintools/manifest.yml b/packages/ti_domaintools/manifest.yml index ec77848cf6a..ba934a24714 100644 --- a/packages/ti_domaintools/manifest.yml +++ b/packages/ti_domaintools/manifest.yml @@ -1,10 +1,10 @@ format_version: 3.3.0 name: ti_domaintools -title: "DomainTools Real Time Unified Feeds" -version: "1.0.0" +title: "DomainTools Feeds" +version: "1.1.0" source: license: "Elastic-2.0" -description: "The DomainTools NOD Feed provides real-time access to newly registered and observed domains, enabling proactive threat detection and defense." +description: "DomainTools Feeds provide data on the different stages of the domain lifecycle: from first-observed in the wild, to newly re-activated after a period of quiet." type: integration categories: - security @@ -26,12 +26,12 @@ icons: type: image/svg+xml policy_templates: - name: domaintools - title: DomainTools Newly Observed Domains - description: "DomainTools Newly Observed Domains provides real-time access to newly registered and observed domains, enabling proactive threat detection and defense" + title: DomainTools Feeds + description: "The DomainTools Feed provides real-time access to newly registered and observed domains, enabling proactive threat detection and defense." inputs: - type: cel - title: "DomainTools Newly Observed Domains" - description: "DomainTools Newly Observed Domains provides real-time access to newly registered and observed domains, enabling proactive threat detection and defense" + title: "Collect DomainTools Feeds" + description: "The DomainTools Feed provides real-time access to newly registered and observed domains, enabling proactive threat detection and defense." owner: github: elastic/security-service-integrations type: partner