[Google Threat Intelligence] Add Integration Package (#13189)

- Added 4 data streams (cryptominer, first_stage_delivery_vectors, infostealer, iot).
- Added data collection logic for the data streams.
- Added the ingest pipeline for the data streams.
- Mapped fields according to the ECS schema and added Fields metadata in the appropriate yaml files.
- Added dashboards and visualizations.
- Added system test cases for the data stream.

Author: niraj-crest

.github/CODEOWNERS

@@ -434,6 +434,7 @@
 /packages/ti_domaintools @elastic/security-service-integrations
 /packages/ti_eclecticiq @elastic/security-service-integrations
 /packages/ti_eset @elastic/security-service-integrations
+/packages/ti_google_threat_intelligence @elastic/security-service-integrations
 /packages/ti_maltiverse @elastic/security-service-integrations
 /packages/ti_mandiant_advantage @elastic/security-service-integrations
 /packages/ti_misp @elastic/security-service-integrations

packages/ti_google_threat_intelligence/_dev/build/build.yml

@@ -0,0 +1,3 @@
+dependencies:
+  ecs:
+    reference: [email protected]

packages/ti_google_threat_intelligence/_dev/build/docs/README.md

@@ -0,0 +1,207 @@
+# Google Threat Intelligence
+
+## Overview
+
+[Google Threat Intelligence](https://gtidocs.virustotal.com/) is a security solution that helps organizations detect, analyze, and mitigate threats. It leverages Google's global telemetry, advanced analytics, and vast infrastructure to provide actionable insights. Key features include threat detection, malware and phishing analysis, and real-time threat alerts.
+
+Google Threat Intelligence uses the **[Threat List API](https://gtidocs.virustotal.com/reference/get-hourly-threat-list)** to deliver hourly data chunks. The Threat Lists feature allows customers to consume **Indicators of Compromise (IOCs)** categorized by various threat types.
+
+## Threat List API Feeds
+
+The Threat List API provides the following types of threat feeds:
+
+- **Cryptominers**
+- **First Stage Delivery Vectors**
+- **Infostealers**
+- **Internet of Things (IoT)**
+
+## GTI Subscription Tiers
+
+Customers can access a subset of the available threat lists based on their **Google Threat Intelligence (GTI) tier**:
+
+- **GTI Standard**: Ransomware, Malicious Network Infrastructure
+- **GTI Enterprise**: Ransomware, Malicious Network Infrastructure, Malware, Threat Actor, Daily Top Trending
+- **GTI Enterprise+**: Access to all available threat lists
+
+## Data Streams
+
+Data collection is available for four feed types: `cryptominer`, `first_stage_delivery_vectors`, `infostealer`, and `iot`, each provided through a separate data stream. Users can enable data streams based on their GTI subscription tier. If a user enables data collection for a data stream they do not have access to, it will result in an error log on the **Discover** page.
+
+## Requirements
+
+Elastic Agent must be installed. For more details, check the Elastic Agent [installation instructions](docs-content://reference/fleet/install-elastic-agents.md).
+
+## Setup
+
+### To collect logs through REST API, follow the below steps:
+
+- VirusTotal URL will work as the base URL for this integration: https://www.virustotal.com
+- An API key will be used to authenticate your request.
+- **Time Selection of Initial Interval and Interval**:
+  - Users need to specify the **initial interval** and **interval** in an hourly format, such as **2h**, **3h**, etc.
+**Note:** Please make sure both initial interval and interval are in hours and greater than 1 hour.
+
+### Enabling the integration in Elastic:
+
+1. In Kibana, go to **Management > Integrations**.
+2. In the "Search for integrations" search bar, type **Google Threat Intelligence**.
+3. Click on the **Google Threat Intelligence** integration from the search results.
+4. Click on the **Add Google Threat Intelligence** button to add the integration.
+5. While adding the integration, to collect logs via REST API, provide the following details:
+   - Enable the type of data stream you have access to.
+   - Access Token
+   - Initial Interval
+   - Interval
+   - (Optional) Query to add custom query filtering on relationship, GTI score, and positives.
+6. Click on **Save and Continue** to save the integration.
+**Note:** Please make only the threat feed types you have the privilege to access are enabled.
+
+## Transforming Data for Up-to-Date Insights
+
+To keep the collected data up to date, **Transforms** are used.
+
+Users can view the transforms by navigating to **Management > Stack Management > Transforms**.
+
+Follow **Steps to enable transforms** to enable transforms and populate `Threat Feed Overview` dashboard.
+
+Here, users can see continuously running transforms and also view the latest transformed GTI data in the **Discover** section.
+
+The `labels.is_transform_source` field indicates log origin:
+- **False** for transformed index
+- **True** for source index
+
+Currently, four transforms are available across all 4 data streams.
+
+The following are four transforms along with their associated pipelines:
+
+| Transform Name                                                                                                                                                           | Description                                              |
+| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------- |
+| IP Transform (ID: `logs-ti_google_threat_intelligence.ip_ioc`, Pipeline: `ti_google_threat_intelligence-latest_ip_ioc-transform-pipeline`)                               | Keeps IP entity type data up to date.                    |
+| URL Transform (ID: `logs-ti_google_threat_intelligence.url_ioc`, Pipeline: `ti_google_threat_intelligence-latest_url_ioc-transform-pipeline`)                            | Keeps URL entity type data up to date.                   |
+| Domain Transform (ID: `logs-ti_google_threat_intelligence.domain_ioc`, Pipeline: `ti_google_threat_intelligence-latest_domain_ioc-transform-pipeline`)                   | Keeps Domain entity type data up to date.                |
+| File Transform (ID: `logs-ti_google_threat_intelligence.file_ioc`, Pipeline: `ti_google_threat_intelligence-latest_file_ioc-transform-pipeline`)                         | Keeps File entity type data up to date.                  |
+
+For example:
+
+- The query `event.module: ti_google_threat_intelligence and labels.is_transform_source: true` indicates that the logs originate from the **source index**.
+- The query `event.module: ti_google_threat_intelligence and labels.is_transform_source: false` indicates that the logs originate from the **transformed index**.
+
+A **retention policy** is used to remove data older than the default retention period. For more details, refer to the [Retention Policy Documentation](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-put-transform#operation-transform-put-transform-body-application-json-retention_policy).
+
+In this integration, all data streams have a **retention period of 30 days**.
+
+### Enrichment with Detection Rules
+
+Detection Rules match the user's Elastic environment data with GTI data, generating an alert if a match is found. To access detection rules:
+
+1. Navigate to **Security > Rules > Detection Rules** and click on **Add Elastic Rules**.
+2. Search for **Google Threat Intelligence** to find prebuilt Elastic detection rules.
+3. Four detection rules are available for **IP, URL, File, and Domain**. Users can install one or more rules as needed.
+
+To tailor a rule based on Elastic environment:
+
+1. Click the three dots on the right side of any detection rule.
+2. Select **Duplicate Rule**.
+3. Modify the duplicated rule to tailor it to your Elastic environment:
+   - **Index Pattern**: Add the index pattern relevant to your data. Keeping this specific ensures optimal performance.
+   - **Custom Query**: Further refine rule conditions.
+   - **Indicator Mapping**: Map relevant fields from your Elastic environment to GTI fields. Do not modify the **indicator index field**.
+   - **Schedule Rules**:
+     - **Set Runs Every** - Defines how frequently the rule runs.
+     - **Additional Lookback Time** - Specifies how far back to check for matches.
+
+Once saved, successfully executed rules will generate alerts. Users can view these alerts in the **Alerts** section.
+
+**Note:** A transform runs in the background to filter relevant data from alerts. The `data_stream.dataset: ti_google_threat_intelligence.enriched_ioc` field represents logs for enriched threat intelligence data, which can be analyzed in the **Discover** section.
+
+The following are the names of the four sample rules:
+
+| Sample Rule Name                                      | Description                                                                                                                |
+| ----------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
+| Google Threat Intelligence URL IOC Correlation        | Detects and alerts on matches between URL IOCs collected by GTI data with user's selected Elastic environment data.        |
+| Google Threat Intelligence Domain IOC Correlation     | Detects and alerts on matches between Domain IOCs collected by GTI data with user's selected Elastic environment data.     |
+| Google Threat Intelligence File IOC Correlation       | Detects and alerts on matches between File IOCs collected by GTI data with user's selected Elastic environment data.       |
+| Google Threat Intelligence IP Address IOC Correlation | Detects and alerts on matches between IP Address IOCs collected by GTI data with user's selected Elastic environment data. |
+
+The following transform and its associated pipelines are used to filter relevant data from alerts. Follow **Steps to enable transforms** to enable these transforms and populate `Threat Intelligence` and `Adversary Intelligence` dashboards.
+
+| Transform Name                                                                                                                                          | Description                                                                     |
+| ------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
+| Detected IOC Transform  (ID: `logs-ti_google_threat_intelligence.rule`, Pipeline: `ti_google_threat_intelligence-correlation_detection_rule-pipeline`)  | Filters and extracts necessary information from Detected IOCs from threat feed. |
+
+### Steps to enable transforms
+
+1. Navigate to **Stack Management > Transforms** in Kibana.
+2. Locate the transform you want to enable by searching for its **Transform ID**.
+3. Click the **three dots** next to the transform, then select **Edit**.
+4. Under the **Destination configuration** section, set the **Ingest Pipeline**:
+   - Each transform in the **Google Threat Intelligence** integration has a corresponding ingest pipeline.
+   - Refer to the **Transforms table** above for the appropriate pipeline name associated with transform.
+   - Prefix the pipeline name with the integration version.  
+     For example:  
+     ```
+     0.1.0-ti_google_threat_intelligence-latest_ip_ioc-transform-pipeline
+     ```
+   - Click **Update** to save the changes.
+5. Click the **three dots** again next to the transform and select **Start** to activate it.
+
+**Note:** After updating the integration, make sure to update the pipeline prefix accordingly.
+
+## Limitations
+
+1. If an event contains multiple matching mappings (e.g., two file hash fields within the same event match GTI data), only one alert per detection rule will be generated for that event.
+2. If an IOC from the user's Elasticsearch index is enriched with GTI information, and the GTI information is updated later, the changes are not reflected in the dashboards because Elastic detection rules only run on live data.
+
+## Troubleshooting
+
+1. If you encounter a privilege error for a threat feed type, such as: `You are not authorized to perform the requested operation`, verify your privilege level and enable only the threat feeds you have access to.
+2. If you see an error like `Package 2025031310 is not available until 2025-03-13 at 11:00 UTC because of privacy policy.`, ensure that your initial interval and interval are set in hours and are greater than one hour.
+3. If events are not appearing in the transformed index, check if transforms are running without errors. If you encounter issues, refer to [Troubleshooting transforms](https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-troubleshooting.html).
+4. If detection rules take longer to run, ensure you have specified index patterns and applied queries to make your source events more specific.
+   **Note:** More events in index patterns mean more time needed for detection rules to run.
+5. Ensure that relevant fields are correctly mapped in the **Indicator Mapping** section. Verify that fields in the specified index pattern are properly mapped, and ensure entity-specific fields (e.g., IP fields to IP fields, keyword fields like file hash SHA256 to corresponding file hash SHA256 fields) are accurately configured.
+6. If any transform is not in a **Healthy** state, try resetting it:
+   - Click the **three dots** next to the transform, then select **Reset**.
+   - After resetting, follow the **Steps to enable transforms** above to reconfigure and restart the transform.
+
+## Logs Reference
+
+### Cryptominers
+
+This is the `Cryptominer` dataset.
+
+#### Example
+
+{{event "cryptominer"}}
+
+{{fields "cryptominer"}}
+
+### First Stage Delivery Vectors
+
+This is the `First Stage Delivery Vectors` dataset.
+
+#### Example
+
+{{event "first_stage_delivery_vectors"}}
+
+{{fields "first_stage_delivery_vectors"}}
+
+### Infostealers
+
+This is the `Infostealers` dataset.
+
+#### Example
+
+{{event "infostealer"}}
+
+{{fields "infostealer"}}
+
+### Internet of Things
+
+This is the `Internet of Things` dataset.
+
+#### Example
+
+{{event "iot"}}
+
+{{fields "iot"}}

packages/ti_google_threat_intelligence/_dev/deploy/docker/docker-compose.yml

@@ -0,0 +1,15 @@
+version: '2.3'
+services:
+  ti_google_threat_intelligence:
+    image: docker.elastic.co/observability/stream:v0.15.0
+    hostname: ti_google_threat_intelligence
+    ports:
+      - 8090
+    volumes:
+      - ./files:/files:ro
+    environment:
+      PORT: '8090'
+    command:
+      - http-server
+      - --addr=:8090
+      - --config=/files/config.yml