Skip to content

[ti_google_threat_intelligence] Remove manual pipeline and field mappings once pipeline-transform dependency handling is supported #14286

@niraj-crest

Description

@niraj-crest

In the ti_google_threat_intelligence integration, we use a transform that queries data from 14 threat list data streams (based on IP, Domain, URL, and Hash) and routes them through a pipeline to map under a common schema.

Currently, due to the limitation that integration version is not automatically tracked or updated within the transform’s pipeline definition, forgetting to manually update it during integration upgrades can lead to issues. There is no existing validation in the Elastic packaging to enforce this, and the issue is being tracked here: elastic/package-spec#833.

To proceed for now:

  • We’ve removed the attached pipeline from the transform.
  • This means incoming documents won’t be filtered unless the pipeline is manually added.
  • To ensure system tests continue to pass, we've included the <product>_fields.yml files for field mapping.

Once elastic/package-spec#833 is resolved, we plan to:

  • Add the transform pipeline directly back into the transform definition.
  • Remove the manually maintained <product>_fields.yml files.

Relates:
#13189

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions