[Change Proposal] Add support for ingest pipelines in transforms

[jsoriano]

Transforms allow to define ingest pipelines, so documents are processed before ingestion. This allows for example to remove fields that are not relevant in the transform, or to adjust fields intended to have different values, as in elastic/elastic-package#2218 (comment).

This is partly supported now, a destination index can have a pipeline, but currently it needs to be hard-coded:

dest:
  index: "logs-ti_anomali_latest.threatstream-3"
  aliases:
    - alias: "logs-ti_anomali_latest.threatstream"
      move_on_creation: true
  pipeline: "1.23.0-latest_ioc"

We should allow to use templates there as we allow in other places, so configuration can be something like this:

dest:
  index: "logs-ti_anomali_latest.threatstream-3"
  aliases:
    - alias: "logs-ti_anomali_latest.threatstream"
      move_on_creation: true
  pipeline: '{{ IngestPipeline "latest_ioc" }}'

This may not need changes in the spec but will need changes in Fleet.

[jsoriano]

While the situation is improved, a workaround is proposed in elastic/elastic-package#2552, to add a validation to check that the package provides the hard-coded pipeline.

[niraj-crest]

Could we please prioritize this issue?

We're currently blocked on two integrations that depend on transform pipelines:

• GreyNoise Draft PR
• Google Threat Intelligence GA tracking issue

Enabling support for pipelines would help us move both of these forward.

We’re okay with either of the following approaches:

• Using a pipeline template, as suggested here.
• Applying the workaround to add an elastic-package check for the pipeline, as discussed in elastic/elastic-package#2552

[kpollich]

Prioritizing this in our next iteration for tech definition.

[niraj-crest]

Thank you!
Could you please share an estimated timeline for this?
It would really help us in planning accordingly. Appreciate your support!