Skip to content

Conversation

joegallo
Copy link
Contributor

@joegallo joegallo commented Sep 10, 2025

Proposed commit message

Prefer the copy_from option of the set processor for high certain high volume integrations.

When the field to be copied is already just a string, and so the set processor with mustache isn't being used for the side effect of converting to a string, then it's quite a bit faster to use copy_from rather than value (with mustache templating).

For example, in a large cluster that I was looking at a few minutes ago, the most expensive single set processor is this one:

{
  "set": {
    "field": "cloud.account.id",
    "if": "ctx.aws?.vpcflow?.account_id != null",
    "value": "{{aws.vpcflow.account_id}}"
  }
}

It's taking 2.8 microseconds per doc, as compared to the average of all set processor invocations for the same pipeline which is only .6 microseconds per doc. The cluster in question is processing billions and billions of documents per hour, though, so microseconds add up (and this particularly-expensive set processor is the eighth-most expensive processor for the entire pipeline).


I'm marking this a draft because I'm not 100% absolutely sure about my changes or the correct process for PRs on this repo.

because copy_from is faster than value (when the value is a mustache
template that merely does field access).
@elastic-vault-github-plugin-prod

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@joegallo
Copy link
Contributor Author

/test benchmark fullreport

@andrewkroh andrewkroh added Integration:panw Palo Alto Next-Gen Firewall Integration:aws AWS Integration:gcp Google Cloud Platform labels Sep 10, 2025
@andrewkroh
Copy link
Member

/test benchmark fullreport

@elasticmachine
Copy link

💚 Build Succeeded

History

Copy link

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>enhancement Integration:aws AWS Integration:gcp Google Cloud Platform Integration:panw Palo Alto Next-Gen Firewall
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants