Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed commit message
Prefer the
copy_from
option of theset
processor for high certain high volume integrations.When the field to be copied is already just a string, and so the set processor with mustache isn't being used for the side effect of converting to a string, then it's quite a bit faster to use
copy_from
rather thanvalue
(with mustache templating).For example, in a large cluster that I was looking at a few minutes ago, the most expensive single
set
processor is this one:It's taking 2.8 microseconds per doc, as compared to the average of all set processor invocations for the same pipeline which is only .6 microseconds per doc. The cluster in question is processing billions and billions of documents per hour, though, so microseconds add up (and this particularly-expensive set processor is the eighth-most expensive processor for the entire pipeline).
I'm marking this a draft because I'm not 100% absolutely sure about my changes or the correct process for PRs on this repo.