fix: parse 'create' responses in elasticsearch bulk API#969
Open
Vitalymt wants to merge 1 commit into
Open
Conversation
When batch_op_type is set to 'create' (required for ES/OpenSearch data streams), the bulk API returns responses with a 'create' key instead of 'index'. The plugin was only checking for the 'index' key, causing it to report errors and mark events as failed even though documents were successfully created. Now the plugin checks for both 'index' and 'create' keys in the response items, falling back to error only when neither is present. Fixes ozontech#898
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When using output.elasticsearch with batch_op_type: create (required for ES/OpenSearch data streams), the plugin logs false errors and increments error metrics even though documents are successfully created.
reportESErrors() only checks for the "index" key in bulk response items. When batch_op_type: create is used, Elasticsearch returns "create": {...} instead, which the plugin doesn't recognize:
unknown elasticsearch response, 'index' field in the response is empty
some events from batch aren't written
Events are actually written successfully (status 201), but operators see error logs and incorrect indexingErrors metrics.
Fix
Added fallback to check for the "create" key when "index" is not found. Status code validation now works correctly for both response types. Added test cases covering "create" and mixed "index" + "create" responses.
Fixes #898