Fix/data stream delete support#1755
Merged
Merged
Conversation
Elasticsearch data streams do not support single-document DELETE operations. When ElastAlert2 uses a data stream as the writeback index, aggregation cleanup fails because es.delete() returns 404. This replaces two es.delete() calls in elastalert.py with es.delete_by_query() using a term query on _id, which is supported on data streams while maintaining the same behavior for regular indices. Affected methods: - send_pending_alerts(): cleanup after sending scheduled aggregation - get_aggregated_matches(): cleanup after collecting aggregated matches Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add delete_by_query to mock_es_client to fix 4 test failures caused by the es.delete() → delete_by_query() change - Add CHANGELOG entry for the data stream fix Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Owner
|
Thanks for the submission. Are you manually creating the data streams? And did you confirm that the delete_by_query() invocation worked (correctly deleted) the documents when using the standard index rather than data stream? |
Contributor
Author
|
Hi Jason,
sorry for late response. Data streams are created from templates that are
used in our corporate environment. Unfortunately I don’t have details.
Unfortunately I haven’t tested
delete_by_query() invocation with standard index. I’d like to execute test
with standard index in following days, and I’ll let you know the outcome.
Best regards
…On Wed, 1 Apr 2026 at 13:26, Jason Ertel ***@***.***> wrote:
*jertel* left a comment (jertel/elastalert2#1755)
<#1755?email_source=notifications&email_token=AUNKOHTCEZHZJQLZCB4CMPD4TT4GLA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMJWHEZTQOJRGIZKM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2LK4DSL5RW63LNMVXHIX3POBSW4X3DNRUWG2Y#issuecomment-4169389122>
Thanks for the submission. Are you manually creating the data streams? And
did you confirm that the delete_by_query() invocation worked (correctly
deleted) the documents when using the standard index rather than data
stream?
—
Reply to this email directly, view it on GitHub
<#1755?email_source=notifications&email_token=AUNKOHTCEZHZJQLZCB4CMPD4TT4GLA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMJWHEZTQOJRGIZKM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2LK4DSL5RW63LNMVXHIX3POBSW4X3DNRUWG2Y#issuecomment-4169389122>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AUNKOHWPOCP3U6DI2AYFCST4TT4GLAVCNFSM6AAAAACXI7CTMKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DCNRZGM4DSMJSGI>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Contributor
Author
|
Hi Jason, i tested suggested changes for compatibility and i do not see any problems. See attached protocol |
jertel
approved these changes
Apr 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Replace es.delete() with delete_by_query() in aggregation cleanup (send_pending_alerts and
add_aggregated_alert) to support Elasticsearch data streams as the writeback index.
Elasticsearch data streams do not support single-document DELETE operations (es.delete()), which
causes aggregation cleanup to fail with an error. The delete_by_query API works on data streams and
is functionally equivalent for this use case.
Checklist
make test-dockerwith my changes.Questions or Comments
configuration options or user-facing API changes.
test_twilio_rest_exception) are pre-existing and unrelated to this change.
aggregation alerts are now correctly cleaned up after being sent.