fix(llmobs): mark APM spans when SDK submits LLMObs event to prevent processor duplication#17212
Draft
ZStriker19 wants to merge 2 commits intomainfrom
Draft
fix(llmobs): mark APM spans when SDK submits LLMObs event to prevent processor duplication#17212ZStriker19 wants to merge 2 commits intomainfrom
ZStriker19 wants to merge 2 commits intomainfrom
Conversation
…processor duplication When the ddtrace SDK submits an LLMObs event for a span (including OTel gen_ai spans handled via the OTel bridge), tag the APM span with _dd.llmobs.submitted=1. The dd-go backend processor reads this tag and skips the span, preventing duplicate LLMObs events when both the SDK and the processor would otherwise cover the same span. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Codeowners resolved as |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #17007 (MLOS-523). Fixes duplicate LLMObs events for customers using both the ddtrace SDK and Strands OTel gen_ai instrumentation.
Root cause: When the SDK submits LLMObs events for OTel gen_ai spans (via the OTel bridge), the dd-go backend processor independently creates another LLMObs event for the same span (because it has
gen_ai.*attributes). This produces two LLMObs events with the same span ID.Fix: Tag the APM span with
_dd.llmobs.submitted=1when the SDK enqueues an LLMObs event for it. The companion dd-go PR reads this tag inisRelevantForLLMObsand skips the span, preventing the duplicate.This only affects spans the SDK explicitly handles. Spans that are purely OTel (not submitted by the SDK writer) are unaffected and continue to be created by the processor as before.
Companion PR
dd-go: DataDog/dd-go (link TBD — branch:
zach.groves/mlos-523/llmobs-dedup)Test plan
test_llmobs_submitted_tag_set_on_apm_span— SDK span gets markertest_llmobs_submitted_tag_not_set_without_llmobs— non-SDK span does not get marker🤖 Generated with Claude Code