Skip to content

DI: support per-method iseqs for line probes on pre-loaded files#5501

Draft
p-datadog wants to merge 27 commits intomasterfrom
di-per-method-iseq
Draft

DI: support per-method iseqs for line probes on pre-loaded files#5501
p-datadog wants to merge 27 commits intomasterfrom
di-per-method-iseq

Conversation

@p-datadog
Copy link
Copy Markdown
Member

@p-datadog p-datadog commented Mar 24, 2026

What does this PR do?
When a file's whole-file (:top) iseq has been garbage collected, the backfill now stores surviving per-method iseqs and the instrumenter can use them to target line probes.

Motivation:
86% of files loaded before DI starts have their whole-file iseq GC'd but still have surviving per-method iseqs (measured on a Rails app with 146 gems: 1987 of 2309 missing files had 22739 surviving method iseqs). Previously these files were completely untargetable by line probes.

Change log entry
None.

Additional Notes:

  • Depends on DI: backfill CodeTracker registry with iseqs for pre-loaded files #5496 (backfill) which depends on DI: add a C extension #5111 (C extension)
  • backfill_registry now stores per-method iseqs in a separate per_method_registry keyed by path
  • New iseq_for_line(suffix, line) method checks whole-file iseq first, falls back to per-method iseqs by checking trace_points for the target line
  • Instrumenter uses iseq_for_line when available, falls back to iseqs_for_path_suffix for compatibility with older code trackers
  • 14% of missing files (322) have no surviving iseqs at all — these are files that only run setup code without defining methods

How to test the change?
Unit tests added + manual testing via gobo targeting stdlib with partial iseqs:

2026-03-24_20-35-46 2026-03-24_20-35-57 2026-03-24_20-36-03

The screenshot shows Set#add captured with argument 1. The gobo demo calls Set.new([1, 2, 3]) then s.add(4) and s.add(2), but Set.new internally calls add for each element (1, 2, 3), so the probe fires on add(1) first. Due to the DI rate limit (one snapshot per second by default), only the first invocation is captured.

@p-datadog p-datadog added the AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos label Mar 24, 2026
@pr-commenter
Copy link
Copy Markdown

pr-commenter bot commented Mar 25, 2026

Benchmarks

Benchmark execution time: 2026-03-28 05:58:43

Comparing candidate commit 4107334 in PR branch di-per-method-iseq with baseline commit 609f934 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 46 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

@p-datadog p-datadog changed the base branch from di-iseq-backfill to master March 25, 2026 00:28
@github-actions github-actions bot added the profiling Involves Datadog profiling label Mar 25, 2026
p-ddsign and others added 18 commits March 27, 2026 14:13
When CodeTracker starts, use the all_iseqs C extension to populate the
registry with instruction sequences for files that were loaded before
tracking began. This enables line probes on third-party code and
application code loaded at boot time.

Only whole-file iseqs (first_lineno == 0) are backfilled — per-method
iseqs require instrumenter changes to select the correct iseq for a
target line and will be supported in a follow-up.

Backfill does not overwrite entries from :script_compiled, which are
authoritative. The C extension availability is checked via
DI.respond_to?(:all_iseqs) so the code gracefully degrades when
the extension is not compiled.

- Added CodeTracker#backfill_registry
- Called from CodeTracker#start after trace point is enabled
- Added RBS signature
- Added tests for backfill behavior and C extension fallback

Co-Authored-By: Claude <noreply@anthropic.com>
- Added rescue block around backfill_registry so failures are
  best-effort (logged + telemetry) rather than propagating
- Replaced all skip-based tests with mock-based tests that exercise
  backfill logic without requiring the compiled C extension
- Added tests for: mixed iseq types, multiple files, error handling,
  suffix/exact lookup on backfilled entries, start ordering
- 27 examples, 0 failures, 0 pending, 0 skipped

Co-Authored-By: Claude <noreply@anthropic.com>
Tests the end-to-end flow: test class loaded before code tracking
starts → CodeTracker#start triggers backfill via all_iseqs C extension
→ iseq recovered from object space → line probe installed on
backfilled iseq → probe fires and captures local variables.

Runs under rake spec:di_with_ext (requires compiled C extension).

Three test cases:
- Probe installs successfully on backfilled iseq
- Probe fires when target line executes
- Snapshot captures local variables from backfilled iseq

Co-Authored-By: Claude <noreply@anthropic.com>
On macOS CI the C extension is compiled, so backfill_registry populates
the CodeTracker registry with pre-loaded files during start. This broke
existing tests that expect the registry to be empty after start or to
contain exactly N explicitly-loaded files.

Fix by stubbing backfill_registry in test contexts that exercise
:script_compiled behavior. Backfill is tested separately in its own
describe blocks.

Affected contexts:
- CodeTracker #start (before block)
- CodeTracker shared context 'when code tracker is running'
- CodeTracker #iseqs_for_path_suffix (around block)
- Instrumenter shared context 'with code tracking'

Co-Authored-By: Claude <noreply@anthropic.com>
…kfill

The backfill filter used first_lineno == 0 to identify whole-file iseqs,
but most whole-file iseqs from all_iseqs have first_lineno == 1. The new
DI.iseq_type method reads the iseq type directly from the Ruby VM struct
and returns a symbol (:top, :method, :block, :class, etc.).

The backfill now filters by type == :top || type == :main, which
correctly identifies whole-file iseqs regardless of first_lineno.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
rb_iseq_type is an internal Ruby function that only exists in Ruby 3.1+.
On Ruby 2.7 and 3.0, referencing it causes an undefined symbol error at
load time, crashing the entire C extension (including all_iseqs and
exception_message which work fine on those versions).

Use have_func in extconf.rb to detect rb_iseq_type at compile time, and
wrap the iseq_type function + registration in #ifdef HAVE_RB_ISEQ_TYPE.
The Ruby code in code_tracker.rb already handles the missing method via
DI.respond_to?(:iseq_type) with a first_lineno fallback.

Co-Authored-By: Claude <noreply@anthropic.com>
… require

- Add doc comments for rb_iseqw_new and rb_iseqw_to_iseq prototypes in di.c
  (internal Ruby functions used without documentation)
- Add error handling test coverage for backfill_registry: verify logger.debug
  is called with the error message and telemetry.report is called when
  DI.current_component is available
- Add test coverage for the first_lineno == 0 fallback path when iseq_type
  is unavailable (Ruby versions without rb_iseq_type)
- Add missing require "datadog/di/spec_helper" to iseq_type_spec.rb for
  consistency with other ext specs
- Fix skip message: iseq_type availability depends on rb_iseq_type in the
  Ruby runtime, not on the DI C extension

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- di.c: Document that rb_iseq_type was added in Ruby 3.1, explain the
  HAVE_RB_ISEQ_TYPE compile-time guard, and note the fallback path
- code_tracker.rb: Replace "first_lineno == 0" YARD doc with full
  description of both strategies (iseq_type on 3.1+, first_lineno
  heuristic on older Rubies) and their tradeoffs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The YARD doc claimed the first_lineno == 0 fallback "can match top-level
eval iseqs" but this is wrong. InstructionSequence.compile passes
first_lineno = 1 (not 0), and require/load passes INT2FIX(0) in Ruby's
rb_iseq_new_top/rb_iseq_new_main. Both strategies produce the same
result in practice.

Verified by reading Ruby 3.0 source (iseq.c lines 813-822):
  rb_iseq_new_with_opt(ast, name, path, realpath, INT2FIX(0), ...)
  → ISEQ_TYPE_TOP with first_lineno = 0

And compile path (iseq.c line 1064):
  rb_iseq_new_with_opt(&ast->body, label, file, realpath, line, ...)
  → line defaults to INT2FIX(1) for compile/eval

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Verify idempotency: calling backfill_registry a second time with the
same iseqs doesn't duplicate entries (registry.key? guard).

Also verify that a second call with new iseqs adds them without
overwriting entries from the first call.

Co-Authored-By: Claude <noreply@anthropic.com>
The guard was purely defensive — the C extension is always compiled
when DI is active (enforced by environment_supported? in component.rb).
The rescue block at the bottom of backfill_registry already catches any
exception if file_iseqs fails, making the guard redundant.

Co-Authored-By: Claude <noreply@anthropic.com>
The method is called for side effects only. Without the explicit nil,
the happy path leaked the synchronize return value and the rescue path
leaked the telemetry report return value.

Co-Authored-By: Claude <noreply@anthropic.com>
On older Rubies, accessing an uninitialized instance variable via &.
produces a warning: "instance variable @current_components not
initialized". This triggers loading_spec failures because
datadog/di/preload produces unexpected output.

The variable is accessed by DI.current_component (called from
backfill_registry's error boundary) before any component is added.
Initializing to nil at module level suppresses the warning while
preserving the existing lazy-init behavior in add_current_component.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RSpec's verify_partial_doubles rejects allow(DI).to receive(:iseq_type)
when the method doesn't exist on the module. On Ruby < 3.1,
rb_iseq_type is not available so DI.iseq_type is never defined.

Fix: conditionally stub iseq_type only when it exists. On older Rubies,
let respond_to?(:iseq_type) return false naturally and exercise the
first_lineno == 0 fallback path — which is what production does.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The pre-loaded test class's iseq can be garbage collected before
backfill walks the object space, causing DITargetNotInRegistry. In
production, application code is referenced by live constants/methods
and survives GC. In the test, the iseq is more ephemeral.

Disable GC around activate_tracking! (which calls backfill_registry)
to ensure the iseq is still in the object space when all_iseqs runs.
Re-enable immediately after.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two root causes:

1. code_tracker_spec.rb: iseq_type was stubbed with and_call_original,
   but the C function expects a real RubyVM::InstructionSequence, not a
   test double. Stub returns :top for first_lineno==0, :method otherwise.

2. backfill_integration_spec.rb: The top-level file iseq (first_lineno=0,
   type=:top) is not referenced by any constant or method after loading.
   GC could collect it between require_relative (file load time) and the
   before block's backfill_registry call. Move GC.disable to file level,
   immediately before require_relative, so the iseq survives until
   backfill walks the object space.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
The ivar is initialized to nil to avoid Ruby 2.6/2.7 warnings.
RBS type needs to reflect this. Silence false positive on <<
after ||= (Steep doesn't track that ||= guarantees non-nil).

Co-Authored-By: Claude <noreply@anthropic.com>
@p-datadog p-datadog force-pushed the di-per-method-iseq branch from 9012b29 to dee647b Compare March 27, 2026 23:45
@p-datadog p-datadog changed the base branch from master to di-iseq-backfill March 27, 2026 23:45
@github-actions
Copy link
Copy Markdown

github-actions bot commented Mar 27, 2026

Typing analysis

Note: Ignored files are excluded from the next sections.

steep:ignore comments

This PR introduces 3 steep:ignore comments, and clears 1 steep:ignore comment.

steep:ignore comments (+3-1)Introduced:
lib/datadog/di/base.rb:109
lib/datadog/di/instrumenter.rb:342
lib/datadog/di/instrumenter.rb:344
Cleared:
lib/datadog/di/instrumenter.rb:341

Untyped methods

This PR introduces 1 partially typed method, and clears 1 partially typed method. It increases the percentage of typed methods from 61.66% to 61.74% (+0.08%).

Partially typed methods (+1-1)Introduced:
sig/datadog/di/code_tracker.rbs:16
└── def iseqs_for_path_suffix: (String suffix) -> untyped
Cleared:
sig/datadog/di/code_tracker.rbs:14
└── def iseqs_for_path_suffix: (String suffix) -> untyped

If you believe a method or an attribute is rightfully untyped or partially typed, you can add # untyped:accept on the line before the definition to remove it from the stats.

@p-datadog p-datadog changed the base branch from di-iseq-backfill to master March 27, 2026 23:46
p-datadog pushed a commit that referenced this pull request Mar 28, 2026
* origin/di-per-method-iseq: (23 commits)
  Improve DITargetNotInRegistry error messages
  Update remote config test for new error message format
  Fix throwable integration test to include stacktrace
  Add integration test for line probe via per-method iseq
  Support per-method iseqs for line probes on pre-loaded files
  Fix Steep: allow nil for @current_components
  Fix StandardRB: add parens to ternary, remove extra blank line
  Fix backfill_registry test failures
  Disable GC during backfill integration test to prevent iseq collection
  Fix backfill_registry tests on Ruby < 3.1 (iseq_type unavailable)
  Initialize @current_components to suppress Ruby 2.6/2.7 warning
  Return nil explicitly from backfill_registry
  Remove respond_to?(:all_iseqs) guard from backfill_registry
  Add tests for calling backfill_registry twice
  Fix inaccurate comment: first_lineno == 0 heuristic matches iseq_type
  Document iseq_type Ruby 3.1 dependency and two-strategy backfill
  Review fixes: doc comments, error handling test coverage, spec_helper require
  Guard rb_iseq_type behind have_func for Ruby < 3.1 compat
  Add DI.iseq_type C extension; use type instead of first_lineno in backfill
  Stub backfill_registry in pre-existing tests
  ...
p-datadog pushed a commit that referenced this pull request Mar 28, 2026
* origin/di-per-method-iseq: (23 commits)
  Improve DITargetNotInRegistry error messages
  Update remote config test for new error message format
  Fix throwable integration test to include stacktrace
  Add integration test for line probe via per-method iseq
  Support per-method iseqs for line probes on pre-loaded files
  Fix Steep: allow nil for @current_components
  Fix StandardRB: add parens to ternary, remove extra blank line
  Fix backfill_registry test failures
  Disable GC during backfill integration test to prevent iseq collection
  Fix backfill_registry tests on Ruby < 3.1 (iseq_type unavailable)
  Initialize @current_components to suppress Ruby 2.6/2.7 warning
  Return nil explicitly from backfill_registry
  Remove respond_to?(:all_iseqs) guard from backfill_registry
  Add tests for calling backfill_registry twice
  Fix inaccurate comment: first_lineno == 0 heuristic matches iseq_type
  Document iseq_type Ruby 3.1 dependency and two-strategy backfill
  Review fixes: doc comments, error handling test coverage, spec_helper require
  Guard rb_iseq_type behind have_func for Ruby < 3.1 compat
  Add DI.iseq_type C extension; use type instead of first_lineno in backfill
  Stub backfill_registry in pre-existing tests
  ...
p-ddsign and others added 2 commits March 27, 2026 23:10
Root cause: the top-level (:top) iseq for the test class file has no
references after loading completes — only class/method child iseqs
survive via BackfillIntegrationTestClass. The previous approach disabled
GC at file load time and re-enabled it in the before block after
backfill. This protected the first test, but after deactivate_tracking!
cleared the registry (the only reference to the iseq), GC could collect
it before the next test's backfill_registry walked object space.

Fix: capture the top-level iseq in a constant (BACKFILL_TEST_TOP_ISEQ)
immediately after loading, before GC can collect it. The constant keeps
the iseq alive for the lifetime of the process, so backfill_registry
can find it in every test regardless of GC activity.

Verified: 0 failures across 8 consecutive full DI suite runs (714
examples each), vs ~20% failure rate before the fix.

Co-Authored-By: Claude <noreply@anthropic.com>
Add "Iseq Lifecycle and GC" section to DynamicInstrumentationDevelopment.md
covering: iseq types created on file load, which survive GC and why,
implications for backfill_registry, and the correct test pattern for
keeping top-level iseqs alive across tests.

Add cross-reference from code_tracker.rb backfill_registry docstring.

Co-Authored-By: Claude <noreply@anthropic.com>
@p-datadog p-datadog force-pushed the di-per-method-iseq branch from dee647b to 4107334 Compare March 28, 2026 05:30
@datadog-prod-us1-6

This comment has been minimized.

p-ddsign and others added 7 commits March 30, 2026 08:42
When a file's whole-file (:top) iseq has been garbage collected,
per-method iseqs from all_iseqs can still be used to target line
probes. This covers 86% of files that were previously untargetable.

Changes:
- backfill_registry stores per-method iseqs in per_method_registry
  (grouped by path) instead of discarding them
- New iseq_for_line(suffix, line) method tries whole-file iseq first,
  then searches per-method iseqs for one whose trace_points include
  the target line
- Instrumenter uses iseq_for_line when available, falls back to
  iseqs_for_path_suffix for compatibility

Verified: 37 code_tracker tests pass, lint clean, types clean.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Loads a test class, GCs the top iseq, then verifies that the backfill
finds the surviving method iseq and a line probe can be installed,
fired, and captures local variables through it.

Precondition checks skip the test if GC didn't collect the top iseq
or if the C extension is unavailable.

Verified: 3 integration tests pass (install, fire, capture locals).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The throwable now includes a stacktrace array (from the C extension
commit). Also update error message assertion for the new
raise_if_probe_in_loaded_features format.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The raise_if_probe_in_loaded_features now reports whether per-method
iseqs exist or not, instead of the generic "not in code tracker
registry" message.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Distinguish between "has per-method iseqs but none cover this line"
and "has no surviving iseqs at all". Include the target line number
in the error. Helps users understand why a line probe failed and
whether the file is partially targetable.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The method gained line_no and code_tracker parameters in the per-method
iseq support commit, but the RBS declaration was not updated.

Co-Authored-By: Claude <noreply@anthropic.com>
On Ruby < 3.1, DI.iseq_type is not defined (the C extension gates it
with have_func for rb_iseq_type). RSpec's partial double verification
rejects stubs for undefined methods, causing 10 failures in the
iseq_for_line and per-method iseq tests.

Fix: guard iseq_type stubs with `if Datadog::DI.respond_to?(:iseq_type)`,
matching the existing pattern in backfill_registry tests (lines 246-255).
On Ruby < 3.1, the code under test falls back to first_lineno == 0.

Co-Authored-By: Claude <noreply@anthropic.com>
@p-datadog p-datadog changed the base branch from master to di-iseq-backfill March 30, 2026 12:42
@p-datadog p-datadog force-pushed the di-per-method-iseq branch from 873a846 to af1e513 Compare March 30, 2026 12:43
@p-datadog p-datadog changed the base branch from di-iseq-backfill to master March 30, 2026 12:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos profiling Involves Datadog profiling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants